the following is a guest post from jeremy karn. this article is excerpted from mongodb + hadoop: a step-by-step tutorial. jeremy is a cofounder at mortar data, a hadoop-as-a-service provider, and creator of mortar, an open source framework
the following is a guest post from jeremy karn. this article is excerpted from ‘mongodb + hadoop: a step-by-step tutorial’. jeremy is a cofounder at mortar data, a hadoop-as-a-service provider, and creator of mortar, an open source framework for data processing.
people who are worried about scalability often find themselves looking at two tools: mongodb for storing large amounts of data easily and hadoop for processing that data. but a common question is: “how do i combine these two to really get the most out of my data?”
here’s a step-by-step tutorial that will get you up and running with mongodb and hadoop in a matter of minutes. and the best part about this tutorial is that at the end you’ll be ready to jump right into using your own mongodb data with hadoop.
for this tutorial you’ll be using apache pig, a high-level data flow language that compiles down into hadoop mapreduce jobs. it was designed to be easy to learn and simple to write. if you’ve written sql, pig will feel familiar, it is like procedural sql.
to run your hadoop jobs, you’re going to use a free mortar account. mortar provides hadoop as a service, which means you can run your jobs without worrying about how to set up and manage a multi-node hadoop cluster.
to get started, we’ve already set up a small mongodb instance on mongolab, populated it with a random sampling of twitter data from a single day (around 120,000 tweets), and created a read-only user for you.
we’ve also set up a public github repo with a mortar project that has three pig scripts ready to run. here’s what you need to do:
if you don’t already have a free github account - create one.? you’ll need a github username in step 4.
git clone firstname.lastname@example.org:mortardata/mongo-pig-examples.git cd mongo-pig-examples mortar register mongo-pig-examples
if you’re like most mongodb users, you may not have a great sense of the different fields, data types, or values in your collection. we built characterize_collection.pig to deeply inspect your collection to extract that information.
from the base directory of the mongo-pig-examples project you just cloned take a look at pigscripts/characterize_collection.pig. it loads all the data in the collection as a map, sends the map to python (udfs/python/mongo_util.py) to gather a bunch of metadata, calculates some basic information about the collection, and then it writes the results out to an s3 bucket.
to see this script in action let’s run it on a 4 node hadoop cluster. in your terminal (from the base directory of your mongo-pig-examples project) run:mortar run characterize_collection --clustersize 4
this job will take about 10 minutes to finish. you can monitor the job’s status on the command line or by going to https://app.mortardata.com/jobs?
once the job has finished, you’ll receive an email with a link to your job results. clicking on this link will bring you into the mortar web app, where you can download the results from s3. the output is described at the top of the characterize_collection script but as an example you can scroll down the output and find:… user.is_translator 2 false unicode 118806 user.is_translator 2 true unicode 31 user.lang 26 en unicode 114108 user.lang 26 es unicode 3462 user.lang 26 fr unicode 532 user.lang 26 pt unicode 281 user.lang 26 ja unicode 79 user.listed_count 398 0 int 73757 user.listed_count 398 1 int 18518
looking at the values for user.lang - we see that there are 26 unique values for the field in our dataset. the most common was “en” with 114108 occurrences, the next most common was “es” with 3462 occurrences, and so on. to see the full results without running the job you can view the output file here.script 2 - mongodb schema generator
it can be tricky to properly declare mongodb’s highly nested schemas in pig. now, pig is graceful—it can roll without a schema, or with inconsistent, or incorrect schemas. but it’s easier to read and write your pig code if you have a schema because it allows you (and the pig optimizer) to focus on just the relevant data.
so this next script automatically generates a pig schema by examining your mongodb collection. if you don’t need the whole schema, you can easily edit it to keep just the fields you want.
running this script is similar to running the previous one. if you ran the characterize collection script in the past hour, the same cluster you used for that job should still be running. in that case, you can just run:mortar run mongo_schema_generator
if you don’t have a cluster that’s still running, just run the job on a new 4 node cluster like this:mortar run mongo_schema_generator --clustersize 4 script 3 – twitter hourly coffee tweets
using a twitter coffee tweets script (pigscripts/hourly_coffee_tweets.pig), we’re going to demonstrate how we can use a small subset of the fields in our mongodb collection. for our example, we’ll look at how often the word “coffee” is tweeted throughout the day. as with the mongo schema generator script, you can run this job on an existing cluster or start up a new one.next steps
if you already have a mongo instance/cluster based in us-east ec2, the first two example scripts should run on one of your collections with only minor modifications. you’ll just need to: