mrjob 0.4-RC1 (beta)

Python MapReduce framework

Installation

From PyPI:

pip install mrjob

From source:

python setup.py install

A Simple Map Reduce Job

Code for this example and more live in mrjob/examples.

.. code:: python

   """The classic MapReduce job: count the frequency of words.
   """
   from mrjob.job import MRJob
   import re

   WORD_RE = re.compile(r"[\w']+")


   class MRWordFreqCount(MRJob):

       def mapper(self, _, line):
           for word in WORD_RE.findall(line):
               yield (word.lower(), 1)

       def combiner(self, word, counts):
           yield (word, sum(counts))

       def reducer(self, word, counts):
           yield (word, sum(counts))


    if __name__ == '__main__':
        MRWordFreqCount.run()

Try It Out!

# locally
python mrjob/examples/mr_word_freq_count.py README.rst > counts
# on EMR
python mrjob/examples/mr_word_freq_count.py README.rst -r emr > counts
# on your Hadoop cluster
python mrjob/examples/mr_word_freq_count.py README.rst -r hadoop > counts

Setting up EMR on Amazon

• create an Amazon Web Services account
• sign up for Elastic MapReduce
• Get your access and secret keys (click "Security Credentials" on your account page)
• Set the environment variables $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY accordingly

Advanced Configuration

To run in other AWS regions, upload your source tree, run make, and use other advanced mrjob features, you'll need to set up mrjob.conf. mrjob looks for its conf file in:

• The contents of $MRJOB_CONF
• ~/.mrjob.conf
• /etc/mrjob.conf

See the mrjob.conf documentation for more information.

Project Links

• Source code
• Documentation
• Discussion group

Reference

• Hadoop MapReduce

• System Message: WARNING/2 (<string>, line 2); backlink

  Duplicate explicit target name: "elastic mapreduce".

  Elastic MapReduce

More Information

• PyCon 2011 mrjob overview

• System Message: WARNING/2 (<string>, line 2); backlink

  Duplicate explicit target name: "source code".

  Introduction to Recommendations and MapReduce with mrjob (source code)

• Social Graph Analysis Using Elastic MapReduce and PyPy

Thanks to Greg Killion (blind-works.net) for the logo.