Welcome, guest | Sign In | My Account | Store | Cart

Notice! PyPM is being replaced with the ActiveState Platform, which enhances PyPM’s build and deploy capabilities. Create your free Platform account to download ActivePython or customize Python with the packages you require and get automatic updates.

Download
ActivePython

slurp is unavailable in PyPM, because there aren't any builds for it in the package repositories. Click the linked icons to find out why.

 Python 2.7Python 3.2Python 3.3
Windows (32-bit)
Windows (64-bit)
Mac OS X (10.5+)
Linux (32-bit)
Linux (64-bit)
 
Links
Author
=====
slurp
=====

.. image:: https://secure.travis-ci.org/bninja/slurp.png?branch=master

Slurp iterates over "entries" in log files (sources), parsed them into
something structured and passes them along to something else (sinks). A log
file is something that:

- is created
- has strings appended to it
- is then possibly deleted.

If a file does not conform to this lifestyle it is not suitable for use with
slurp.

In the slurp world files are mapped to consumers which are python dictionaries
describing:

- what files are associated with the consumer
- how to identify raw "entry" strings in them
- how to parse those "entries" to something structured
- where to send those parsed entries

The motivating use-case for slurp is feeding entries streamed to centralized
syslog spool(s) to elastic search and other data mining tools.

Dependencies
------------

- `Python `_ >= 2.5, < 3.0
- `pyinotify `_ >= 0.9.3
- `lockfile `_  >= 1.9
- `python-daemon `_ >= 1.5

Install
-------

Simply::

    $ pip install slurp
    
or if you prefer::
    
    $ easy_install slurp

Usage
=====

Slurp has both programming and command-line interfaces.

To use the programming interface import it and read doc strings::

    $ python
    >>> import slurp

To use the command-line interface run the slurp script::

    $ slurp --help
    Usage: 
    slurp s|seed path-1 .. path-n [options]
    slurp m|monitor path-1 .. path-n [options]
    slurp e|eat path-1 .. path-n [options]
    
    Options:
      -h, --help            show this help message and exit
      -s STATE_PATH, --state-path=STATE_PATH
      -c CONSUMERS, --consumer=CONSUMERS
      -l LOG_LEVEL, --log-level=LOG_LEVEL
      --enable-syslog       
      --disable-stderrlog   
      -d, --daemonize       
      --disable-locking     
      --lock-timeout=LOCK_TIMEOUT
      --disable-tracking    
      --pid-file=PID_FILE   
      --sink=SINK           
      --batch-size=BATCH_SIZE

Another common use case is to run the slurp script as a monitor daemon. See
extras/slurp.init for an example init script.

Seed
----

Slurp does what it does using three functions: seed, eat and monitor. Seed is
used to initialize offset tracking information for files. These offsets tell
slurp where to resume eating from within the file. This is automatically done
by monitor.

Eat
---

Eat tells slurp to consume any newly added entries appended to tracked files. 

Monitor
-------

Monitor sets up a watch on files and directories and consumes any newly added
entries appended in response to change events trigger by the watch. Slurp uses
pyinotify to watch.

Examples
========

Check it out::

    $ cd ~/code
    $ git checkout git://github.com/bninja/slurp.git
    $ cd slurp
    $ mkvirtualenv slurp
    (slurp)$ python setup.py develop


And run the example::

    $ workon slurp
    (slurp)$ cd ~/code/slurp/extras
    (slurp)$ slurp eat access.log -c example.py -l debug --disable-locking --disable-tracking


.. :changelog:

History
-------

0.2 (2012-05-28)
++++++++++++++++++

* Hope you like it.

0.1 (2012-01-01)
++++++++++++++++++

* Its alive!

Subscribe to package updates

What does the lock icon mean?

Builds marked with a lock icon are only available via PyPM to users with a current ActivePython Business Edition subscription.

Need custom builds or support?

ActivePython Enterprise Edition guarantees priority access to technical support, indemnification, expert consulting and quality-assured language builds.

Plan on re-distributing ActivePython?

Get re-distribution rights and eliminate legal risks with ActivePython OEM Edition.