oyster is a service for tracking regularly-accessed pages, a sort of proactive cache.
Oyster intends to provide a command line client for interacting with the list of tracked documents and web frontend for viewing the status and retrieving data. Behind the scenes it uses a celery queue to manage the documents it is tasked with keeping up to date.
oyster was created by James Turk for Sunlight Labs.
Source is available via GitHub
oyster is available on PyPI: oyster.
The recommended way to install oyster is to simply pip install oyster
- python 2.7
- mongodb 2.0
- pymongo 2.0
- scrapelib 0.5+
- Run celeryd with beat celeryd -B --config=oyster.celeryconfig
- Run oyster HTTP portal python oyster/web.py
- Use oyster.client.Client to add new documents & query the store