Latest recipes by Staffan Malmgren http://code.activestate.com/recipes/users/1149346/new/2006-04-14T15:59:41-07:00ActiveState Code RecipesCaching and throttling for urllib2 (Python)
2006-04-14T15:59:41-07:00Staffan Malmgrenhttp://code.activestate.com/recipes/users/1149346/http://code.activestate.com/recipes/491261-caching-and-throttling-for-urllib2/
<p style="color: grey">
Python
recipe 491261
by <a href="/recipes/users/1149346/">Staffan Malmgren</a>
(<a href="/recipes/tags/web/">web</a>).
</p>
<p>This code implements a cache (CacheHandler) and a throttling mechanism (ThrottlingProcessor) for urllib2. By using them, you can ensure that subsequent GET requests for the same URL returns a cached copy instead of causing a roundtrip to the remote server, and/or that subsequent requests to a server are paused for a couple of seconds to avoid overloading it. The test code at the end explains all there is to it.</p>