Popular recipes tagged "processing" but not "batch"http://code.activestate.com/recipes/tags/processing-batch/2016-01-30T00:40:20-08:00ActiveState Code RecipesDecoupled Proxy of a State Bearing Object in a Multiprocessing Environment (Python)
2010-04-08T04:36:05-07:00Mateyuzohttp://code.activestate.com/recipes/users/4172453/http://code.activestate.com/recipes/577180-decoupled-proxy-of-a-state-bearing-object-in-a-mul/
<p style="color: grey">
Python
recipe 577180
by <a href="/recipes/users/4172453/">Mateyuzo</a>
(<a href="/recipes/tags/access/">access</a>, <a href="/recipes/tags/multiprocessing/">multiprocessing</a>, <a href="/recipes/tags/process/">process</a>, <a href="/recipes/tags/processing/">processing</a>, <a href="/recipes/tags/subprocess/">subprocess</a>, <a href="/recipes/tags/synchronization/">synchronization</a>, <a href="/recipes/tags/syncmanager/">syncmanager</a>, <a href="/recipes/tags/threading/">threading</a>, <a href="/recipes/tags/__getattribute__/">__getattribute__</a>).
Revision 2.
</p>
<p>Describes a simple example of the potential for shared Singleton or Borg object methods to be proxy'd by an object that SyncManger can present to requesting subprocesses.</p>
Thread pool with same API as (multi)processing.Pool (Python)
2016-01-30T00:40:20-08:00david decotignyhttp://code.activestate.com/recipes/users/4129454/http://code.activestate.com/recipes/576519-thread-pool-with-same-api-as-multiprocessingpool/
<p style="color: grey">
Python
recipe 576519
by <a href="/recipes/users/4129454/">david decotigny</a>
(<a href="/recipes/tags/multiprocessing/">multiprocessing</a>, <a href="/recipes/tags/pool/">pool</a>, <a href="/recipes/tags/processing/">processing</a>, <a href="/recipes/tags/thread/">thread</a>).
Revision 10.
</p>
<p>There are probably <write your guess here>s of recipes presenting how to implement a pool of threads. Now that multiprocessing is becoming mainstream, this recipe takes multiprocessing.Pool as a model and re-implements it entirely with threads. Even the comments should look familiar... This recipe also adds 2 new methods: imap_async() and imap_unordered_async().</p>
CallPipe: Call the methods of an object in a remote process (Python)
2008-09-17T17:33:36-07:00david decotignyhttp://code.activestate.com/recipes/users/4129454/http://code.activestate.com/recipes/576509-callpipe-call-the-methods-of-an-object-in-a-remote/
<p style="color: grey">
Python
recipe 576509
by <a href="/recipes/users/4129454/">david decotigny</a>
(<a href="/recipes/tags/call/">call</a>, <a href="/recipes/tags/multiprocessing/">multiprocessing</a>, <a href="/recipes/tags/pipe/">pipe</a>, <a href="/recipes/tags/processing/">processing</a>, <a href="/recipes/tags/remote/">remote</a>).
Revision 6.
</p>
<p>I am process A and I own object Obj. Now I decide to fork(): process B is born.</p>
<p>How do I do to make process B call the methods of A's object Obj (and not the methods of its own copy of Obj...) ?</p>
<p>This is what this module does: you create the "CallPipe" for Obj prior to fork()ing, and then process B can call any method of A's object Obj through it.</p>
processing.Pool variation which allows multiple threads to send the same requests without incurring duplicate processing (Python)
2008-09-17T17:01:21-07:00david decotignyhttp://code.activestate.com/recipes/users/4129454/http://code.activestate.com/recipes/576462-processingpool-variation-which-allows-multiple-thr/
<p style="color: grey">
Python
recipe 576462
by <a href="/recipes/users/4129454/">david decotigny</a>
(<a href="/recipes/tags/map/">map</a>, <a href="/recipes/tags/parallel/">parallel</a>, <a href="/recipes/tags/pool/">pool</a>, <a href="/recipes/tags/processing/">processing</a>, <a href="/recipes/tags/threads/">threads</a>).
Revision 3.
</p>
<p>processing.Pool (<a href="http://pypi.python.org/pypi/processing" rel="nofollow">http://pypi.python.org/pypi/processing</a>) is a nice tool to "parallelize" map() on multiple CPUs.
However, imagine you have X threads which send the same request Pool.map(getNthPrimeNumber, [100000, 10000000, 10000]) at (almost) the same time. Obviously, you don't want to compute X times getNthPrimeNumber for 100000, 10000000, 10000... unless you have 3.X processors available. You would like one thread to submit the 3 requests, and then the X-1 others would notice that the requests have already been submitted and will then just wait for the result.
This is what this code is about: a kind of "trensient memoize" for processing.Pool::imap().</p>