Latest recipes tagged "processing"

Zero (Batch) Programs (Python)

2012-07-10T12:37:37-07:00

Python recipe 578205 by Stephen Chappell (archive, batch, old, processing, utility). Revision 2.

Having written many programs that work with groups of files, the zero programs were written based on a simple batch engine in the zero utility. All of the other programs import the first program to take advantage of its batch processor while supplementing there own functionality in place of zeroing out file data. This is committed for archival to be run under Python 2.5 or later versions.

Decoupled Proxy of a State Bearing Object in a Multiprocessing Environment (Python)

2010-04-08T04:36:05-07:00

Python recipe 577180 by Mateyuzo (access, multiprocessing, process, processing, subprocess, synchronization, syncmanager, threading, __getattribute__). Revision 2.

Describes a simple example of the potential for shared Singleton or Borg object methods to be proxy'd by an object that SyncManger can present to requesting subprocesses.

Thread pool with same API as (multi)processing.Pool (Python)

2016-01-30T00:40:20-08:00

Python recipe 576519 by david decotigny (multiprocessing, pool, processing, thread). Revision 10.

There are probably <write your guess here>s of recipes presenting how to implement a pool of threads. Now that multiprocessing is becoming mainstream, this recipe takes multiprocessing.Pool as a model and re-implements it entirely with threads. Even the comments should look familiar... This recipe also adds 2 new methods: imap_async() and imap_unordered_async().

CallPipe: Call the methods of an object in a remote process (Python)

2008-09-17T17:33:36-07:00

Python recipe 576509 by david decotigny (call, multiprocessing, pipe, processing, remote). Revision 6.

I am process A and I own object Obj. Now I decide to fork(): process B is born.

How do I do to make process B call the methods of A's object Obj (and not the methods of its own copy of Obj...) ?

This is what this module does: you create the "CallPipe" for Obj prior to fork()ing, and then process B can call any method of A's object Obj through it.

processing.Pool variation which allows multiple threads to send the same requests without incurring duplicate processing (Python)

2008-09-17T17:01:21-07:00

Python recipe 576462 by david decotigny (map, parallel, pool, processing, threads). Revision 3.

processing.Pool (http://pypi.python.org/pypi/processing) is a nice tool to "parallelize" map() on multiple CPUs. However, imagine you have X threads which send the same request Pool.map(getNthPrimeNumber, [100000, 10000000, 10000]) at (almost) the same time. Obviously, you don't want to compute X times getNthPrimeNumber for 100000, 10000000, 10000... unless you have 3.X processors available. You would like one thread to submit the 3 requests, and then the X-1 others would notice that the requests have already been submitted and will then just wait for the result. This is what this code is about: a kind of "trensient memoize" for processing.Pool::imap().