Popular recipes by david decotigny

Thread pool with same API as (multi)processing.Pool (Python)

2016-01-30T00:40:20-08:00

Python recipe 576519 by david decotigny (multiprocessing, pool, processing, thread). Revision 10.

There are probably <write your guess here>s of recipes presenting how to implement a pool of threads. Now that multiprocessing is becoming mainstream, this recipe takes multiprocessing.Pool as a model and re-implements it entirely with threads. Even the comments should look familiar... This recipe also adds 2 new methods: imap_async() and imap_unordered_async().

Multithreaded Multiplexer/Demultiplexer to allow multiple transactions to be performed in parallel on a single channel (Python)

2008-10-04T00:12:00-07:00

Python recipe 576520 by david decotigny (demultiplexer, multiplexer, thread, transaction). Revision 8.

Imagine you have one wire and you want to send a message from a machine A to another machine B, and wait for an answer from B (a "request/response transaction" in the following). Now imagine you have many threads on A which can make this kind of transaction "simultaneously", then you have to be cautious, otherwise you might get the wrong responses for your transactions. One solution is to serialize the transactions: the threads on A will have to wait for their turn to make their request/response transaction.

Another solution is to map a thread requesting a transaction on A, to a thread handling it on B. This allows for several transactions to be in flight at the same time on the same wire. This is what this module does: on A lives a multiplexer and threads call its transaction() method to initiate the request/response. On B, process_transaction() of a demultiplexer will get called by worker threads. All this in a multi-threaded way on a single wire. The module takes care of managing the interleaving requests/responses.

Track new/unreclaimed objects between 2 points in the code (Python)

2008-10-03T12:14:52-07:00

Python recipe 576523 by david decotigny (cycle, garbage, leak, memory, reclaim). Revision 3.

This module provides 3 ways of detecting which objects have been allocated (methods 1 and 3) or became un-reclaimable (method 2) between 2 points in the code. It can be very useful to detect memory leaks (eg. cycles involving objects with a __del__ method).

Convert datetime.datetime objects to/from Boost.python posix time (Python)

2008-09-17T23:55:23-07:00

Python recipe 576395 by david decotigny (boost). Revision 10.

This recipe allows to transparently convert python's datetime.datetime objects to and from boost's boost::posix_time::ptime objects.

CallPipe: Call the methods of an object in a remote process (Python)

2008-09-17T17:33:36-07:00

Python recipe 576509 by david decotigny (call, multiprocessing, pipe, processing, remote). Revision 6.

I am process A and I own object Obj. Now I decide to fork(): process B is born.

How do I do to make process B call the methods of A's object Obj (and not the methods of its own copy of Obj...) ?

This is what this module does: you create the "CallPipe" for Obj prior to fork()ing, and then process B can call any method of A's object Obj through it.

processing.Pool variation which allows multiple threads to send the same requests without incurring duplicate processing (Python)

2008-09-17T17:01:21-07:00

Python recipe 576462 by david decotigny (map, parallel, pool, processing, threads). Revision 3.

processing.Pool (http://pypi.python.org/pypi/processing) is a nice tool to "parallelize" map() on multiple CPUs. However, imagine you have X threads which send the same request Pool.map(getNthPrimeNumber, [100000, 10000000, 10000]) at (almost) the same time. Obviously, you don't want to compute X times getNthPrimeNumber for 100000, 10000000, 10000... unless you have 3.X processors available. You would like one thread to submit the 3 requests, and then the X-1 others would notice that the requests have already been submitted and will then just wait for the result. This is what this code is about: a kind of "trensient memoize" for processing.Pool::imap().

JSON RPC Server and client (Python)

2008-04-01T09:50:49-07:00

Python recipe 552751 by david decotigny (network).

This recipe shows how to create JSON RPC client and server objects. The aim is to mimic the standard python XML-RPC API both on the client and server sides, but using JSON marshalling. It depends on cjson (http://pypi.python.org/pypi/python-cjson) for the encoding/decoding of JSON data. This recipe tries to reuse the code of XML-RPC as much as possible.

Create a restricted python function from a string (Python)

2008-08-05T22:15:18-07:00

Python recipe 550804 by david decotigny (programs). Revision 2.

The createFunction(sourceCode) below returns a python function that executes the given sourceCode (a string containing python code). The function, being a real python function, doesn't incur any overhead compared to any normal python function. And its environment is controlled: by default only safe operations are permitted (ie. map, reduce, filter, list, etc. ; others like import, open, close, eval, etc. are forbidden by default). But it is possible to extend this environment.