Simple way to execute multiple process in parallel (Python)

2010-08-31T00:11:00-07:00

Python recipe 577376 by Benjamin Sergeant (multiprocessing, process). Revision 4.

Does not require multiprocessing, easy to hack, maybe not optimal but did the job for a make like tool I wrote.

Thread pool with same API as (multi)processing.Pool (Python)

2016-01-30T00:40:20-08:00

Python recipe 576519 by david decotigny (multiprocessing, pool, processing, thread). Revision 10.

There are probably <write your guess here>s of recipes presenting how to implement a pool of threads. Now that multiprocessing is becoming mainstream, this recipe takes multiprocessing.Pool as a model and re-implements it entirely with threads. Even the comments should look familiar... This recipe also adds 2 new methods: imap_async() and imap_unordered_async().

Blocking, Polling, Pipes, Queues in multiprocessing and how they are affected by the OS's time-slicing schedule (Python)

2010-05-08T18:18:01-07:00

Python recipe 577223 by Kaushik Ghose (events, multiprocessing, multitasking, polling, timeslice).

This short code demonstrates how blocking calls to a Queue, while consuming less CPU, are limited in their response time by the minimum time slice the OS is willing to allocate (typically 10ms for Mac OS X and Linux). Non-blocking calls to Pipe, using poll() to check if there is data, on the other hand, give us millisecond or less response times, though they consume more CPU. In this respect doing a blocking call to a CPU is no different than adding sleep(.01) statements to a polling loop. In a way, if you execute a sleep(.01) only when you have no events in your poll you will be more efficient than if you had a blocking call pull events off your Queue one by one - because each call to Queue.get() consumes a time-slice, whereas the sleep(.01) only occurs once.

timeout decorator (with multiprocessing) (Python)

2011-08-23T05:25:09-07:00

Python recipe 577853 by matt harrison (decorator, decorators, multiprocessing, timeout).

Given that threads shouldn't be killed, multiprocessing is one mechanism for limiting time on a potentially long running computation.

CallPipe: Call the methods of an object in a remote process (Python)

2008-09-17T17:33:36-07:00

Python recipe 576509 by david decotigny (call, multiprocessing, pipe, processing, remote). Revision 6.

I am process A and I own object Obj. Now I decide to fork(): process B is born.

How do I do to make process B call the methods of A's object Obj (and not the methods of its own copy of Obj...) ?

This is what this module does: you create the "CallPipe" for Obj prior to fork()ing, and then process B can call any method of A's object Obj through it.

A Python chat program using process intercommunication (Python)

2012-09-12T10:02:21-07:00

Python recipe 578260 by Chaobin Tang (唐超斌) (multiprocessing).

This demo chat program, at some aspect, mimic the way that Erlang makes a distributable application. Because it utilizes process, the CPU cores are used as much as possible.

NondurableLogger class for use with concurrent.futures.ProcessPoolExecutor's submit and map methods (Python)

2014-02-10T17:50:59-08:00

Python recipe 578827 by Peter Santoro (logging, multiprocessing). Revision 2.

I needed a simple logging solution that I could use with with concurrent.futures.ProcessPoolExecutor and this is my initial recipe.

Multiprocessing import wrapper (Python)

2011-08-23T22:11:42-07:00

Python recipe 577856 by Matt Keranen (database, etl, import, multiprocessing). Revision 2.

A script used to launch multiple import scripts using the multiprocessing module. Developed to parallelize loading of multiple log files into a database for aggregate analysis

Parallelize your shell commands (Bash)

2010-04-02T00:14:38-07:00

Bash recipe 577171 by Kevin L. Sitze (fifo, multiprocessing, parallel). Revision 2.

This script is used to processes a batch job of commands in parallel. The script dispatches new commands up to a user specified limit, each generated from a template provided on the command line using arguments taken from STDIN. The basic combining semantics are similar to "xargs -1", though support for multiple arguments and parallel processing of commands are provided.

Available disk space monitoring for concurrent writes (Python)

2010-03-25T01:10:14-07:00

Python recipe 577140 by Mateyuzo (files, multiprocessing, stat, writing). Revision 5.

A DiskSpaceProctor is registered with a SyncManager to be shared by forked processes and threads. This object will check the available free space and add up filesz of concurrent writes. If there is not enough space for the requesting write, an exception is raised.

Decoupled Proxy of a State Bearing Object in a Multiprocessing Environment (Python)

2010-04-08T04:36:05-07:00

Python recipe 577180 by Mateyuzo (access, multiprocessing, process, processing, subprocess, synchronization, syncmanager, threading, __getattribute__). Revision 2.

Describes a simple example of the potential for shared Singleton or Borg object methods to be proxy'd by an object that SyncManger can present to requesting subprocesses.

Most viewed recipes tagged "multiprocessing"