This is a simple function that runs another function in a different process by forking a new process which runs the function and waiting for the result in the parent. This can be useful for releasing resources used by the function such as memory.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
#!/usr/bin/env python from __future__ import with_statement import os, cPickle def run_in_separate_process(func, *args, **kwds): pread, pwrite = os.pipe() pid = os.fork() if pid > 0: os.close(pwrite) with os.fdopen(pread, 'rb') as f: status, result = cPickle.load(f) os.waitpid(pid, 0) if status == 0: return result else: raise result else: os.close(pread) try: result = func(*args, **kwds) status = 0 except Exception, exc: result = exc status = 1 with os.fdopen(pwrite, 'wb') as f: try: cPickle.dump((status,result), f, cPickle.HIGHEST_PROTOCOL) except cPickle.PicklingError, exc: cPickle.dump((2,exc), f, cPickle.HIGHEST_PROTOCOL) os._exit(0) #an example of use def treble(x): return 3 * x def main(): #calling directly print treble(4) #calling in separate process print run_in_separate_process(treble, 4)
Frequently, one might write code such like:
for x in alist: result = do_work(params)
where do_work consumes a lot of memory which is not useful for the lifetime of the program. I actually wrote this function when I was doing large amounts of computations (using numpy) and using large number of temporary arrays. A good way of actually reclaiming the memory is forking a child process, doing the computation in the child process, and returning the results to the parent. This pattern was mentioned in .
The function run_in_separate_process encodes this pattern. It handles exceptions as well, partially. Basically the child process returns a status code as well as a result. If the status is 0, then the function returned successfully and its result is returned. If the status is 1, then the function raised an exception, which will be raised in the parent. If the status is 2, then the function has returned successfully but the result is not picklable, an exception is raised. Exceptions such as SystemExit and KeyboardInterrupt in the child are not checked. As the function stands, it will result in an EOFError in the parent.
This function works on Linux and Mac OS X. It should work on all "reasonable" systems.
Thanks to Alex Martelli for confirming the basic approach and supplying the code for pipe communication. His message and the (short) thread  contain more explanation.
As it stands it works on Python 2.5 although there is nothing specific about 2.5 other than the use of the with statement. If that is rewritten to use normal exception handling is should work on much earlier versions, but I didn't test it.