ActiveState Code

Recipe 84317: Easy threading with Futures

Although Python's thread syntax is nicer than in many languages, it can still be a pain if all one wants to do is run a time-consuming function in a separate thread, while allowing the main thread to continue uninterrupted. A Future provides a legible and intuitive way to achieve such an end.

Python 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
from threading import *
import copy

class Future:

    def __init__(self,func,*param):
        # Constructor
        self.__done=0
        self.__result=None
        self.__status='working'

        self.__C=Condition()   # Notify on this Condition when result is ready

        # Run the actual function in a separate thread
        self.__T=Thread(target=self.Wrapper,args=(func,param))
        self.__T.setName("FutureThread")
        self.__T.start()

    def __repr__(self):
        return '<Future at '+hex(id(self))+':'+self.__status+'>'

    def __call__(self):
        self.__C.acquire()
        while self.__done==0:
            self.__C.wait()
        self.__C.release()
        # We deepcopy __result to prevent accidental tampering with it.
        a=copy.deepcopy(self.__result)
        return a

    def Wrapper(self, func, param):
        # Run the actual function, and let us housekeep around it
        self.__C.acquire()
        try:
            self.__result=func(*param)
        except:
            self.__result="Exception raised within Future"
        self.__done=1
        self.__status=`self.__result`
        self.__C.notify()
        self.__C.release()

Discussion

To run a function in a separate thread, simply put it in a Future:

>>> A=Future(longRunningFunction, arg1, arg2 ...)

It will continue on its merry way until you need the result of your function. You can read the result by calling the Future like a function, for example:

>>> print A()

If the Future has completed executing, the call returns immediately. If it is still running, then the call blocks until the function completes. The result of the function is stored in the Future, so subsequent calls to it return immediately.

A few caveats: Since one wouldn't expect to be able to change the result of a function, Futures are not meant to be mutable. This is enforced by requiring the Future to be "called", rather than directly reading __result. If desired, stronger enforcement of this rule can be achieved by playing with __getattr__ and __setattr__.

The Future only runs the function once, no matter how many times you read it. You will have to re-create the Future if you want to re-run your function; for example, if the function is sensitive to the time of day.

For more information on Futures, and other useful parallel programming constructs, read Gregory V. Wilson's _Practical Parallel Programming_.

Comments

  1. 1. At 8:48 p.m. on 12 dec 2001, Matthew King said:

    what about exceptions? cool idea. minor nit... if the function being run in a Future generates an exception the Condition is never released...

    maybe a try / finally around the self.__result=func(*param)?

  2. 2. At 1:55 p.m. on 16 apr 2002, David Perry (the author) said:

    Re: what about exceptions? That's a very good point, thanks for pointing it out. A try/finally where you suggested is probably the best way to deal with it. I'll give it a shot.

  3. 3. At 2:14 p.m. on 16 apr 2002, David Perry (the author) said:

    Fixed. And done. I made it a try / except, rather than try / finally, so the Future can report an error message rather than dying silently.

  4. 4. At 5:08 p.m. on 17 apr 2002, H. Krekel said:

    very nice. This is really nice. it's a pity you can't easily stop the function from where you started the future. Sometimes one would certainly like timeouts or other termination conditions.

  5. 5. At 2:55 p.m. on 23 apr 2002, Jerry McRae said:

    Am I done? If a thread takes a REAL long time, one might want to query the thread to see if it is done. Something like this (in the class) should work:

    def isDone(self): return self.__done

    If one stored the starting time, one could query the lenght of time the thread has been running.

    Question. Is there any way that the actual running thread can pass information back to the instance of Future? So I could know it is working on iteration 23 of 100, or some such?

  6. 6. At 11:33 a.m. on 14 aug 2002, Bjorn Pettersen said:

    Exception handling. In my version I added self.__excpt = None in __init__, then in __call__ I added (just before copying the result):

    # an exception was thrown in the thread, re-raise it here.
if self.__excpt: raise self.__excpt

    and finally in Wrapper, I added an extra layer of exception handling:

    try:
    self.__result == func(*param)
except Exception, e: # we got an exception object, save it.
    self.__excpt = e
except:
    self.__result = "Unknown exception raised within Future"

    Assuming the user inherits his exception from Exception (which he is supposed to), this will make futures semi transparent in exception cases, i.e. the exception will be thrown when the user fetches the value of the future.

  7. 7. At 10:07 p.m. on 30 jun 2004, John Costello said:

    Back-communication is simple: just pass the 'self' object into the function in Wrapper, and it will be able to assign to an appropriate property on the Future instance.

  8. 8. At 11:50 p.m. on 5 jan 2005, Anonymous said:

    Maximum number of feeds. Is there a maximum number of feeds this can handle?

  9. 9. At 8:30 a.m. on 24 mar 2005, Graham Horler said:

    Better exceptions. In my version, Wrapper() does this:

    try:
    self.__result = func(*args, **kwargs)
except:
    self.__result = "Exception raised within Future"
    self.__excpt = sys.exc_info()

    and __call__() does this:

    if self.__excpt:
    raise self.__excpt[0], self.__excpt[1], self.__excpt[2]

    This way, all exceptions are caught, regardless of inheriting from Exception, and you even get a nice traceback (with a little gc expense).

  10. 10. At 9:32 p.m. on 2 jun 2009, Brian Quinlan said:

    I'm working on a more complete implementation at: http://code.google.com/p/pythonfutures

Sign in to comment