Welcome, guest | Sign In | My Account | Store | Cart

How to kick off a slow process without waiting around for the result. The process is run in the background until the value is actually needed at which time it blocks until the value is ready.

Python, 101 lines
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
from threading import Thread
LARGE = 100000

class Future:
    def __init__(self, f):
        self.__f = f
        self.__thread = None
        self.__exception = None
        
    def __call__(self, *args, **kw):
        self.__thread = Thread(target=self.runfunc, args=args, kwargs=kw)
        self.__thread.start()
        return self
    
    def runfunc(self, *args, **kw):
        try:
            self.__val = self.__f(*args, **kw)
        except Exception, e:
            self.__exception = e

    def __coerce__(self, other):
        return (self.__val, other)
        
    def __getattr__(self, name):
        self.__thread.join()
        if self.__exception is not None:
            raise self.__exception
        out = getattr(self.__val, name)
        return out

def future(f):
    return Future(f)

@future
def long_running():
    print 'Starting long_running()'
    out = 0
    for i in range(LARGE):
        out += 1
    print 'Done with long_running()'
    return out

@future
def long_str():
    print 'Starting long_running()'
    out = 0
    for i in range(LARGE):
        out += 1
    print 'Done with long_running()'
    return str(out)

@future
def raises():
    raise Exception('This is a test')

print 'Getting value result from long running'
v = long_running()

print 'Doing other stuff ...'
print 'Use result of long_running()'
print v, LARGE
print v + v
print v
print xrange(v)

v = long_str()
print v[0]
print v * 2
print v.split

v = raises()
print v

# RESULT
'''
>>> import deferred
Getting value result from long running
Starting long_running()
Doing other stuff ...
Use result of long_running()
Done with long_running()
100000 100000
200000
100000
xrange(100000)
Starting long_running()
Done with long_running()
1
100000100000
<built-in method split of str object at 0x00B4E5C0>

Traceback (most recent call last):
  File "<pyshell#1>", line 1, in -toplevel-
    import deferred
  File "C:\Python\Python24\deferred.py", line 72, in -toplevel-
    print v
  File "C:\Python\Python24\deferred.py", line 27, in __getattr__
    raise self.__exception
Exception: This is a test
>>> 
'''

Discussion: The Twisted Team introduced the deferred object at PyCon 2002. As I remember, the idea was to be able to kick off slow process, like a remote DB access, then make use of the interim time before the process finishes.

This recipe IS TOTALLY DIFFERENT, BUT seems to accomplish A SIMILAR result without having to learn the deferred interface. As far as I can tell you can treat the not_deferred result of the function just like you would the actual result of an equivalent un-not_deferred function. Give it a whirl and see if you can break it.

Note: Any actual implementation should probably utilize a thread pool!

10 comments

S W 16 years, 12 months ago  # | flag

What about Exceptions? This recipe:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/84317

does the same thing, but also handles exceptions in the thread. Perhaps some of those ideas can be used here?

Justin Shaw (author) 16 years, 12 months ago  # | flag

Exceptions Caught. Deferred now catches exceptions and re-throws them at an appropriate time.

Thanks for the input and pointing out the similar recipe. If any advantage is gained here it is the fact that client code need not be concerned with the tricky business happening behind the scenes. The client calls the function, and treats the result as though it was an ordinary value.

Glyph Lefkowitz 16 years, 12 months ago  # | flag

Missing the point completely. The idea behind Deferreds in Twisted is to provide a sensible interface for asynchronous return values, i.e. those which you have to allow your code to return to a main-loop to retrieve. In other words, it is an interface for retrieving results without using threads.

I cannot see a place where this code example would be useful in practice, but even if it is, it is extremely misleading to claim that this code is in any way related to the concept of Deferreds in Twisted.

Please re-name the class so as to avoid such confusion.

Allen Short 16 years, 12 months ago  # | flag

Misleading description. Despite the title, this code _does_ wait around for the result; thread.join blocks until the thread produces a value. This is completely different from the way Deferred works in Twisted, where you register callbacks instead of waiting.

Justin Shaw (author) 16 years, 12 months ago  # | flag

Not deferred. As several have pointed out this recipe is does not implement deferred. I have made several changes to the code to avoid the mayham that was narrowly avoided.

Thanks for all the watchful eyes!

Justin Shaw (author) 16 years, 12 months ago  # | flag

Not deferred. As several have pointed out this recipe does not implement deferred. I have made several changes to the code to avoid the mayhem that was narrowly avoided.

Thanks for all the watchful eyes!

Nick Coghlan 16 years, 12 months ago  # | flag

It may not be a Twisted Deferred, but it is a deferred result. A common name for this type of object is a "Future" object (the object isn't 'real' when the function returns, but it will be real in the future when you need it).

Justin Shaw (author) 16 years, 11 months ago  # | flag

Future. Thanks for the comment. Thats a good name for it that wont offend any one.

Justin

Rod Hyde 16 years, 11 months ago  # | flag

There's a good description of various kinds of futures in the "Alice" tour: http://www.ps.uni-sb.de/alice/manual/tour.html

Yair Chuchem 16 years, 9 months ago  # | flag

Other recipe about futures. Excellent recipe.

But note that recipe num 84317 is also about futures.

In there the usage is more explicit.

Also the category for this recipe should really be Threads like there

Created by Justin Shaw on Sun, 5 Dec 2004 (PSF)
Python recipes (4591)
Justin Shaw's recipes (11)

Required Modules

Other Information and Tasks