How to kick off a slow process without waiting around for the result. The process is run in the background until the value is actually needed at which time it blocks until the value is ready.
Download
Copy to clipboard1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | from threading import Thread
LARGE = 100000
class Future:
def __init__(self, f):
self.__f = f
self.__thread = None
self.__exception = None
def __call__(self, *args, **kw):
self.__thread = Thread(target=self.runfunc, args=args, kwargs=kw)
self.__thread.start()
return self
def runfunc(self, *args, **kw):
try:
self.__val = self.__f(*args, **kw)
except Exception, e:
self.__exception = e
def __coerce__(self, other):
return (self.__val, other)
def __getattr__(self, name):
self.__thread.join()
if self.__exception is not None:
raise self.__exception
out = getattr(self.__val, name)
return out
def future(f):
return Future(f)
@future
def long_running():
print 'Starting long_running()'
out = 0
for i in range(LARGE):
out += 1
print 'Done with long_running()'
return out
@future
def long_str():
print 'Starting long_running()'
out = 0
for i in range(LARGE):
out += 1
print 'Done with long_running()'
return str(out)
@future
def raises():
raise Exception('This is a test')
print 'Getting value result from long running'
v = long_running()
print 'Doing other stuff ...'
print 'Use result of long_running()'
print v, LARGE
print v + v
print v
print xrange(v)
v = long_str()
print v[0]
print v * 2
print v.split
v = raises()
print v
# RESULT
'''
>>> import deferred
Getting value result from long running
Starting long_running()
Doing other stuff ...
Use result of long_running()
Done with long_running()
100000 100000
200000
100000
xrange(100000)
Starting long_running()
Done with long_running()
1
100000100000
<built-in method split of str object at 0x00B4E5C0>
Traceback (most recent call last):
File "<pyshell#1>", line 1, in -toplevel-
import deferred
File "C:\Python\Python24\deferred.py", line 72, in -toplevel-
print v
File "C:\Python\Python24\deferred.py", line 27, in __getattr__
raise self.__exception
Exception: This is a test
>>>
'''
|
Discussion: The Twisted Team introduced the deferred object at PyCon 2002. As I remember, the idea was to be able to kick off slow process, like a remote DB access, then make use of the interim time before the process finishes.
This recipe IS TOTALLY DIFFERENT, BUT seems to accomplish A SIMILAR result without having to learn the deferred interface. As far as I can tell you can treat the not_deferred result of the function just like you would the actual result of an equivalent un-not_deferred function. Give it a whirl and see if you can break it.
Note: Any actual implementation should probably utilize a thread pool!
What about Exceptions? This recipe:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/84317
does the same thing, but also handles exceptions in the thread. Perhaps some of those ideas can be used here?
Exceptions Caught. Deferred now catches exceptions and re-throws them at an appropriate time.
Thanks for the input and pointing out the similar recipe. If any advantage is gained here it is the fact that client code need not be concerned with the tricky business happening behind the scenes. The client calls the function, and treats the result as though it was an ordinary value.
Missing the point completely. The idea behind Deferreds in Twisted is to provide a sensible interface for asynchronous return values, i.e. those which you have to allow your code to return to a main-loop to retrieve. In other words, it is an interface for retrieving results without using threads.
I cannot see a place where this code example would be useful in practice, but even if it is, it is extremely misleading to claim that this code is in any way related to the concept of Deferreds in Twisted.
Please re-name the class so as to avoid such confusion.
Misleading description. Despite the title, this code _does_ wait around for the result; thread.join blocks until the thread produces a value. This is completely different from the way Deferred works in Twisted, where you register callbacks instead of waiting.
Not deferred. As several have pointed out this recipe is does not implement deferred. I have made several changes to the code to avoid the mayham that was narrowly avoided.
Thanks for all the watchful eyes!
Not deferred. As several have pointed out this recipe does not implement deferred. I have made several changes to the code to avoid the mayhem that was narrowly avoided.
Thanks for all the watchful eyes!
It may not be a Twisted Deferred, but it is a deferred result. A common name for this type of object is a "Future" object (the object isn't 'real' when the function returns, but it will be real in the future when you need it).
Future. Thanks for the comment. Thats a good name for it that wont offend any one.
Justin
There's a good description of various kinds of futures in the "Alice" tour: http://www.ps.uni-sb.de/alice/manual/tour.html
Other recipe about futures. Excellent recipe.
But note that recipe num 84317 is also about futures.
In there the usage is more explicit.
Also the category for this recipe should really be Threads like there