Twisted FAQs clearly state that "deferreds do not magically convert blocking code into non-blocking code". So, how do you magically convert blocking code into non-blocking code?
This recipe is the solution!
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
## version 1.1, changed according to the suggestions in the comments from twisted.internet import reactor, defer from twisted.python import threadable; threadable.init(1) import sys, time ## the wrong way def callInThread(func, *args): """Takes a blocking function an converts it into a deferred-valued function running in a separate thread. """ de = defer.Deferred() de.addCallback(func) reactor.callInThread(de.callback, *args) return de deferred = callInThread.__get__ # decorator associated to callInThread # the right way from twisted.internet.threads import deferToThread deferred = deferToThread.__get__ ## example code def print_(result): print result def running(): "Prints a few dots on stdout while the reactor is running." sys.stdout.write("."); sys.stdout.flush() reactor.callLater(.1, running) @deferred def sleep(sec): "A blocking function magically converted in a non-blocking one." print 'start sleep %s' % sec time.sleep(sec) print '\nend sleep %s' % sec return "ok" if __name__ == "__main__": sleep(2).addBoth(print_) reactor.callLater(.1, running) reactor.callLater(3, reactor.stop) reactor.run()
How to make blocking code non-blocking is the obvious question for everybody using Twisted, but the Twisted documentation does not make easy to find the solution :-(
The trick is to run the blocking function in a separate thread. Here
all the magic is performed by the decorator,
deferred, which converts
sleep, a blocking function, into a deferred function i.e. a
non-blocking function that returns a deferred object. The code for
callInThread is clear, and the
.__get__ trick converts
callInThread in a one-argument function returning a closure,
i.e. an object suitable to be used as a decorator. I have seen this
trick in Alex Martelli's lectures at PyCon 2005. If you are confused
by it, you should read Raymond Hettinger essay on descriptors
In short: every time you have a blocking function in your code, wrap
it with the
deferred decorator and live happy!