I like how gevent is making async code to look like sync but non blocking without all the ugly callbacks. I tried doing that with threads and object proxy (I found great one at: http://pypi.python.org/pypi/ProxyTypes written by Phillip J. Eby, and this is where the actual magic happens).
For every function that is decorated it returns a proxy and the io call (or anything else) won't block until the value is actually needed. (should be some pools and args pickling there, to make it more like message passing but I didn't want to fuzzy the example) To use it as actor model, I guess it would require to queue requests to decorated object's methods and create a single thread to process them an in LazyProxy callback set q.get() instead of t.join()
Download
Copy to clipboard1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | from threading import Thread, Lock
from peak.util.proxies import LazyProxy
def function_thread(f, lock=None):
def args_wrapper(*args, **kwargs):
class FunctionRunner(Thread):
def run(self):
if lock is not None: lock.acquire()
try:
self.result = f(*args, **kwargs)
finally:
if lock is not None: lock.release()
t = FunctionRunner()
t.start()
def callback():
t.join()
return t.result
proxy = LazyProxy(callback)
return proxy
return args_wrapper
from time import sleep
@function_thread
def power(a, b):
print 'looong power operation', a
sleep(a)
return a ** b
result10_10 = power(1, 10)
result20_20 = power(2, 20)
result30_30 = power(3, 30)
result40_40 = power(4, 40)
sleep(1)
print '\nresult 1**10 = ', result10_10,\
'\nresult 2**20 = ', result20_20,\
'\nresult 3**30 = ', result30_30,\
'\nresult 4**40 = ', result40_40, '\n\n'
##############################
import socket
urls = ['www.google.com', 'www.example.com',
'www.python.org', 'code.activestate.com']
def gethostbyname(*args, **kwargs):
print 'sleeping...',
sleep(2) # makes gethostbyname take a looong time
return socket.gethostbyname(*args, **kwargs)
##############################
gethostbyname_threaded = function_thread(gethostbyname)
results = []
for url in urls:
results.append(gethostbyname_threaded(url))
print 'url appended', url
print '\ndoing some other stuff'
sleep(2)
print '\npar results =', results
##############################
gethostbyname_locked = function_thread(gethostbyname, lock=Lock())
results = []
for url in urls:
results.append(gethostbyname_locked(url))
print '\nseq results =', results
|
I really like how gevent tries to make everything looking as sync code without callbacks like in the first example on the page:
>>> urls = ['www.google.com', 'www.example.com', 'www.python.org']
>>> jobs = [gevent.spawn(socket.gethostbyname, url) for url in urls]
>>> gevent.joinall(jobs, timeout=2)
>>> [job.value for job in jobs]
['74.125.79.106', '208.77.188.166', '82.94.164.162']
So basically this is what tried to do, to wrap every blocking function, make thread, and hope it will block a little less now, and also user is free from all nasty forking, spawning and joining.
I also tried, once the thread finishes to substitute the proxy itself with the actual result with: __dict__, __class__ = result.__dict__, result.__class__ But I didn't know how to synchronize that with other threads that might have been using proxy at the moment and there is also a problem with __slots__ that I don't know how to solve.
