The SpawnedGenerator class, initialised with a generator, will run that generator in a separate thread and either yield successive values obtained from it (when called) or let you iterate itself to get the same results. It's mainly useful for tasks which make blocking OS calls, e.g. traversing directory structures. The queue size may be specified to limit how far ahead of the main task the spawned generator can get.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
from __future__ import generators import threading class SpawnedGenerator(threading.Thread): "Class to spawn a generator." def __init__(self, generator, queueSize=0): "Initialise the spawned generator from a generator." threading.Thread.__init__(self) self.generator = generator self.queueSize = queueSize def stop(self): "Request a stop." self.stopRequested = 1 def run(self): "Called in a separate thread by Thread.start()." queue = self.queue try: it = iter(self.generator) while 1: next = it.next() queue.put((1, next)) # will raise StopIteration if self.stopRequested: raise StopIteration, "stop requested" except: queue.put((0, sys.exc_info())) def __call__(self): "Yield results obtained from the generator." self.queue = queue = Queue.Queue(self.queueSize) self.stopRequested = 0 self.start() # run() will stuff the queue in a separate Thread keepGoing, item = queue.get() while keepGoing: yield item keepGoing, item = queue.get() # if keepGoing is false, item is exc_info() result self.exc_info = item # stash it for the curious type, value, traceback = item if isinstance(type, StopIteration): return else: raise type, value, traceback def __iter__(self): "Return an iterator for our executed self." return iter(self())
Twisted et al work fine for non-blocking calls like socket reads, but aren't so good for things you know are going to block, like directory scans. I haven't yet performed extensive testing to find out what works best spawned and what doesn't, but there was an advantage to what I was doing at the time.