Welcome, guest | Sign In | My Account | Store | Cart

Spawned Generators (Python recipe) by Garth Kidd
ActiveState Code (http://code.activestate.com/recipes/196890/)

The SpawnedGenerator class, initialised with a generator, will run that generator in a separate thread and either yield successive values obtained from it (when called) or let you iterate itself to get the same results. It's mainly useful for tasks which make blocking OS calls, e.g. traversing directory structures. The queue size may be specified to limit how far ahead of the main task the spawned generator can get.

Python, 49 lines
Download
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
from __future__ import generators
import threading

class SpawnedGenerator(threading.Thread):
    "Class to spawn a generator."

    def __init__(self, generator, queueSize=0):
        "Initialise the spawned generator from a generator."
        threading.Thread.__init__(self)
        self.generator = generator
        self.queueSize = queueSize

    def stop(self):
        "Request a stop."
        self.stopRequested = 1

    def run(self):
        "Called in a separate thread by Thread.start()."
        queue = self.queue
        try:
            it = iter(self.generator)
            while 1:
                next = it.next()
                queue.put((1, next)) # will raise StopIteration
                if self.stopRequested:
                    raise StopIteration, "stop requested"
        except:
            queue.put((0, sys.exc_info()))

    def __call__(self):
        "Yield results obtained from the generator."
        self.queue = queue = Queue.Queue(self.queueSize)
        self.stopRequested = 0
        self.start() # run() will stuff the queue in a separate Thread 
        keepGoing, item = queue.get()
        while keepGoing:
            yield item
            keepGoing, item = queue.get()
        # if keepGoing is false, item is exc_info() result
        self.exc_info = item # stash it for the curious
        type, value, traceback = item
        if isinstance(type, StopIteration):
            return
        else:
            raise type, value, traceback

    def __iter__(self):
        "Return an iterator for our executed self."
        return iter(self())

Twisted et al work fine for non-blocking calls like socket reads, but aren't so good for things you know are going to block, like directory scans. I haven't yet performed extensive testing to find out what works best spawned and what doesn't, but there was an advantage to what I was doing at the time.