Welcome, guest | Sign In | My Account | Store | Cart

Here is an implementation of cooperative multithreading using generators that handles signals (SIGINT only in this recipe).

Python, 48 lines
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
from __future__ import generators

import signal

# An implementation of cooperative multithreading using generators
# that handles signals; by Brian O. Bush

# credit: based off an article by David Mertz
# http://gnosis.cx/publish/programming/charming_python_b7.txt

def empty():
    """ This is an empty task. """
    while True:
        print "<empty process>"
        yield None
        
def delay(duration):
    import time
    while True:
        print "<sleep %d>" % duration
        time.sleep(duration)
        yield None

class GenericScheduler:
    def __init__(self):
        signal.signal(signal.SIGINT, self.shutdownHandler)
        self.shutdownRequest = False
        self.threads = []
        # add some "processes"
        self.threads.append(delay(1))
        self.threads.append(delay(2))
        self.threads.append(empty())
    def shutdownHandler(self, n, frame):
        """ Initiate a request to shutdown cleanly on SIGINT."""
        print "Request to shut down."
        self.shutdownRequest = True        
    def scheduler(self):
        try:
            while 1:
                map(lambda t: t.next(), self.threads)
                if self.shutdownRequest:
                    break
        except StopIteration:
            pass

if __name__== "__main__":
    s = GenericScheduler()
    s.scheduler()

I wanted a simple framework to handle signals and process multiple tasks "simultaneously." Only issues are that you can starve your processes if you let a process (function) work on a long running task. If you want something simple this recipe is for you, however, if you are doing tasks such as web crawling, server/client I/O, etc. stick with regular threads. Otherwise enjoy!

3 comments

Troy Melhase 21 years, 5 months ago  # | flag

Cool Idea! Minor Changes... i like this idea a lot! i've made a few cosmetic changes:

moved the 'import time' statement to the top of the module. i didn't see any need to execute the import on every call to 'delay'.

replaced every 'True' with 1 and 'False' with 0. the change should make the code backward compatible with older Python versions.

replaced the 'self.thread=[]' statement in GenericScheduler.__init__ with an assigment to the 'thread' parameter. i favor allowing clients to specify as many attributes as possible.

moved the try/except block inside the while loop in GenericScheduler.scheduler.

from __future__ import generators

import signal
import time

# An implementation of cooperative multithreading using generators
# that handles signals; by Brian O. Bush

# credit: based off an article by David Mertz
# http://gnosis.cx/publish/programming/charming_python_b7.txt

def empty():
    """ This is an empty task. """
    while 1:
        print ""
        yield None

def delay(duration):
    while 1:
        print "" % duration
        time.sleep(duration)
        yield None

class GenericScheduler:
    def __init__(self, threads):
        self.shutdownRequest = 0
        self.threads = threads
        signal.signal(signal.SIGINT, self.shutdownHandler)

    def shutdownHandler(self, n, frame):
        """ Initiate a request to shutdown cleanly on SIGINT."""
        print "Request to shut down."
        self.shutdownRequest = 1

    def scheduler(self):
        while 1:
            try:
                map(lambda t: t.next(), self.threads)
                if self.shutdownRequest:
                    break
            except StopIteration:
                break


if __name__== "__main__":
    ts = [delay(1), delay(2), empty()]
    s = GenericScheduler(ts)
    s.scheduler()

as i said, the changes are mostly cosmetic. thanks for this recipie, i'm sure i'll use it!

David Beach 21 years, 5 months ago  # | flag

More "improvements" Since the code already uses the generators feature, it's pretty damn safe to assume that True and False exist. Moreover, anyone who has read the "Python Regrets" presentation from Guido (our fearless leader), will know to use the "list-builder" notation in preference to "map". (The generated code is actually faster, too!) Here are my picky changes:

from __future__ import generators

import signal
import time

# An implementation of cooperative multithreading using generators
# that handles signals; by Brian O. Bush

# credit: based off an article by David Mertz
# http://gnosis.cx/publish/programming/charming_python_b7.txt

def empty():
    """ This is an empty task. """
    while True:
        print ""
        yield None

def delay(duration):
    while True:
        print "" % duration
        time.sleep(duration)
        yield None

class GenericScheduler:
    def __init__(self, threads):
        self.shutdownRequest = False
        self.threads = threads
        signal.signal(signal.SIGINT, self.shutdownHandler)

    def shutdownHandler(self, n, frame):
        """ Initiate a request to shutdown cleanly on SIGINT."""
        print "Request to shut down."
        self.shutdownRequest = True

    def scheduler(self):
        while True:
            try:
                [ thread.next() for thread in self.threads ]
                if self.shutdownRequest:
                    break
            except StopIteration:
                break


if __name__== "__main__":
    ts = [delay(1), delay(2), empty()]
    s = GenericScheduler(ts)
    s.scheduler()
Martin Miller 21 years, 1 month ago  # | flag

Re: More "improvements" Regardless of Regrets Guido may have, there doesn't appear to be a need to build a list of results from calling each thread / generator's next() method. So instead of using map() or "list-builder" notation, all that's needed is simply:

for thread in self.threads: thread.next()

in the scheduler() method.

In IMHO the above is the most explicit statement of what needs and is being done (and is probably even faster, since it doesn't bother building a throw-away list nor incur any unnecessary function / method-call overhead).

BTW, some of the contents of the print statements in the original recipe's print statements do not show up in either commentator's code -- specifically those that contained text that looked like html tags.

For example, the

print "&lt;sleep %d&gt;" % duration

statement became

print "" % duration

which will cause a

TypeError: not all arguments converted

interpreter error to occur when it's run.

FWIW, to put literal "<" and/or ">" characters in comments here, they need to be replaced with their HTML character entity reference names, which are "&lt;" and "&gt;" repectively (sans the quotes). HTH.