Run asynchronous tasks using coroutines (Python)

2010-08-06T16:16:20-07:00

Python recipe 577129 by Arnau Sanchez (coroutine, event, generator, gobject, gtk, gui, network, nonblocking, pygtk). Revision 20.

This recipe shows a simple, transparent (and hopefully pythonic) way of running asynchronous tasks when writing a event-driven application (i.e. GUI). The aim is to allow a programmer to write time-consuming functions (usually IO-bound, but not only) with sequential-looking code, instead of scattering the logic over a bunch of callbacks. We will take advantage of the coroutines introduced in Python 2.5 (see http://www.python.org/dev/peps/pep-0342).

The goal: wouldn't it be great if we could write something like this?

def myjob(entry, arg1, arg2, arg3):
    result1 = function_that_takes_eons_to_complete(arg1, arg2)    
    result2 = another_function_that_downloads_a_big_really_big_file(result1, arg3)
    entry.set_text("The result is: %d" % result2)

def on_start_button___clicked(button, entry):
    myjob(entry, 1, 2, 3)

...    
gtk.main()

Indeed, but we can't! The GUI will hang until the job is done and the user will be rightfully angry. Coroutines to the rescue: the absolute minimal change we can make to this code is transforming myjob into a coroutine and yield every time we do blocking stuff:

def myjob(entry, arg1, arg2, arg3):
    result1 = yield some_task(arg1, arg2)    
    result2 = yield some_other_task(result1, arg3)
    entry.set_text("The result is: %d" % result2)

def on_start__clicked(button, entry):
    start_job(myjob(entry, 1, 2, 3))

some_task and some_other_task are here the asynchronous implementation of the sequential tasks used in the first fragment, and start_job the wrapper around the coroutine. Note that we still have to implement non-blocking versions of the tasks, but they are usually pretty generic (wait some time, download a file, ...) and can be re-used. If you happen to have a CPU-bound function or even a IO-bound code you cannot split (urllib2 anyone?), you can always use a generic threaded task (granted, the whole point of using co-routines should be avoiding threads, but there is no alternative here).

At the end, all the plumbing we need to make it work is just 1 function: start_job (wrapper around the job to manage the flow of the coroutine). The rest of the code -two asynchronous tasks (sleep_task, threaded_task) and a demo app- are shown solely as an example.

Non-blocking readlines() (Python)

2014-06-30T20:30:57-07:00

Python recipe 578900 by Zack Weinberg (concurrency, input, line, nonblocking).

A generator function which takes a file object (assumed to be some sort of pipe or socket, open for reading), and yields lines from it without blocking. If there is no input available, it will yield an endless stream of empty strings until input becomes available again; caller is responsible for not going into a busy loop. (Newlines are normalized but not stripped, so if there is actually a blank line in the input, the value yielded will be '\n'.) The intended use case is a thread which must respond promptly to input from a pipe, and also something else which cannot be fed to select (e.g. a queue.Queue). Note that the file object is ignored except for its fileno.

Only tested on Unix. Only tested on 3.4; ought to work with any python that has bytearray, locale.getpreferredencoding, and fcntl.

Multicontext (e.g. asynchronous) inline execution framework using coroutines (Python)

2012-12-06T19:32:20-08:00

Python recipe 576965 by Glenn Eychaner (asynchronous, coroutine, decorator, generator, inline, nonblocking, pattern, thread). Revision 14.

A framework for executing inline code, contained in a generator, across multiple execution contexts, by pairing it with an executor that handles the context switching at each yield. An example of a generator which executes some iterations synchronously and some asynchronously is provided. The framework is general enough to be applied to many different coroutine situations.

asyncore scheduler (Python)

2011-07-25T23:42:21-07:00

Python recipe 577808 by Giampaolo Rodolà (asynchronous, asyncore, heapq, nonblocking, scheduler, twisted). Revision 5.

The thing I miss mostly in asyncore is a system for calling a function after a certain amount of time without blocking. This is crucial for simple tasks such as disconnecting a peer after a certain time of inactivity or more advanced use cases such as bandwidth throttling.

This recipe was initially inspired by Twisted's internet.base.DelayedCall class:

http://twistedmatrix.com/trac/browser/tags/last_vfs_and_web2/twisted/internet/base.py#L34

...then included into pyftpdlib:

http://code.google.com/p/pyftpdlib/issues/detail?id=72

...and finally proposed for inclusion into asyncore:

http://bugs.python.org/issue1641

Tail multiple pidgin IRC logfiles (Python)

2015-06-06T12:13:00-07:00

Python recipe 579066 by Anton Vredegoor (colorize, irc, iterators, logfiles, merging, nonblocking, pidgin, tail, threads). Revision 2.

Tail multiple pidgin IRC logfiles.

Pidgin should be connected to IRC with the channels one wants to tail joined, and it should save logs as text.

The script needs two arguments:

the directory containing the directories with channel logs

a list of channel names, quoted and separated by spaces

Example command:

python pidgin-irctail.py -d ~/.purple/logs/irc/YOUR_IRC_HANDLE@irc.freenode.net -c "#chan1 #chan2 #chan3"

Some text elements are higlighted, and channel names are inserted into the log lines after the time info.

If more than one channel is entered, the output of the logs is merged.

Most viewed recipes tagged "nonblocking"

Run asynchronous tasks using coroutines (Python)

Non-blocking readlines() (Python)

Multicontext (e.g. asynchronous) inline execution framework using coroutines (Python)

asyncore scheduler (Python)

Tail multiple pidgin IRC logfiles (Python)