Primitive Peer to Peer Chat (Python)

2013-07-07T02:09:57-07:00

Python recipe 578591 by teddy_k (chat, peer, select, socket, threading).

This took me longer than it should have. I am putting it here in the hopes that this post will spare others from having to Google things for extended periods of time.

In short, you either listen for a connection (Chat_Server), or connect to a remote IP address (Chat_Client). From there, you can send text strings back and forth.

This is a bit rough-hewn, obviously; I apologize in advance.

Simple threading decorator (Python)

2009-03-10T01:38:51-07:00

Python recipe 576684 by david.gaarenstroom (async, decorator, run_asynchronously, thread, threading). Revision 4.

When you're new at threading, using threads can be a bit daunting at first. If all you want is just to "run this function in parallel (= asynchronously) to the main program code", then this recipe can be of use. Simply use "@run_async" as a decorator for the function you want to run asynchronously. A call to that function will return immediately but the function itself will run in parallel.

Multiprocess-safe logging file-handler + interprocess RLock (Python)

2010-09-22T17:30:10-07:00

Python recipe 577395 by Jan Kaliszewski (concurrency, file, file_lock, flock, handler, logging, process, rlock, threading, threadsafe). Revision 11.

A Python 2.x/3.x-compatibile multiprocess-safe logging file-handler (logging.FileHandler replacement, designed for logging to a single file from multiple independent processes) together with a simple interprocess recursive lock -- universal abstract classes + Unix/Linux implementation.

Update: It's is a deeply revised version. Especially, now it --

is Python 2.4, 2.5, 2.6, 3.1 -compatibile (previously Py>=2.6 was needed); probably works also with 2.7, 3.0 and 3.2 (but not tested if it does);
is multiprocess-safe as well as thread-safe (proviously thread safety within a process was missed);
is based on public interfaces only (previously FileHandler._open() was called and overriden);
implement full RLock instance interface, as documented for threading.RLock (previously non-blocking mode and context-manager interface were missing).

The module contains:

Unix/Linux-only example implementation (with flock-based locking): FLockRLock and FLockFileHandler classes.
universal abstract classes -- which may be useful at developing implementation for non-Unix platforms: MultiprocessRLock, MultiprocessFileHandler, LockedFileHandler,

Also a quick-and-dirty test was added.

It is still an alpha version -- I'll be very grateful for any feedback.

Further updates:

2010-09-20: Some corrections, especially: non-blocking mode bug in MultiprocessRLock.acquire() fixed; _test() function improved; plus fixes in the description below.
2010-09-22: _test() improved and moved to description section. Mistaken copyright-notice removed.

Run OS command with timeout on a list of files using several threads (Python)

2015-05-19T19:31:59-07:00

Python recipe 579056 by Antoni Gual (command, subprocess, threading). Revision 3.

This hack runs a command sequentially on a list of files using two simultaneous threads. If one of the commands takes more than a set time, it's killed and the program goes for the next file. EDITED to add some exception handling.

Win32 named mutex class for system-wide mutex (Python)

2011-07-15T21:12:40-07:00

Python recipe 577794 by Ben Hoyt (ctypes, lock, mutex, process, threading, win32, windows).

NamedMutex is a class for using Windows (Win32) named mutexes for system-wide locks. For example, we use these to lock system-wide log files that multiple processes can write to.

Distributed lock manager for Python (Python)

2012-07-04T21:03:32-07:00

Python recipe 578194 by pavel (distributed, lock, locking, locks, manager, socket, thread, threading, wsgi). Revision 2.

Distributed lock manager provides mutex(es) over network. It is used to synchronize processes running on different machines, e.g. WSGI processes in case of web applications. Lock object is compatible with threading.Lock and can be used as a context manager ("with statement"). It can be easily modified to use UNIX sockets instead of TCP/IP. Communication protocol is text based.

First start server process:

$ chmod +x dlm.py
$ ./dlm.py

Usage:

from dlm import LockClient

client = LockClient('localhost', 27272, 'client_name')

lock = client.mkLock('lock_name')

lock.acquire()
# critical section here...
lock.release()

# using context manager
with lock:
    # critical section here...

Multithreaded FIFO Gate (Python)

2008-09-08T08:29:14-07:00

Python recipe 576418 by Anand (semaphores, synchronization, system, threading). Revision 5.

While programming with multiple threads, sometimes one needs a construct which allows to suspend the execution of a set of running threads. This is normally required by an outside thread which wants to suspend the running threads for performing a specific action. The threads need to resume after the action in the same order in which they got suspended.

A thread gate (or gateway) allows you to do this. It acts like a gate through which only one thread can pass at a time. By default the gate is open, allowing all threads to "enter" the gate. When a thread calls "close", the gate is closed, blocking any threads which make a further call to "enter", till the gate is re-opened by the owner, whence the threads resume the order in which they got blocked.

The real-life parallel for this is a human operated level cross, which allows only one vehicle to pass at a time.

Thread pool (grows on demand) (Python)

2009-11-03T02:28:12-08:00

Python recipe 576910 by Pepe Aracil (threading, threads, thread_pool). Revision 3.

Thread pool that grows on demand.

Multi-threaded Mandelbrot Fractal (Python)

2011-05-01T17:33:20-07:00

Python recipe 577680 by FB36 (fractal, graphics, image, math, mathematics, thread, threading, threads). Revision 3.

Multi-threaded Mandelbrot Fractal.

Simple python thread manager (Python)

2012-12-17T16:19:12-08:00

Python recipe 578385 by Tom Wissing (manager, python, threading, threads).

A simple python static class handling basic threading. Designed for "worker" threads that loop forever over a function and the like.

pyGTK threading (Python)

2009-04-01T07:07:56-07:00

Python recipe 576705 by oohay_email2004 (pygtk, threading). Revision 2.

This is my first successful attempt at threading in pyGTK.

Suggestions please.

Command Threads (Python)

2010-12-25T23:23:49-08:00

Python recipe 577505 by Shao-chuan Wang (threading). Revision 4.

A thread that accepts callable objects as "commands" or "jobs" to execute.

Concurrent buffer for generators (Python)

2010-05-05T22:47:33-07:00

Python recipe 576999 by Javier Ruere (concurrency, generator, iterator, threading). Revision 2.

A buffer that will partially consume an iterator in the background.

Very useful for reading files and merging the data using the excellent http://code.activestate.com/recipes/491285/

Simple python thread manager (Python)

2012-11-17T23:18:17-08:00

Python recipe 578331 by Tino Deak (manager, python, threading, threads).

A simple python static class handling basic threading. Designed for "worker" threads that loop forever over a function and the like.

threadbox.py (Python)

2012-10-09T22:40:09-07:00

Python recipe 578152 by Stephen Chappell (metaclass, threading). Revision 2.

Provide a way to run instance methods on a single thread.

This module allows hierarchical classes to be cloned so that their instances run on one thread. Method calls are automatically routed through a special execution engine. This is helpful when building thread-safe GUI code.

Decoupled Proxy of a State Bearing Object in a Multiprocessing Environment (Python)

2010-04-08T04:36:05-07:00

Python recipe 577180 by Mateyuzo (access, multiprocessing, process, processing, subprocess, synchronization, syncmanager, threading, __getattribute__). Revision 2.

Describes a simple example of the potential for shared Singleton or Borg object methods to be proxy'd by an object that SyncManger can present to requesting subprocesses.

Python SQLSMO - Threading Example (Python)

2015-02-10T02:01:42-08:00

Python recipe 579023 by Jorge Besada (database, python, smo, sql, sqlsmo, threading).

This is an example of the use of the SQLSMO module. Using a csv file DBLIST_ACTIONS.csv with list of databases where you can launch multiple different database operations in parallel

Some lines of the configuration file DBLIST_ACTIONS.csv used shown below: SERVERNAME,DBNAME1,SOURCESERVER,DATAFOLDER,LOGFOLDER,DBNAME2,ACTIONS,ENABLED (local)\sql2014,AdventureWorks2012,C:\SQL2014\BACKUPS,C:\SQL2014\DATA,C:\SQL2014\LOG,AdventureWorks_COPY1,RESTOREDBS1.CFG,Y (local)\sql2014,AdventureWorks2012,C:\SQL2014\BACKUPS,C:\SQL2014\DATA,C:\SQL2014\LOG,AdventureWorks_COPY2,RESTOREDBS1.CFG,Y (local)\sql2014,AdventureWorks2012,C:\SQL2014\BACKUPS,C:\SQL2014\DATA,C:\SQL2014\LOG,AdventureWorks_COPY3,RESTOREDBS1.CFG,Y (local)\sql2014,AdventureWorks2012,C:\SQL2014\BACKUPS,C:\SQL2014\DATA,C:\SQL2014\LOG,AdventureWorks_COPY4,RESTOREDBS1.CFG,

Where: SERVERNAME: server where the database to act upon resides

DBNAME1: source database

DBNAME2: destination database (may be different from source when we restore a copy with a different name)

SOURCESERVER: this is the network (or local) folder where backups are placed

DATAFOLDER: folder for data files

LOGFOLDER: folder for log files

ACTIONS: this is the name of the configuration file (.CFG) with the list of actions

ENABLED: a Y value here will mean we want to process the line

For each line (database) you specify a configuration file (in this case RESTOREDBS1.CFG), see sample below: (one line for each, no blank lines)

RESTORE DATABASE

SET DBOWNER: sa

SYNC LOGINS

SET SIMPLE MODE

SHRINK LOG

The program will process each line in the source CSV file and for each one it will perform the set of operations described in the configuration file. This system is being used in my workplace with different configuration files for different databases (there are configuration files for restores, specific restores with more actions, backups, etc, not included here for brevity).

Every time you do a database task you just add a line in the DBLIST_ACTIONS.CSV and create (if needed) a configuration file). If you are going to actually use it include a "Y" in the ENABLED column Note: every line action in the configuration file must have been implemented in the function ActionParsing as one the entries in the big if statement.

Special features: You can specify at the start of the program if you want to really execute or not. You may want to do first a trial run setting NOEXECUTE_OPTION = 1 (instead of the default of 0). In this case the program will run and create the SQL script of the operations, not executing them. Note: it has been implemented in the restores so far, will add it to the other options later.

Threading: by default it will run as many threads as lines in the DBLIST_ACTIONS.CSV file. But you can change this option by setting a value to THREAD_POOL different than 0.

Future decorator in a pretty pythonic way (Python)

2011-10-20T16:51:48-07:00

Python recipe 577915 by Filippo Squillace (threading).

Without thinking in thread creation the idea is to call several times a function assigning a thread for each call with related parameters and returning the list of results in a pretty pythonic way. For example, if we have already defined the function 'func': res = func(par) we want to call that several times and get the values in the future. So: func(par1) func(par2) func(par3).get() func(par4).get_all()

assigning a thread for each call. The 'get' method returns the first possible results, and 'get_all' returns all the values in a list. The decorator works fine with kwargs too.

This recipe is based on: http://code.activestate.com/recipes/355651-implementing-futures-with-decorators/ that use only one call for the function. The problem in that recipe is that each call blocks the execution.

Vote if you like it!

actorish decorator for making async code look more like sync one and a less blocking (Python)

2011-10-30T19:59:44-07:00

Python recipe 577931 by Przemyslaw Podczasi (actor, threading).

I like how gevent is making async code to look like sync but non blocking without all the ugly callbacks. I tried doing that with threads and object proxy (I found great one at: http://pypi.python.org/pypi/ProxyTypes written by Phillip J. Eby, and this is where the actual magic happens).

For every function that is decorated it returns a proxy and the io call (or anything else) won't block until the value is actually needed. (should be some pools and args pickling there, to make it more like message passing but I didn't want to fuzzy the example) To use it as actor model, I guess it would require to queue requests to decorated object's methods and create a single thread to process them an in LazyProxy callback set q.get() instead of t.join()

Most viewed recipes tagged "threading"

Primitive Peer to Peer Chat (Python)

Simple threading decorator (Python)

Multiprocess-safe logging file-handler + interprocess RLock (Python)

Run OS command with timeout on a list of files using several threads (Python)

Win32 named mutex class for system-wide mutex (Python)

Distributed lock manager for Python (Python)

Multithreaded FIFO Gate (Python)

Thread pool (grows on demand) (Python)

Multi-threaded Mandelbrot Fractal (Python)

Simple python thread manager (Python)

pyGTK threading (Python)

Command Threads (Python)

Concurrent buffer for generators (Python)

Simple python thread manager (Python)

threadbox.py (Python)

Decoupled Proxy of a State Bearing Object in a Multiprocessing Environment (Python)

Python SQLSMO - Threading Example (Python)

Future decorator in a pretty pythonic way (Python)

actorish decorator for making async code look more like sync one and a less blocking (Python)