Locks or mutexes are very basic primitives used to coordinate threads operations in multi-threaded programs. Unfortunately, even if Python provides a low-level implementation of locks in the thread module, the high level implementation of threading RLock is still in Python code, which is a bit worrying since locking is always a time critical task. This recipe implements both locks in Pyrex to get the speed of native code, and gives an example of how great Pyrex is.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 | #### This file is lock.pyx ###
"""This modules natively implements Lock and RLock from the threading module."""
cdef extern from "pythread.h":
ctypedef void* PyThread_type_lock
PyThread_type_lock PyThread_allocate_lock()
void PyThread_free_lock(PyThread_type_lock lock)
int PyThread_acquire_lock(PyThread_type_lock lock, int mode)
void PyThread_release_lock(PyThread_type_lock lock)
long PyThread_get_thread_ident()
cdef extern from "python.h":
ctypedef struct PyThreadState:
# this is a place holder
pass
PyThreadState* PyEval_SaveThread()
void PyEval_RestoreThread(PyThreadState* state)
global WAIT_LOCK
global NO_WAIT_LOCK
WAIT_LOCK = 1
NO_WAIT_LOCK = 0
cdef class Lock:
"""A basic, non-reentrant, Lock."""
cdef PyThread_type_lock lock
cdef int locked
def __new__(self):
self.lock = PyThread_allocate_lock()
self.locked=0
def __dealloc__(self):
PyThread_free_lock(self.lock)
def acquire(self,int mode=1):
"""Lock the lock. Without argument, this blocks if the lock is already
locked (even by the same thread), waiting for another thread to release
the lock, and return None once the lock is acquired.
With an argument, this will only block if the argument is true,
and the return value reflects whether the lock is acquired.
The blocking operation is not interruptible."""
cdef int result
cdef PyThreadState* state
# this is the equivalent of Py_BEGIN_ALLOW_THREADS
state=PyEval_SaveThread()
result = PyThread_acquire_lock(self.lock,mode)
# this is the equivalent of Py_END_ALLOW_THREADS
PyEval_RestoreThread(state)
if result==1:
self.locked = 1
return True
else:
return False
def release(self):
"""Release the lock, allowing another thread that is blocked waiting for
the lock to acquire the lock. The lock must be in the locked state,
but it needn't be locked by the same thread that unlocks it."""
if self.locked == 0:
raise Exception('this lock is not locked')
PyThread_release_lock(self.lock)
self.locked = 0
cdef class RLock(Lock):
"""A reentrant Lock. It can be locked many times by the same thread."""
cdef long locker
def acquire(self,int mode=1):
"""Lock the lock. Without argument, this blocks if the lock is already
locked (even by the same thread), waiting for another thread to release
the lock, and return None once the lock is acquired.
With an argument, this will only block if the argument is true,
and the return value reflects whether the lock is acquired.
The blocking operation is not interruptible."""
cdef long candidate
cdef int result
cdef PyThreadState* state
candidate = PyThread_get_thread_ident()
if self.locked==0 or candidate!=self.locker:
state=PyEval_SaveThread()
result = PyThread_acquire_lock(self.lock,mode)
PyEval_RestoreThread(state)
if result==1:
self.locked = 1
self.locker = candidate
return True
else:
return False
else:
self.locked = self.locked + 1
return True
def release(self):
"""Release the lock, allowing another thread that is blocked waiting for
the lock to acquire the lock. The lock must be in the locked state,
but it needn't be locked by the same thread that unlocks it."""
cdef long candidate
if self.locked==0:
raise Exception('this lock is not locked')
else:
candidate = PyThread_get_thread_ident()
if candidate!=self.locker:
raise Exception('thread %i cannot release lock owned by thread %i'%(candidate,self.locker))
else:
self.locked = self.locked - 1
if self.locked==0:
PyThread_release_lock(self.lock)
### This file is setup.py ###
from distutils.core import setup
from distutils.extension import Extension
from Pyrex.Distutils import build_ext
setup(
name = 'Lock module',
ext_modules=[
Extension("lock", ["lock.pyx"]),
],
cmdclass = {'build_ext': build_ext}
)
|
To build this module, install Pyrex 0.9.3 (http://nz.cosc.canterbury.ac.nz/~greg/python/Pyrex/) and run :
python setup.py build python setup.py install
Under a Win32 OS (I use Windows XP for instance), my advice is to install the MinGW compiler set (http://www.mingw.org/), put it on your path and run :
python setup.py build --compiler=mingw32 python setup.py install
You can have a look at the C code Pyrex generates in lock.c. It's not the cleanest and most straightforward code ; you could certainly write "better" code by hand. But what Pyrex generates is safe (it handles exceptions, INCREFs and DECREFs automatically) and it's VERY relieving not to have to write all this !
Many thanks to the GIL. Note that this code is quite simple thanks to the existence of the Global Interpreter Lock and the fact that the methods are natively implemented. This means that not two threads will ever run concurrently the native methods of Lock and RLock, except where I said it could using the (equivalent of) Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros.
The GIL is required because I make some tests on self.locked, then do something based on the its value (same thing with self.locker). If I was in a truly multithreaded environment, things could change between the test and its consequences, so I would have to think about putting locks in my locks...
Anyway, this is the result of an hour and a half of discovering Pyrex and the (partly hidden) PyThread C API. From now on, I'm going to try to minimize the dependencies on the GIL, if it's possible.
Nitpicking? AFAIK, locking is not time-critical. It is critical, in the sense that the manipulation of the lock object should be atomic (i.e. no other thread should manipulate it at the same time), but the time factor shouldn't matter.
Of course, making it faster helps for general performance, but it doesn't automatically make it safer. If the locking mechanism is flawed, speeding it up will only lower the probability that it fails, but it will still fail eventually.
After all this rambling, I still want to thank you for this useful example of Pyrex, it makes me want to look more into it.
Well... ...it can be nitpicking indeed since I have to confess that I have yet to see the result of a Python profile showing "ha ! RLock.acquire() is an hotspot !". Anyway, I usually use locks on time critical paths (all the more critical that they are contention points for threads) and I feel much better with a native, lightweight implementation than the current threading.RLock implementation (which is not so lightweight).