Welcome, guest | Sign In | My Account | Store | Cart

Locks or mutexes are very basic primitives used to coordinate threads operations in multi-threaded programs. Unfortunately, even if Python provides a low-level implementation of locks in the thread module, the high level implementation of threading RLock is still in Python code, which is a bit worrying since locking is always a time critical task. This recipe implements both locks in Pyrex to get the speed of native code, and gives an example of how great Pyrex is.

Python, 122 lines
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
#### This file is lock.pyx ###
"""This modules natively implements Lock and RLock from the threading module."""

cdef extern from "pythread.h":
    ctypedef void* PyThread_type_lock
    PyThread_type_lock PyThread_allocate_lock()
    void  PyThread_free_lock(PyThread_type_lock lock)
    int PyThread_acquire_lock(PyThread_type_lock lock, int mode)
    void PyThread_release_lock(PyThread_type_lock lock)
    long PyThread_get_thread_ident()

cdef extern from "python.h":
    ctypedef struct PyThreadState:
        # this is a place holder
        pass
    PyThreadState* PyEval_SaveThread()
    void PyEval_RestoreThread(PyThreadState* state)

global WAIT_LOCK
global NO_WAIT_LOCK
WAIT_LOCK = 1
NO_WAIT_LOCK = 0

cdef class Lock:
    """A basic, non-reentrant, Lock."""
    cdef PyThread_type_lock lock
    cdef int locked

    def __new__(self):
        self.lock = PyThread_allocate_lock()
        self.locked=0
    
    def __dealloc__(self):
        PyThread_free_lock(self.lock)
        
    def acquire(self,int mode=1):
        """Lock the lock.  Without argument, this blocks if the lock is already
           locked (even by the same thread), waiting for another thread to release
           the lock, and return None once the lock is acquired.
           With an argument, this will only block if the argument is true,
           and the return value reflects whether the lock is acquired.
           The blocking operation is not interruptible."""
        cdef int result
        cdef PyThreadState* state
        # this is the equivalent of Py_BEGIN_ALLOW_THREADS
        state=PyEval_SaveThread()
        result = PyThread_acquire_lock(self.lock,mode)
        # this is the equivalent of Py_END_ALLOW_THREADS
        PyEval_RestoreThread(state)
        if result==1:
            self.locked = 1
            return True
        else:
            return False
        
    def release(self):
        """Release the lock, allowing another thread that is blocked waiting for
           the lock to acquire the lock.  The lock must be in the locked state,
           but it needn't be locked by the same thread that unlocks it."""
        if self.locked == 0:
            raise Exception('this lock is not locked')
        PyThread_release_lock(self.lock)
        self.locked = 0

cdef class RLock(Lock):
    """A reentrant Lock. It can be locked many times by the same thread."""
    cdef long locker
    
    def acquire(self,int mode=1):
        """Lock the lock.  Without argument, this blocks if the lock is already
           locked (even by the same thread), waiting for another thread to release
           the lock, and return None once the lock is acquired.
           With an argument, this will only block if the argument is true,
           and the return value reflects whether the lock is acquired.
           The blocking operation is not interruptible."""
        cdef long candidate
        cdef int result
        cdef PyThreadState* state

        candidate = PyThread_get_thread_ident()
        if self.locked==0 or candidate!=self.locker:
            state=PyEval_SaveThread()
            result = PyThread_acquire_lock(self.lock,mode)
            PyEval_RestoreThread(state)
            if result==1:
                self.locked = 1
                self.locker = candidate
                return True
            else:
                return False
        else:
            self.locked = self.locked + 1
            return True

    def release(self):
        """Release the lock, allowing another thread that is blocked waiting for
           the lock to acquire the lock.  The lock must be in the locked state,
           but it needn't be locked by the same thread that unlocks it."""
        cdef long candidate
        if self.locked==0:
            raise Exception('this lock is not locked')
        else:
            candidate = PyThread_get_thread_ident()
            if candidate!=self.locker:
                raise Exception('thread %i cannot release lock owned by thread %i'%(candidate,self.locker))
            else:
                self.locked = self.locked - 1
                if self.locked==0:
                    PyThread_release_lock(self.lock)

### This file is setup.py ###
from distutils.core import setup 
from distutils.extension import Extension 
from Pyrex.Distutils import build_ext 
 
setup( 
  name = 'Lock module', 
  ext_modules=[ 
    Extension("lock",         ["lock.pyx"]), 
  ], 
  cmdclass = {'build_ext': build_ext} 
) 

To build this module, install Pyrex 0.9.3 (http://nz.cosc.canterbury.ac.nz/~greg/python/Pyrex/) and run :

python setup.py build python setup.py install

Under a Win32 OS (I use Windows XP for instance), my advice is to install the MinGW compiler set (http://www.mingw.org/), put it on your path and run :

python setup.py build --compiler=mingw32 python setup.py install

You can have a look at the C code Pyrex generates in lock.c. It's not the cleanest and most straightforward code ; you could certainly write "better" code by hand. But what Pyrex generates is safe (it handles exceptions, INCREFs and DECREFs automatically) and it's VERY relieving not to have to write all this !

3 comments

Nicolas Lehuen (author) 17 years, 1 month ago  # | flag

Many thanks to the GIL. Note that this code is quite simple thanks to the existence of the Global Interpreter Lock and the fact that the methods are natively implemented. This means that not two threads will ever run concurrently the native methods of Lock and RLock, except where I said it could using the (equivalent of) Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros.

The GIL is required because I make some tests on self.locked, then do something based on the its value (same thing with self.locker). If I was in a truly multithreaded environment, things could change between the test and its consequences, so I would have to think about putting locks in my locks...

Anyway, this is the result of an hour and a half of discovering Pyrex and the (partly hidden) PyThread C API. From now on, I'm going to try to minimize the dependencies on the GIL, if it's possible.

Gerald Squelart 17 years, 1 month ago  # | flag

Nitpicking? AFAIK, locking is not time-critical. It is critical, in the sense that the manipulation of the lock object should be atomic (i.e. no other thread should manipulate it at the same time), but the time factor shouldn't matter.

Of course, making it faster helps for general performance, but it doesn't automatically make it safer. If the locking mechanism is flawed, speeding it up will only lower the probability that it fails, but it will still fail eventually.

After all this rambling, I still want to thank you for this useful example of Pyrex, it makes me want to look more into it.

Nicolas Lehuen (author) 17 years, 1 month ago  # | flag

Well... ...it can be nitpicking indeed since I have to confess that I have yet to see the result of a Python profile showing "ha ! RLock.acquire() is an hotspot !". Anyway, I usually use locks on time critical paths (all the more critical that they are contention points for threads) and I feel much better with a native, lightweight implementation than the current threading.RLock implementation (which is not so lightweight).

Created by Nicolas Lehuen on Tue, 19 Oct 2004 (PSF)
Python recipes (4591)
Nicolas Lehuen's recipes (7)

Required Modules

  • (none specified)

Other Information and Tasks