ActiveState Code

Recipe 65203: portalocker - Cross-platform (posix/nt) API for flock-style file locking.

Synopsis:

import portalocker file = open("somefile", "r+") portalocker.lock(file, portalocker.LOCK_EX) file.seek(12) file.write("foo") file.close()

Python 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
# portalocker.py - Cross-platform (posix/nt) API for flock-style file locking.
#                  Requires python 1.5.2 or better.
"""Cross-platform (posix/nt) API for flock-style file locking.

Synopsis:

   import portalocker
   file = open("somefile", "r+")
   portalocker.lock(file, portalocker.LOCK_EX)
   file.seek(12)
   file.write("foo")
   file.close()

If you know what you're doing, you may choose to

   portalocker.unlock(file)

before closing the file, but why?

Methods:

   lock( file, flags )
   unlock( file )

Constants:

   LOCK_EX
   LOCK_SH
   LOCK_NB

Exceptions:

    LockException

Notes:

For the 'nt' platform, this module requires the Python Extensions for Windows.
Be aware that this may not work as expected on Windows 95/98/ME.

History:

I learned the win32 technique for locking files from sample code
provided by John Nielsen <nielsenjf@my-deja.com> in the documentation
that accompanies the win32 modules.

Author: Jonathan Feinberg <jdf@pobox.com>,
        Lowell Alleman <lalleman@mfps.com>
Version: $Id: portalocker.py 5474 2008-05-16 20:53:50Z lowell $

"""


__all__ = [
    "lock",
    "unlock",
    "LOCK_EX",
    "LOCK_SH",
    "LOCK_NB",
    "LockException",
]

import os

class LockException(Exception):
    # Error codes:
    LOCK_FAILED = 1

if os.name == 'nt':
    import win32con
    import win32file
    import pywintypes
    LOCK_EX = win32con.LOCKFILE_EXCLUSIVE_LOCK
    LOCK_SH = 0 # the default
    LOCK_NB = win32con.LOCKFILE_FAIL_IMMEDIATELY
    # is there any reason not to reuse the following structure?
    __overlapped = pywintypes.OVERLAPPED()
elif os.name == 'posix':
    import fcntl
    LOCK_EX = fcntl.LOCK_EX
    LOCK_SH = fcntl.LOCK_SH
    LOCK_NB = fcntl.LOCK_NB
else:
    raise RuntimeError, "PortaLocker only defined for nt and posix platforms"

if os.name == 'nt':
    def lock(file, flags):
        hfile = win32file._get_osfhandle(file.fileno())
        try:
            win32file.LockFileEx(hfile, flags, 0, -0x10000, __overlapped)
        except pywintypes.error, exc_value:
            # error: (33, 'LockFileEx', 'The process cannot access the file because another process has locked a portion of the file.')
            if exc_value[0] == 33:
                raise LockException(LockException.LOCK_FAILED, exc_value[2])
            else:
                # Q:  Are there exceptions/codes we should be dealing with here?
                raise
    
    def unlock(file):
        hfile = win32file._get_osfhandle(file.fileno())
        try:
            win32file.UnlockFileEx(hfile, 0, -0x10000, __overlapped)
        except pywintypes.error, exc_value:
            if exc_value[0] == 158:
                # error: (158, 'UnlockFileEx', 'The segment is already unlocked.')
                # To match the 'posix' implementation, silently ignore this error
                pass
            else:
                # Q:  Are there exceptions/codes we should be dealing with here?
                raise

elif os.name == 'posix':
    def lock(file, flags):
        try:
            fcntl.flock(file.fileno(), flags)
        except IOError, exc_value:
            #  IOError: [Errno 11] Resource temporarily unavailable
            if exc_value[0] == 11:
                raise LockException(LockException.LOCK_FAILED, exc_value[1])
            else:
                raise
    
    def unlock(file):
        fcntl.flock(file.fileno(), fcntl.LOCK_UN)



if __name__ == '__main__':
    from time import time, strftime, localtime
    import sys
    import portalocker

    log = open('log.txt', "a+")
    portalocker.lock(log, portalocker.LOCK_EX)

    timestamp = strftime("%m/%d/%Y %H:%M:%S\n", localtime(time()))
    log.write( timestamp )

    print "Wrote lines. Hit enter to release lock."
    dummy = sys.stdin.readline()

    log.close()

Discussion

I think this will be most helpful to former Perl users such as myself, who are used to essentially transparent use of the lock() system call across platforms.

Updated 12/12/2007 to make it compatible with contemporary Pythons, per the comments below.

Updated 5/16/2008 with improvements from Lowell Alleman lalleman@mfps.com. Thanks, Lowell!

Comments

  1. 1. At 12:36 a.m. on 15 jan 2003, James Kew said:

    Note however this fails on Win95, Win98. Although these platforms have os.name == 'nt', the Win32 API on them does not support the LockFileEx and UnlockFileEx functions -- this recipe will fail with api_error at the win32 function calls.

    Win32 supports LockFile, UnlockFile on all platforms, but with lower functionality: no shared/exclusive flag, no block-until-locked mode.

  2. 2. At 7:12 a.m. on 16 jun 2005, Jordan Rastrick said:

    How to make the recipe work for Python 2.4. The code does cause an error under Python 2.4

    Apparently, this is because the hexidecimal constant, 0xffff0000, which evaluated to the integer -65536 in Python 2.3, evaluates to the long integer 4294901760 in Python 2.4 due to the unification of long and short integers.

    Changing the constant to -0x10000 seems to correct the problem.

  3. 3. At 4:34 p.m. on 7 jul 2005, ben w said:

    home-made file locking.

    here's an approach to file locking that might be useful to somebody, especially for a consumer-producer type situation. It uses the fact that on windows, renaming a file that is already open will raise an exception.

____Producer___:

import time, os

#lock readers out by renaming the file
if os.access("filename", os.F_OK):
     for failure_counter in range(0, 600):
         try:
          os.rename("filename", "filename.temp")
              break #removed the file successfully
     except:
          time.sleep(1) #someone is reading from the file
    else:
    raise Exception #consumers kept the file open for over 10 min

#file is now locked
file = open("filename.temp", "w")
file.write("bla bla")
file.close()
os.rename("filename.temp", "filename") #unlock the file


____Consumers___:

import time, os

#wait until file exists (eg. is unlocked
for i in range(0,600):
  if os.acces("filename", os.F_OK):
      break
  else:
      time.sleep(1)
else:
  raise Exception #file still locked after 10 minutes

#access the file
file = open("filename", "r")
contents = file.read()
file.close()
  4. 4. At 8:59 p.m. on 2 nov 2005, Minsue Hur said:

    error in WinXpsp2. ActivePython 2.4.1 Build 247 (ActiveState Corp.) based on Python 2.4.1 (#65, Jun 20 2005, 17:01:55) [MSC v.1310 32 bit(Intel)] on win32

    <p></p> Traceback (most recent call last): File "portalocker.py", line 80, in ? portalocker.lock(log, portalocker.LOCK_EX) File "C:\portalocker.py", line 61, in lock win32file.LockFileEx(hfile, flags, 0, 0xffff0000, __overlapped) OverflowError: long int too large to convert to int

  5. 5. At 7:40 p.m. on 12 dec 2007, Jonathan Feinberg (the author) said:

    I have addressed this problem, per the comments above.

  6. 6. At 7:40 p.m. on 12 dec 2007, Jonathan Feinberg (the author) said:

    Thanks. Jordan, I have incorporated your fix into the code, above. Thanks very much.

  7. 7. At 4:31 p.m. on 16 may 2008, Lowell Alleman said:

    Minor changes: Making portalocker.py slightly more portable... I sent some improvements to Jonathan and he updated portalocker.py on this page to include them. But for the rest of you out there, I wanted to give a quick explanation. In a sentence, all of the changes come down to this: Getting portalocker to deal with more platform-specific-stuff, so that you don't have to.

    Changes and notes:

    1. I added the LockException class to make error handling more uniform. Previously you had to capture IOError for Unix and pywintypes.error for windows, etc. So I modified both lock() functions to raise just LockException. (BTW. You should only get an exception if your using the LOCK_NB flag, otherwise it just waits (or "blocks") until the lock is available, which means it never raises an exception.)

    2. I updated the windows unlock() function to ignore requests to unlock an already unlocked file. This matches the 'posix' behavior, thus making it making it more predictable. (Not that you would normally unlock a file twice, but I somehow manged run into the problem, so perhaps this will help some one else one day.)

    3. UNSOLVED: If you request to lock a file multiple time using LOCK_EX on windows, it will block forever! On unix, it lets you run lock() as many times as you want. Perhaps someone with windows API experience can provide a way to determine if we have already locked the file? But in the mean time, just know that locking the same file multiple times will work differently between platforms.

    There are number of example on how to use this module, but I haven't seen many (any?) with exception handling. Here is a best-practices example with the new LockException class:

    fp = open("my_lock_file", "w")
try:
   portalocker.lock(fp, portalocker.LOCK_EX | portalocker.LOCK_NB)
except portalocker.LockException:
   # File locked by somebody else. Do some other work, then try again later.
   pass
else:
   # Lock acquired.  Do your thing
   try:
      while chapters in book:
         fp.write(chapters)
   finally:
      unlock(fp)

    Ok that's it. Enjoy.

Sign in to comment