portalocker - Cross-platform (posix/nt) API for flock-style file locking. (Python recipe) by Jonathan Feinberg

ActiveState Code (http://code.activestate.com/recipes/65203/)

Note however this fails on Win95, Win98. Although these platforms have os.name == 'nt', the Win32 API on them does not support the LockFileEx and UnlockFileEx functions -- this recipe will fail with api_error at the win32 function calls.

Win32 supports LockFile, UnlockFile on all platforms, but with lower functionality: no shared/exclusive flag, no block-until-locked mode.

How to make the recipe work for Python 2.4. The code does cause an error under Python 2.4

Apparently, this is because the hexidecimal constant, 0xffff0000, which evaluated to the integer -65536 in Python 2.3, evaluates to the long integer 4294901760 in Python 2.4 due to the unification of long and short integers.

Changing the constant to -0x10000 seems to correct the problem.

home-made file locking.

here's an approach to file locking that might be useful to somebody, especially for a consumer-producer type situation. It uses the fact that on windows, renaming a file that is already open will raise an exception.

____Producer___:

import time, os

#lock readers out by renaming the file
if os.access("filename", os.F_OK):
     for failure_counter in range(0, 600):
         try:
          os.rename("filename", "filename.temp")
              break #removed the file successfully
     except:
          time.sleep(1) #someone is reading from the file
    else:
    raise Exception #consumers kept the file open for over 10 min

#file is now locked
file = open("filename.temp", "w")
file.write("bla bla")
file.close()
os.rename("filename.temp", "filename") #unlock the file


____Consumers___:

import time, os

#wait until file exists (eg. is unlocked
for i in range(0,600):
  if os.acces("filename", os.F_OK):
      break
  else:
      time.sleep(1)
else:
  raise Exception #file still locked after 10 minutes

#access the file
file = open("filename", "r")
contents = file.read()
file.close()

error in WinXpsp2. ActivePython 2.4.1 Build 247 (ActiveState Corp.) based on Python 2.4.1 (#65, Jun 20 2005, 17:01:55) [MSC v.1310 32 bit(Intel)] on win32

<p></p> Traceback (most recent call last): File "portalocker.py", line 80, in ? portalocker.lock(log, portalocker.LOCK_EX) File "C:\portalocker.py", line 61, in lock win32file.LockFileEx(hfile, flags, 0, 0xffff0000, __overlapped) OverflowError: long int too large to convert to int

I have addressed this problem, per the comments above.

Thanks. Jordan, I have incorporated your fix into the code, above. Thanks very much.

Minor changes: Making portalocker.py slightly more portable... I sent some improvements to Jonathan and he updated portalocker.py on this page to include them. But for the rest of you out there, I wanted to give a quick explanation. In a sentence, all of the changes come down to this: Getting portalocker to deal with more platform-specific-stuff, so that you don't have to.

Changes and notes:

1. I added the LockException class to make error handling more uniform. Previously you had to capture IOError for Unix and pywintypes.error for windows, etc. So I modified both lock() functions to raise just LockException. (BTW. You should only get an exception if your using the LOCK_NB flag, otherwise it just waits (or "blocks") until the lock is available, which means it never raises an exception.)

2. I updated the windows unlock() function to ignore requests to unlock an already unlocked file. This matches the 'posix' behavior, thus making it making it more predictable. (Not that you would normally unlock a file twice, but I somehow manged run into the problem, so perhaps this will help some one else one day.)

3. UNSOLVED: If you request to lock a file multiple time using LOCK_EX on windows, it will block forever! On unix, it lets you run lock() as many times as you want. Perhaps someone with windows API experience can provide a way to determine if we have already locked the file? But in the mean time, just know that locking the same file multiple times will work differently between platforms.

There are number of example on how to use this module, but I haven't seen many (any?) with exception handling. Here is a best-practices example with the new LockException class:

fp = open("my_lock_file", "w")
try:
   portalocker.lock(fp, portalocker.LOCK_EX | portalocker.LOCK_NB)
except portalocker.LockException:
   # File locked by somebody else. Do some other work, then try again later.
   pass
else:
   # Lock acquired.  Do your thing
   try:
      while chapters in book:
         fp.write(chapters)
   finally:
      unlock(fp)

Ok that's it. Enjoy.

The code above should suffice for many case, but it requires pywin32, it doesn't provide file record locking, and it doesn't deal with the fcntl() flaw (i.e the loss of all locks on a given file when any descriptor to it is closed).

I advise you to use the portable file stream I've implemented on a basis similar to your code, it offers file locking, disk cache synchonization and a whole bunch of cool features (advanced opening flags, stream inheritance management, fcntl workaround etc.) Besides, it works with py26, py27, py3K, and theoretically with other python implementations (ironpython, jython etc.)

http://bitbucket.org/pchambon/python-rock-solid-tools/

Thank you very much Chamon Pascal! Your hint that fcntl looses all locks when any descriptor to it is closed, helped me very much. If you remove the lock file before closing the lock-fd, you have a race condition: Between open() and fcntl() a different process can do: open,fcntl,unlink,close. Then you have a locked filedescriptor but the file in the filesystem is gone. The solution is to check for the inode after locking (if the inodes are different, you don't have the real lock) and don't close the second fd. If you would close the second fd (for os.stat) the lock of the first fd would get lost.

Removed pywin32 dependency by porting to ctypes. See http://roundup.hg.sourceforge.net/hgweb/roundup/roundup/file/tip/roundup/backends/portalocker.py