1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141 | # portalocker.py - Cross-platform (posix/nt) API for flock-style file locking.
# Requires python 1.5.2 or better.
"""Cross-platform (posix/nt) API for flock-style file locking.
Synopsis:
import portalocker
file = open("somefile", "r+")
portalocker.lock(file, portalocker.LOCK_EX)
file.seek(12)
file.write("foo")
file.close()
If you know what you're doing, you may choose to
portalocker.unlock(file)
before closing the file, but why?
Methods:
lock( file, flags )
unlock( file )
Constants:
LOCK_EX
LOCK_SH
LOCK_NB
Exceptions:
LockException
Notes:
For the 'nt' platform, this module requires the Python Extensions for Windows.
Be aware that this may not work as expected on Windows 95/98/ME.
History:
I learned the win32 technique for locking files from sample code
provided by John Nielsen <nielsenjf@my-deja.com> in the documentation
that accompanies the win32 modules.
Author: Jonathan Feinberg <jdf@pobox.com>,
Lowell Alleman <lalleman@mfps.com>
Version: $Id: portalocker.py 5474 2008-05-16 20:53:50Z lowell $
"""
__all__ = [
"lock",
"unlock",
"LOCK_EX",
"LOCK_SH",
"LOCK_NB",
"LockException",
]
import os
class LockException(Exception):
# Error codes:
LOCK_FAILED = 1
if os.name == 'nt':
import win32con
import win32file
import pywintypes
LOCK_EX = win32con.LOCKFILE_EXCLUSIVE_LOCK
LOCK_SH = 0 # the default
LOCK_NB = win32con.LOCKFILE_FAIL_IMMEDIATELY
# is there any reason not to reuse the following structure?
__overlapped = pywintypes.OVERLAPPED()
elif os.name == 'posix':
import fcntl
LOCK_EX = fcntl.LOCK_EX
LOCK_SH = fcntl.LOCK_SH
LOCK_NB = fcntl.LOCK_NB
else:
raise RuntimeError, "PortaLocker only defined for nt and posix platforms"
if os.name == 'nt':
def lock(file, flags):
hfile = win32file._get_osfhandle(file.fileno())
try:
win32file.LockFileEx(hfile, flags, 0, -0x10000, __overlapped)
except pywintypes.error, exc_value:
# error: (33, 'LockFileEx', 'The process cannot access the file because another process has locked a portion of the file.')
if exc_value[0] == 33:
raise LockException(LockException.LOCK_FAILED, exc_value[2])
else:
# Q: Are there exceptions/codes we should be dealing with here?
raise
def unlock(file):
hfile = win32file._get_osfhandle(file.fileno())
try:
win32file.UnlockFileEx(hfile, 0, -0x10000, __overlapped)
except pywintypes.error, exc_value:
if exc_value[0] == 158:
# error: (158, 'UnlockFileEx', 'The segment is already unlocked.')
# To match the 'posix' implementation, silently ignore this error
pass
else:
# Q: Are there exceptions/codes we should be dealing with here?
raise
elif os.name == 'posix':
def lock(file, flags):
try:
fcntl.flock(file.fileno(), flags)
except IOError, exc_value:
# IOError: [Errno 11] Resource temporarily unavailable
if exc_value[0] == 11:
raise LockException(LockException.LOCK_FAILED, exc_value[1])
else:
raise
def unlock(file):
fcntl.flock(file.fileno(), fcntl.LOCK_UN)
if __name__ == '__main__':
from time import time, strftime, localtime
import sys
import portalocker
log = open('log.txt', "a+")
portalocker.lock(log, portalocker.LOCK_EX)
timestamp = strftime("%m/%d/%Y %H:%M:%S\n", localtime(time()))
log.write( timestamp )
print "Wrote lines. Hit enter to release lock."
dummy = sys.stdin.readline()
log.close()
|
Download
Copy to clipboard
Note however this fails on Win95, Win98. Although these platforms have os.name == 'nt', the Win32 API on them does not support the LockFileEx and UnlockFileEx functions -- this recipe will fail with api_error at the win32 function calls.
Win32 supports LockFile, UnlockFile on all platforms, but with lower functionality: no shared/exclusive flag, no block-until-locked mode.
How to make the recipe work for Python 2.4. The code does cause an error under Python 2.4
Apparently, this is because the hexidecimal constant, 0xffff0000, which evaluated to the integer -65536 in Python 2.3, evaluates to the long integer 4294901760 in Python 2.4 due to the unification of long and short integers.
Changing the constant to -0x10000 seems to correct the problem.
home-made file locking.
error in WinXpsp2. ActivePython 2.4.1 Build 247 (ActiveState Corp.) based on Python 2.4.1 (#65, Jun 20 2005, 17:01:55) [MSC v.1310 32 bit(Intel)] on win32
<p></p> Traceback (most recent call last): File "portalocker.py", line 80, in ? portalocker.lock(log, portalocker.LOCK_EX) File "C:\portalocker.py", line 61, in lock win32file.LockFileEx(hfile, flags, 0, 0xffff0000, __overlapped) OverflowError: long int too large to convert to int
I have addressed this problem, per the comments above.
Thanks. Jordan, I have incorporated your fix into the code, above. Thanks very much.
Minor changes: Making portalocker.py slightly more portable... I sent some improvements to Jonathan and he updated portalocker.py on this page to include them. But for the rest of you out there, I wanted to give a quick explanation. In a sentence, all of the changes come down to this: Getting portalocker to deal with more platform-specific-stuff, so that you don't have to.
Changes and notes:
I added the LockException class to make error handling more uniform. Previously you had to capture IOError for Unix and pywintypes.error for windows, etc. So I modified both lock() functions to raise just LockException. (BTW. You should only get an exception if your using the LOCK_NB flag, otherwise it just waits (or "blocks") until the lock is available, which means it never raises an exception.)
I updated the windows unlock() function to ignore requests to unlock an already unlocked file. This matches the 'posix' behavior, thus making it making it more predictable. (Not that you would normally unlock a file twice, but I somehow manged run into the problem, so perhaps this will help some one else one day.)
UNSOLVED: If you request to lock a file multiple time using LOCK_EX on windows, it will block forever! On unix, it lets you run lock() as many times as you want. Perhaps someone with windows API experience can provide a way to determine if we have already locked the file? But in the mean time, just know that locking the same file multiple times will work differently between platforms.
There are number of example on how to use this module, but I haven't seen many (any?) with exception handling. Here is a best-practices example with the new LockException class:
Ok that's it. Enjoy.
The code above should suffice for many case, but it requires pywin32, it doesn't provide file record locking, and it doesn't deal with the fcntl() flaw (i.e the loss of all locks on a given file when any descriptor to it is closed).
I advise you to use the portable file stream I've implemented on a basis similar to your code, it offers file locking, disk cache synchonization and a whole bunch of cool features (advanced opening flags, stream inheritance management, fcntl workaround etc.) Besides, it works with py26, py27, py3K, and theoretically with other python implementations (ironpython, jython etc.)
http://bitbucket.org/pchambon/python-rock-solid-tools/
Thank you very much Chamon Pascal! Your hint that fcntl looses all locks when any descriptor to it is closed, helped me very much. If you remove the lock file before closing the lock-fd, you have a race condition: Between open() and fcntl() a different process can do: open,fcntl,unlink,close. Then you have a locked filedescriptor but the file in the filesystem is gone. The solution is to check for the inode after locking (if the inodes are different, you don't have the real lock) and don't close the second fd. If you would close the second fd (for os.stat) the lock of the first fd would get lost.
Removed pywin32 dependency by porting to ctypes. See http://roundup.hg.sourceforge.net/hgweb/roundup/roundup/file/tip/roundup/backends/portalocker.py