Welcome, guest | Sign In | My Account | Store | Cart

Improved TemporaryFile (Python recipe) by Andy Chambers
ActiveState Code (http://code.activestate.com/recipes/496744/)

From PEP 42...

I wonder if it would be a good idea to have a new kind of temporary file that stored data in memory unless:

    - The data exceeds some size, or

    - Somebody asks for a fileno.

Then the cgi module (and other apps) could use this thing in a uniform way.

Python, 68 lines
Download
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
from StringIO import StringIO 
from tempfile import NamedTemporaryFile 
import sys 

class TempFile(object): 
    """A temporary file implementation that uses memory unless 
    either capacity is breached or fileno is requested, at which 
    point a real temporary file will be created and the relevant 
    details returned 
    """ 

    _strategies = (StringIO, NamedTemporaryFile) 

    def __init__(self, buffer, capacity): 
        """Creates a TempFile object containing the specified buffer. 
        If capacity is specified, we use a real temporary file once the 
        file gets larger than that size.  Otherwise, the data is stored 
        in memory. 
        """ 
        self.capacity = capacity 
        self._delegate = self._strategies[len(buffer) > self.capacity]() 
        self.write(buffer) 
    
    def fileno(self):
        """Forces this buffer to use a temporary file as the underlying.
        object and returns the fileno associated with it.
        """
        if isinstance(self._delegate, self._strategies[0]):
            new_delegate = self._strategies[1]()
            new_delegate.write(self.getvalue())
            self._delegate = new_delegate
        return self._delegate.fileno()

    def write(self, value):
        if isinstance(self._delegate, self._strategies[0]): 
            len_value = len(value) 
            if len_value >= self.capacity: 
                needs_new_strategy = True 
            else: 
                self.seek(0, 2)  # find end of file 
                needs_new_strategy = \
                    (self.tell() + len_value) >= self.capacity 
            if needs_new_strategy: 
                new_delegate = self._strategies[1]() 
                new_delegate.write(self.getvalue()) 
                self._delegate = new_delegate 
        self._delegate.write(value)

    def __getattr__(self, name): 
        try: 
            return getattr(self._delegate, name) 
        except AttributeError: 
            # hide the delegation 
            e = "object '%s' has no attribute '%s'" \
                     % (self.__class__.__name__, name) 
            raise AttributeError(e) 


if __name__ == "__main__": 
    print "testing tempfile:" 
    tmp = TempFile("", 100) 
    ten_chars = "1234567890" 
    tmp.write(ten_chars * 5) 
    print "tmp < 100: ", tmp._delegate
    tmp.write(ten_chars * 5) 
    print "tmp == 100: " , tmp._delegate
    tmp.write("the last straw") 
    print "tmp > 100: " , tmp._delegate
Tags: sysadmin

3 comments

Ori Peleg 17 years, 10 months ago  # | flag

Cool idea! I personally dislike the magic in _strategies's indexes, and Python altruistically casting booleans to 0/1... :)

Morten Lied Johansen 17 years, 10 months ago  # | flag

What happens when...

tmp = TempFile("some",100)
tmp.write("stuff")
fileno = tmp.fileno()
tmp.write("other stuff")

What is the contents of tmp now? I suspect you may have lost the "somestuff" part of the buffer when you asked for a fileno?

Andy Chambers (author) 17 years, 6 months ago  # | flag

tmp = TempFile("some",100)

tmp.write("stuff")

fileno = tmp.fileno()

tmp.write("other stuff")

What is the contents of tmp now? I suspect you may have lost the

"somestuff" part of the buffer when you asked for a fileno?

You're right. I've edited the recipe so that it writes the content of the buffer to the file before calling fileno. It should work correctly now.