Welcome, guest | Sign In | My Account | Store | Cart

A 'fields' class is a class that acts like a struct full of fields, e.g.:

<pre> class Titles: def __init__(self, name): self.name = name def dr(self): return "Dr. " + self.name def mr(self): return "Mr. " + self.name </pre>

Once an instance is constructed, the return value of x.dr() or x.mr() doesn't change.

I sometimes have 'fields' classes that I need to pickle (e.g. they go over a wire) but they contain unpickleable members, like file objects or tracebacks.

This code adds picklers to such classes that automatically 'flattens' the instances on pickling, saving the return values and making the unpickled instance return the saved values.

Python, 144 lines
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
def _is_pickleable(obj):
    "Is the object pickleable?"
    import cPickle as pickle
    try:
        pickle.dumps(obj)
        return True
    except TypeError:
        return False

class UnpickleableFieldError(Exception): pass

def add_fields_pickling(klass, disable_unpickleable_fields=False):
    """
    Add pickling for 'fields' classes.

    A 'fields' class is a class who's methods all act as fields - accept no
    arguments and for a given class's state always return the same value.

    Useful for 'fields' classes that contain unpickleable members.

    Used in TestOOB, http://testoob.sourceforge.net

    A contrived example for a 'fields' class:

      class Titles:
        def __init__(self, name):
          self.name = name
        def dr(self):
          return "Dr. " + self.name
        def mr(self):
          return "Mr. " + self.name

    If a method returns an unpickleable value there are two options:
    Default:
      Allow the instance to be pickled. If the method is called on the
      unpickled instance, an UnpickleableFieldError exception is raised.
      There is a possible performance concern here: each return value is
      pickled twice when pickling the instance.

    With disable_unpickleable_fields=True:
      Disallow pickling of instances with a method returning an unpickleable
      object.
    """
    def state_extractor(self):
        from types import MethodType

        fields_dict = {}
        unpickleable_fields = []

        def save_field(name, method):
            try:
                retval = method()
                if disable_unpickleable_fields or _is_pickleable(retval):
                    fields_dict[name] = retval # call the method
                else:
                    unpickleable_fields.append(name)
            except TypeError:
                raise TypeError("""not a "fields" class, problem with method '%s'""" % name)

        for attr_name in dir(self):
            if attr_name in ("__init__", "__getstate__", "__setstate__"):
                continue # skip constructor and state magic methods

            attr = getattr(self, attr_name)

            if type(attr) == MethodType:
                save_field(attr_name, attr)

        return (fields_dict, unpickleable_fields)

    def build_from_state(self, state):
        fields_dict, unpickleable_fields = state
        # saved fields
        for name in fields_dict.keys():
            # set the default name argument to prevent overwriting the name
            setattr(self, name, lambda name=name:fields_dict[name])

        # unpickleable fields
        for name in unpickleable_fields:
            def getter(name=name):
                raise UnpickleableFieldError(
                        "%s()'s result wasn't pickleable" % name)
            setattr(self, name, getter)

    klass.__getstate__ = state_extractor
    klass.__setstate__ = build_from_state

#
# Example
#

class FileInfo:
    "Unpickleable because of a file field"
    def __init__(self, filename):
        self.file = file(filename)

    def get_fileno(self):
        return self.file.fileno()

    def get_num_lines(self):
        self.file.seek(0)
        return len(self.file.readlines())

    def get_name(self):
        return self.file.name

    def get_file(self):
        return self.file


import pickle
finfo = FileInfo("some_file")

try:
    # This won't work, can't pickle a file object
    pickled_string = pickle.dumps(finfo)
except TypeError:
    print "Pickle failed"

# Registering the new pickler
add_fields_pickling(FileInfo)

finfo = FileInfo("some_file")

pickled_string = pickle.dumps(finfo) # Succeeds
print "Pickle succeeded"

finfo2 = pickle.loads(pickled_string)

print "name:     ", finfo2.get_name()
print "fileno:   ", finfo2.get_fileno()
print "num_lines:", finfo2.get_num_lines()
print "file:     ",
try:
    print finfo2.get_file()
except UnpickleableFieldError, e:
    print e

# This will print:
#
# name:      some_file
# fileno:    3
# num_lines: 218
# file:      get_file()'s result wasn't pickleable

In TestOOB (http://testoob.sourceforge.net/) we're using this to wrap unpickleable instances that are passed as parameters to Pyro objects (http://pyro.sourceforge.net/).

PyUnit test fixtures are not always pickleable for a variety of reasons. Instead of passing the test fixtures directly and extracting information from them by hand (e.g. the test name, the test source file) we wrap tests in a TestInfo class and pass that instead.

<pre> class TestInfo: def __init__(self, test): ... def name(self): ... def source_file(self): ... </pre>

2 comments

Martin Miller 18 years, 4 months ago  # | flag

Incomplete? It seems that not all the code is present. For example, there's no definition of the called function 'register_fields_pickler()' and the function 'add_fields_pickling()', which is defined, is not called anywhere...

Ori Peleg (author) 18 years, 4 months ago  # | flag

Oops, you're right. The example was old, plus there was a bug (__setstate__ would recurse into itself).

I uploaded a new version which also allows pickling 'field' instances with methods returning unpickleable instances - an exception will be raised remotely when running these methods.

I hope the number of embarrassing mistakes has decreased...