A 'fields' class is a class that acts like a struct full of fields, e.g.:
<pre> class Titles: def __init__(self, name): self.name = name def dr(self): return "Dr. " + self.name def mr(self): return "Mr. " + self.name </pre>
Once an instance is constructed, the return value of x.dr() or x.mr() doesn't change.
I sometimes have 'fields' classes that I need to pickle (e.g. they go over a wire) but they contain unpickleable members, like file objects or tracebacks.
This code adds picklers to such classes that automatically 'flattens' the instances on pickling, saving the return values and making the unpickled instance return the saved values.
Download
Copy to clipboard1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 | def _is_pickleable(obj):
"Is the object pickleable?"
import cPickle as pickle
try:
pickle.dumps(obj)
return True
except TypeError:
return False
class UnpickleableFieldError(Exception): pass
def add_fields_pickling(klass, disable_unpickleable_fields=False):
"""
Add pickling for 'fields' classes.
A 'fields' class is a class who's methods all act as fields - accept no
arguments and for a given class's state always return the same value.
Useful for 'fields' classes that contain unpickleable members.
Used in TestOOB, http://testoob.sourceforge.net
A contrived example for a 'fields' class:
class Titles:
def __init__(self, name):
self.name = name
def dr(self):
return "Dr. " + self.name
def mr(self):
return "Mr. " + self.name
If a method returns an unpickleable value there are two options:
Default:
Allow the instance to be pickled. If the method is called on the
unpickled instance, an UnpickleableFieldError exception is raised.
There is a possible performance concern here: each return value is
pickled twice when pickling the instance.
With disable_unpickleable_fields=True:
Disallow pickling of instances with a method returning an unpickleable
object.
"""
def state_extractor(self):
from types import MethodType
fields_dict = {}
unpickleable_fields = []
def save_field(name, method):
try:
retval = method()
if disable_unpickleable_fields or _is_pickleable(retval):
fields_dict[name] = retval # call the method
else:
unpickleable_fields.append(name)
except TypeError:
raise TypeError("""not a "fields" class, problem with method '%s'""" % name)
for attr_name in dir(self):
if attr_name in ("__init__", "__getstate__", "__setstate__"):
continue # skip constructor and state magic methods
attr = getattr(self, attr_name)
if type(attr) == MethodType:
save_field(attr_name, attr)
return (fields_dict, unpickleable_fields)
def build_from_state(self, state):
fields_dict, unpickleable_fields = state
# saved fields
for name in fields_dict.keys():
# set the default name argument to prevent overwriting the name
setattr(self, name, lambda name=name:fields_dict[name])
# unpickleable fields
for name in unpickleable_fields:
def getter(name=name):
raise UnpickleableFieldError(
"%s()'s result wasn't pickleable" % name)
setattr(self, name, getter)
klass.__getstate__ = state_extractor
klass.__setstate__ = build_from_state
#
# Example
#
class FileInfo:
"Unpickleable because of a file field"
def __init__(self, filename):
self.file = file(filename)
def get_fileno(self):
return self.file.fileno()
def get_num_lines(self):
self.file.seek(0)
return len(self.file.readlines())
def get_name(self):
return self.file.name
def get_file(self):
return self.file
import pickle
finfo = FileInfo("some_file")
try:
# This won't work, can't pickle a file object
pickled_string = pickle.dumps(finfo)
except TypeError:
print "Pickle failed"
# Registering the new pickler
add_fields_pickling(FileInfo)
finfo = FileInfo("some_file")
pickled_string = pickle.dumps(finfo) # Succeeds
print "Pickle succeeded"
finfo2 = pickle.loads(pickled_string)
print "name: ", finfo2.get_name()
print "fileno: ", finfo2.get_fileno()
print "num_lines:", finfo2.get_num_lines()
print "file: ",
try:
print finfo2.get_file()
except UnpickleableFieldError, e:
print e
# This will print:
#
# name: some_file
# fileno: 3
# num_lines: 218
# file: get_file()'s result wasn't pickleable
|
In TestOOB (http://testoob.sourceforge.net/) we're using this to wrap unpickleable instances that are passed as parameters to Pyro objects (http://pyro.sourceforge.net/).
PyUnit test fixtures are not always pickleable for a variety of reasons. Instead of passing the test fixtures directly and extracting information from them by hand (e.g. the test name, the test source file) we wrap tests in a TestInfo class and pass that instead.
<pre> class TestInfo: def __init__(self, test): ... def name(self): ... def source_file(self): ... </pre>
Incomplete? It seems that not all the code is present. For example, there's no definition of the called function 'register_fields_pickler()' and the function 'add_fields_pickling()', which is defined, is not called anywhere...
Oops, you're right. The example was old, plus there was a bug (__setstate__ would recurse into itself).
I uploaded a new version which also allows pickling 'field' instances with methods returning unpickleable instances - an exception will be raised remotely when running these methods.
I hope the number of embarrassing mistakes has decreased...