Welcome, guest | Sign In | My Account | Store | Cart

This recipe is a followup of the Copying Generators recipe. It defines a GeneratorSnapshot object that stores all relevant data of a running generator but being serializable by pickle. During unpickling the generator is restored from the snapshot.

Python, 54 lines
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
###
#
#  W A R N I N G 
#  
#  This recipe is obsolete! 
#  
#  When you are looking for copying and pickling functionality for generators
#  implemented in pure Python download the 
#
#             generator_tools 
#
#  package at the cheeseshop or at www.fiber-space.de
#              
###

import pickle
import types

class any_obj:
    "Used to create objects for spawning arbitrary attributes by assignment"

class GeneratorSnapshot(object):
    '''
    Object used to hold data for living generators. 
    '''
    def __init__(self, f_gen):
        f_code = f_gen.gi_frame.f_code
        self.gi_frame = any_obj()
        self.gi_frame.f_code   = any_obj()
        self.gi_frame.f_code.__dict__.update((key, getattr(f_code, key))
                                                   for key in dir(f_code) if key.startswith("co_"))
        self.gi_frame.f_lasti  = f_gen.gi_frame.f_lasti
        self.gi_frame.f_locals = {}
        for key, value in f_gen.gi_frame.f_locals.items():
            if isinstance(value, types.GeneratorType):
                self.gi_frame.f_locals[key] = GeneratorSnapshot(value)
            else:
                self.gi_frame.f_locals[key] = value

def pickle_generator(f_gen, filename):
    '''
    @param f_gen: generator object
    @param filename: destination file for pickling generator
    '''
    output_pkl = open(filename, "wb")
    pickle.dump(GeneratorSnapshot(f_gen), output_pkl)

def unpickle_generator(filename):
    '''
    @param filename: source file of pickled generator
    '''
    input_pkl = open(filename, "rb")
    gen_snapshot = pickle.load(input_pkl)
    return copy_generator(gen_snapshot)[0]                

2 comments

Klaus Muller 14 years, 2 months ago  # | flag

Runtime error. Running this code, I get the error

Traceback (most recent call last):
  File "copytest.py", line 16, in ?
    pi.pickle_generator(inc(8),"pickle")
  File "/home/user/Desktop/picklinggenerators.py", line 37, in pickle_generator
    pickle.dump(GeneratorSnapshot(f_gen), output_pkl)
  File "/home/user/Desktop/picklinggenerators.py", line 22, in __init__
    self.gi_frame.f_code.__dict__.update((key, getattr(f.func_code, key)) for key in dir(f.func_code) if key.startswith("co_"))
NameError: global name 'f' is not defined
Running this code, I get the error
<pre>Traceback (most recent call last):
  File "copytest.py", line 16, in ?
    pi.pickle_generator(inc(8),"pickle")
  File "/home/user/Desktop/picklinggenerators.py", line 37, in pickle_generator
    pickle.dump(GeneratorSnapshot(f_gen), output_pkl)
  File "/home/user/Desktop/picklinggenerators.py", line 22, in __init__
    self.gi_frame.f_code.__dict__.update((key, getattr(f.func_code, key)) for key in dir(f.func_code) if key.startswith("co_"))
NameError: global name 'f' is not defined

</pre>

kay schluehr (author) 14 years, 2 months ago  # | flag

Problem fixed. The GeneratorSnapshot shall not contain unbound variables anymore ( the types module as an exception ).