Welcome, guest | Sign In | My Account | Store | Cart

Decorator to expose the local variables defined in the inner scope of a function. At the exit of the decorated function (regular exit or exceptions), the local dictionary is copied to a read-only property, locals.

The main implementation is based on injecting bytecode into the original function, and requires the lightweight module byteplay (available here). See below for an alternative implementation that only uses the standard library.

Python, 99 lines
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
import new
import byteplay as bp
import inspect

def persistent_locals(f):
    """Function decorator to expose local variables after execution.

    Modify the function such that, at the exit of the function
    (regular exit or exceptions), the local dictionary is copied to a
    read-only function property 'locals'.

    This decorator wraps the function in a callable object, and
    modifies its bytecode by adding an external try...finally
    statement equivalent to the following:

    def f(self, *args, **kwargs):
        try:
            ... old code ...
        finally:
            self._locals = locals().copy()
            del self._locals['self']
    """

    # ### disassemble f
    f_code = bp.Code.from_code(f.func_code)

    # ### use bytecode injection to add try...finally statement around code
    finally_label = bp.Label()
    # try:
    code_before = (bp.SETUP_FINALLY, finally_label)
    #     [original code here]
    # finally:
    code_after = [(finally_label, None),
                  # self._locals = locals().copy()
                  (bp.LOAD_GLOBAL, 'locals'),
                  (bp.CALL_FUNCTION, 0),
                  (bp.LOAD_ATTR, 'copy'),
                  (bp.CALL_FUNCTION, 0),
                  (bp.LOAD_FAST, 'self'),
                  (bp.STORE_ATTR, '_locals'),
                  #   del self._locals['self']
                  (bp.LOAD_FAST, 'self'),
                  (bp.LOAD_ATTR, '_locals'),
                  (bp.LOAD_CONST, 'self'),
                  (bp.DELETE_SUBSCR, None),
                  (bp.END_FINALLY, None),
                  (bp.LOAD_CONST, None),
                  (bp.RETURN_VALUE, None)]
    
    f_code.code.insert(0, code_before)
    f_code.code.extend(code_after)

    # ### re-assemble
    f_code.args =  ('self',) + f_code.args
    func = new.function(f_code.to_code(), f.func_globals, f.func_name,
                        f.func_defaults, f.func_closure)
                        
    return  PersistentLocalsFunction(func)


_docpostfix = """
        
This function has been decorated with the 'persistent_locals'
decorator. You can access the dictionary of the variables in the inner
scope of the function via the 'locals' attribute.

For more information about the original function, query the self._func
attribute.
"""
        
class PersistentLocalsFunction(object):
    """Wrapper class for the 'persistent_locals' decorator.

    Refer to the docstring of instances for help about the wrapped
    function.
    """
    def __init__(self, func):
        self._locals = {}
        
        # make function an instance method
        self._func = new.instancemethod(func, self, PersistentLocalsFunction)
        
        # create nice-looking doc string for the class
        signature = inspect.getargspec(func)
        signature[0].pop(0) # remove 'self' argument
        signature = inspect.formatargspec(*signature)
        
        docprefix = func.func_name + signature
        
        default_doc = '<no docstring>'
        self.__doc__ = (docprefix + '\n\n' + (func.__doc__ or default_doc)
                        + _docpostfix)
        
    def __call__(self, *args, **kwargs):
        return self._func(*args, **kwargs)
    
    @property
    def locals(self):
        return self._locals

In some context, as for example scientific development, functions represent complex data processing algorithm that transform input data into a desired output. Internally, the function typically requires several intermediate results to be computed and stored in local variables.

As a simple toy example, consider the following function, that takes three arguments and returns True if the sum of the arguments is smaller than the product:

def is_sum_lt_prod(a,b,c):
    sum = a+b+c
    prod = a*b*c
    return sum<prod

A frequently occurring problem is that one may need to access the intermediate results at a later stage, because of the need of analyzing the detailed behavior of the algorithm, or in order to write more comprehensive tests for the algorithm.

A possible solution would be to re-define the function to return a dictionary with the desired variables in addition to the regular output, e.g.

def is_sum_lt_prod(a,b,c, internals=False):
    sum = a+b+c
    prod = a*b*c
    if internals:
        return sum<prod, {'sum': sum, 'prod': prod}
    else:
        return sum<prod

This solution keeps the existing code intact, but has several drawbacks: 1) the local variables are not accessible in case the function raises an exception, while they would be useful for debugging the algorithm; 2) it requires access to the source code of the function, which might not be feasible (e.g., if the function is defined in a third-party library) 3) it requires to modify the code of the function every time a new variable is needed.

The proposed decorator makes the local variables accessible from a read-only property of the function, locals. For example:

@persistent_locals
def is_sum_lt_prod(a,b,c):
    sum = a+b+c
    prod = a*b*c
    return sum<prod

after calling the function as, e.g., is_sum_lt_prod(2,1,2), we can access the intermediate results as is_sum_lt_prod.locals == {'a': 2, 'b': 1, 'c': 2, 'prod': 4, 'sum': 5}.

Unfortunately, the local variables in the inner scope of a function are not easily accessible. A candidate decorator should also be robust under these conditions:

  1. When the function to be decorated is defined in a closure

  2. When the original function is deleted, as in @persistent_locals def f(): pass g=f del f

  3. When the function raises an exception (it should return the locals computed up to the point the exception was raised)

  4. When the function is defined using an *args argument

1 and 2 imply that the decorator cannot refer to the global name of the function; 4 implies that the decorator cannot add new keyword arguments.

How it works

The proposed approach is to inject bytecode into the function's code to add an external try...finally statement that can be translated as:

try:
    ... old code ...
finally:
    self.locals = locals().copy()
    del self.locals['self']

The function is wrapped in a class so that it can access a stable namespace: there are all sorts of complications in referring from within a function to the function itself. For example, referring to an attribute directly, e.g., f.locals results in the Python interpreter to look for the name f in the namespace, and therefore moving the function, e.g. with

g = f
del f

would break g. There are even more problems for functions defined in a closure.

Alternative implementation

There is a simpler and arguably cleaner implementation, based on an idea by Andrea Maffezoli, that does not require the external library byteplay. This approach works by defining a profile tracer function, which has access to the internal frame of a function, and is called at the entry and exit of functions and when an exception is called. Unfortunately, this solution interferes with profilers:

# persistent_locals2 has been co-authored with Andrea Maffezzoli
class persistent_locals2(object):
    def __init__(self, func):
        self._locals = {}
        self.func = func

    def __call__(self, *args, **kwargs):
        def tracer(frame, event, arg):
            if event=='return':
                self._locals = frame.f_locals.copy()

        # tracer is activated on next call, return or exception
        sys.setprofile(tracer)
        try:
            # trace the function call
            res = self.func(*args, **kwargs)
        finally:
            # disable tracer and replace with old one
            sys.setprofile(None)
        return res

    def clear_locals(self):
        self._locals = {}

    @property
    def locals(self):
        return self._locals

This does not work

It would be better to get rid of the wrapping class, but one needs to find a way to refer to the function itself in a stable way, as explained above. I tried modifying f.func_globals with a custom dictionary which keeps a reference to f.func_globals, adding a static element to f, but this does not work as the Python interpreter does not call the func_globals dictionary with Python calls but directly with PyDict_GetItem (see here). It is thus impossible to re-define __getitem__ to return f as needed. Ideally, one would like to define a new closure for the function with a cell variable containing the reference, but this is impossible as far as I can tell.