Decorator to expose local variables of a function after execution (Python recipe) by Pietro Berkes

ActiveState Code (http://code.activestate.com/recipes/577283/)

In some context, as for example scientific development, functions represent complex data processing algorithm that transform input data into a desired output. Internally, the function typically requires several intermediate results to be computed and stored in local variables.

As a simple toy example, consider the following function, that takes three arguments and returns True if the sum of the arguments is smaller than the product:

def is_sum_lt_prod(a,b,c):
    sum = a+b+c
    prod = a*b*c
    return sum<prod

A frequently occurring problem is that one may need to access the intermediate results at a later stage, because of the need of analyzing the detailed behavior of the algorithm, or in order to write more comprehensive tests for the algorithm.

A possible solution would be to re-define the function to return a dictionary with the desired variables in addition to the regular output, e.g.

def is_sum_lt_prod(a,b,c, internals=False):
    sum = a+b+c
    prod = a*b*c
    if internals:
        return sum<prod, {'sum': sum, 'prod': prod}
    else:
        return sum<prod

This solution keeps the existing code intact, but has several drawbacks: 1) the local variables are not accessible in case the function raises an exception, while they would be useful for debugging the algorithm; 2) it requires access to the source code of the function, which might not be feasible (e.g., if the function is defined in a third-party library) 3) it requires to modify the code of the function every time a new variable is needed.

The proposed decorator makes the local variables accessible from a read-only property of the function, locals. For example:

@persistent_locals
def is_sum_lt_prod(a,b,c):
    sum = a+b+c
    prod = a*b*c
    return sum<prod

after calling the function as, e.g., is_sum_lt_prod(2,1,2), we can access the intermediate results as is_sum_lt_prod.locals == {'a': 2, 'b': 1, 'c': 2, 'prod': 4, 'sum': 5}.

Unfortunately, the local variables in the inner scope of a function are not easily accessible. A candidate decorator should also be robust under these conditions:

1. When the function to be decorated is defined in a closure

2. When the original function is deleted, as in @persistent_locals def f(): pass g=f del f

3. When the function raises an exception (it should return the locals computed up to the point the exception was raised)

4. When the function is defined using an *args argument

1 and 2 imply that the decorator cannot refer to the global name of the function; 4 implies that the decorator cannot add new keyword arguments.

How it works

The proposed approach is to inject bytecode into the function's code to add an external try...finally statement that can be translated as:

try:
    ... old code ...
finally:
    self.locals = locals().copy()
    del self.locals['self']

The function is wrapped in a class so that it can access a stable namespace: there are all sorts of complications in referring from within a function to the function itself. For example, referring to an attribute directly, e.g., f.locals results in the Python interpreter to look for the name f in the namespace, and therefore moving the function, e.g. with

g = f
del f

would break g. There are even more problems for functions defined in a closure.

Alternative implementation

There is a simpler and arguably cleaner implementation, based on an idea by Andrea Maffezoli, that does not require the external library byteplay. This approach works by defining a profile tracer function, which has access to the internal frame of a function, and is called at the entry and exit of functions and when an exception is called. Unfortunately, this solution interferes with profilers:

# persistent_locals2 has been co-authored with Andrea Maffezzoli
class persistent_locals2(object):
    def __init__(self, func):
        self._locals = {}
        self.func = func

    def __call__(self, *args, **kwargs):
        def tracer(frame, event, arg):
            if event=='return':
                self._locals = frame.f_locals.copy()

        # tracer is activated on next call, return or exception
        sys.setprofile(tracer)
        try:
            # trace the function call
            res = self.func(*args, **kwargs)
        finally:
            # disable tracer and replace with old one
            sys.setprofile(None)
        return res

    def clear_locals(self):
        self._locals = {}

    @property
    def locals(self):
        return self._locals

This does not work

It would be better to get rid of the wrapping class, but one needs to find a way to refer to the function itself in a stable way, as explained above. I tried modifying f.func_globals with a custom dictionary which keeps a reference to f.func_globals, adding a static element to f, but this does not work as the Python interpreter does not call the func_globals dictionary with Python calls but directly with PyDict_GetItem (see here). It is thus impossible to re-define __getitem__ to return f as needed. Ideally, one would like to define a new closure for the function with a cell variable containing the reference, but this is impossible as far as I can tell.