Decorator for BindingConstants at compile time (Python recipe) by Raymond Hettinger

ActiveState Code (http://code.activestate.com/recipes/277940/)

Very cool ... One suggestion. This is a very cool decorator Raymond. I do have one small suggestion. Instead of hard-coding the various bytecode numbers, I'd import symbols from the opcode module just to future-proof it a little:

from opcode import opmap, HAVE_ARGUMENT, EXTENDED_ARG
LOAD_GLOBAL = opmap['LOAD_GLOBAL']
LOAD_CONST = opmap['LOAD_CONST']

Skip

Done. Incorporated the suggested change. Thanks!

Exceedingly Cool. I have a voxel rendering class, with several nested loops. It needs a very tight inner loop, and as such I have 17 _len = len style optimisations to avoid excessive dict lookups.

Thanks Raymond, this is brilliant stuff, and has tidied up my code very nicely indeed!

compile time? Since the constants are bound when the module is imported, the recipe should be named "BindingConstants at import time" or something like that, imo.

Metaclass makes things more handy. Here's a metaclass that calls make_constants for all methods of a class.

class OptimizingMetaclass(type):
    def __init__(cls, name, bases, dict):
        super(OptimizingMetaclass, cls).__init__(name, bases, dict)

        import types
        for name, attribute in dict.items():
            if type(attribute) is types.FunctionType:
                dict[name] = _make_constants(attribute)

I know, decorators make you think about the right place to put them - but if it doesn't break the code, there should be nothig bad about optimizing everywhere.

BTW, if you still want to keep it configurable, try:

def build_optimizing_metaclass(builtin_only=False, stoplist=[], verbose=False):
    from types import FunctionType
    class _OptimizingMetaclass(type):
        def __init__(cls, name, bases, dict):
            super(_OptimizingMetaclass, cls).__init__(name, bases, dict)

            for name, attribute in dict.items():
                if type(attribute) is FunctionType:
                    dict[name] = _make_constants(attribute, builtin_only, stoplist, verbose)

    return _OptimizingMetaclass

__metaclass__ = build_optimizing_metaclass(verbose=True)

bind_all misses imports. bind_all misses

from types import FunctionType, ClassType

and, I guess, bind_all simplifies my last comment to

class OptimizingMetaclass(type):
    def __init__(cls, name, bases, dict):
        super(OptimizingMetaclass, cls).__init__(name, bases, dict)
        bind_all(cls)

def build_optimizing_metaclass(builtin_only=False, stoplist=[], verbose=False):
    class _OptimizingMetaclass(type):
        def __init__(cls, name, bases, dict):
            super(_OptimizingMetaclass, cls).__init__(name, bases, dict)
            bind_all(cls, builtin_only, stoplist, verbose)

    return _OptimizingMetaclass

getting into closures as well. I use closures fairly frequently, so I've found it useful to add code to this to also prebind constants in code objects in func_code.co_consts and functions under closures.

The code objects part is pretty simple; just split out the part of _make_constants that deals only with the code object into a separate function (say, code_make_constants) which takes the code object co, the environment dict env, stoplist, and verbose as arguments and do

newconsts = [(isintance(c, CodeType)
               and code_make_constants(c, env, stoplist, verbose)
               or c)
             for c in co.co_consts]

somewhere up at the top.

Binding the constants on functions in closures is a little more tricky, but it can be done using the get_cell_value and change_cell_value functions from some recipes I've posted.

I have this code block in _make_constants, before the function object is created:

for cell in (f.func_closure or []):
    value = get_cell_value(cell)
    if isinstance(value, FunctionType):
        change_cell_value(
            cell,
            _make_constants(value, builtin_only, stoplist, verbose)
        )

For those that wonder why I would do such a thing: Optimizing code objects in .co_consts is useful when you have subfunctions, i.e.

def foo(n):
    def bar():
        return n
    return bar

bar will show up as a code object in foo.func_code.co_consts. Optimizing functions in a func_closure tuple is useful when you use wrapper functions in decorators, i.e.:

def the_decorator(f):
    def wrapper(*args, **kwargs):
        result = f(*args, **kwargs)
        return do_something_with(result)
    return wrapper

@the_decorator
def baz():
    return do_stuff()

With this pattern, the original baz code will not be accessible except in a cell in baz.func_closure.

Another side effect. I used to wrap all the functions and classes in a "main" function, which returned a dictionary of names to be exported for two reasons:

1. speed (LOAD_DEREF was faster than LOAD_GLOBAL)

2. less namespace clutter (ie all the imported modules did not show up in the module's globals)

The first part is handled by the recipe. The second part can easily be dealt with by deleting the imported modules after running the bind_all function (assuming that all LOAD_GLOBALS have been replaced with LOAD_CONST, of course :)

There's a typo in sample function, ininstance should be isinstance

I think i have not get the stoplist usage, i have tried this:

@binding_constants.make_constants(verbose=True, stoplist=[util,])

but still i get util module optimized, so i have introduced in the decorator this print for debug.

if verbose: print name, stoplist, name in stoplist

and i get this:

util [<module 'util' from 'util.pyc'>] False

util --> <module 'util' from 'util.pyc'>

why is the util module still optimized?

Thanks.

See recipe 576904 for a Python 3 compatible version.

I am bothered by this code:

    if opcode in (EXTENDED_ARG, STORE_GLOBAL):
        return f    # for simplicity, only optimize common cases

If the function being optimized contains a STORE_GLOBAL, the process bails and does nothing to that function. But the implication of finding a STORE_GLOBAL go beyond the one function. Say the module has a statement SOME_GLOB = 0 and we apply build_all(). In any function in the module that lacks a STORE_GLOBAL opcode, any reference to SOME_GLOB will be optimized to a reference to constant 0 instead. Then this function contains global SOME_GLOB; SOME_GLOB = 99. So this function is not optimized -- but all the others are! In effect, this function's assignment to SOME_GLOB is nullified. "I know I'm setting the global but the other functions act like it was never set..."

It seems to me that if any function in a module contains STORE_GLOBAL, then that global name needs to never be optimized. What am I not getting?