Bound Inner Classes « Python recipes

This recipe provides the class decorator BoundInnerClass. The decorator makes inner classes symmetric with method calls. Functions declared inside classes become "methods", and when you call them through an object they automatically get a reference to "self". The BoundInnerClass decorator makes this work for inner classes: an inner class decorated with BoundInnerClass gets a reference to that same (now "outer") object passed in automatically to the inner class's __init__.

The recipe works unchanged in Python 2.6 and 3.1, and is licensed using the Zlib license.

      """
http://code.activestate.com/recipes/577070-bound-inner-classes/

Copyright (C) 2010-2011 by Alex Martelli and Larry Hastings

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
"""

__all__ = ["BoundInnerClass", "UnboundInnerClass"]

import weakref

class _Worker(object):
    def __init__(self, cls):
        self.cls = cls

    def __get__(self, outer, outer_class):
        if not outer:
            return self.cls

        name = self.cls.__name__

        # if we're already cached in outer, use that version
        # (but don't use getattr, that would call __get__ recursively)
        if name in outer.__dict__:
            return outer.__dict__[name]

        wrapper_bases = [self.cls]

        # iterate over cls's bases and look in outer to see
        # if any have bound inner classes in outer.
        # if so, multiply inherit from the bound inner version(s).
        multiply_inherit = False

        for base in self.cls.__bases__:
            # if we can't find a bound inner base,
            # add the original unbound base instead.
            # this is harmless but helps preserve the original MRO.
            inherit_from = base

            # if we inherit from a boundinnerclass from another outer class,
            # we might have the same name as a legitimate base class.
            # but if we look it up, we'll call __get__ recursively... forever.
            # if the name is the same as our own name, there's no way it's a
            # bound inner class we need to inherit from, so just skip it.
            if base.__name__ != name:
                bound_inner_base = getattr(outer, base.__name__, None)
                if bound_inner_base:
                    bases = getattr(bound_inner_base, "__bases__", (None,))
                    # the unbound class is always the first base of
                    # the bound inner class.
                    if bases[0] == base:
                        inherit_from = bound_inner_base
                        multiply_inherit = True
            wrapper_bases.append(inherit_from)

        Wrapper = self._wrap(outer, wrapper_bases[0])
        Wrapper.__name__ = name

        # Assigning to __bases__ is startling, but it's the only way to get
        # this code working simultaneously in both Python 2 and Python 3.
        if multiply_inherit:
            Wrapper.__bases__ = tuple(wrapper_bases)

        # cache in outer, but only if we're replacing the class
        # (if we're a bound inner class of A, and there's class B(A),
        #  and in class B we're some other kind of attribute,
        #  *don't* stomp on it!)
        if getattr(outer.__class__, name, None) == self.cls:
            setattr(outer, name, Wrapper)
        return Wrapper

class BoundInnerClass(_Worker):
    def _wrap(self, outer, base):
        wrapper_self = self
        assert outer
        outer_weakref = weakref.ref(outer)
        class Wrapper(base):
            def __init__(self, *args, **kwargs):
                wrapper_self.cls.__init__(self,
                                          outer_weakref(), *args, **kwargs)

            # give the bound inner class a nice repr
            # (but only if it doesn't already have a custom repr)
            if wrapper_self.cls.__repr__ is object.__repr__:
                def __repr__(self):
                    return "".join([
                        "<",
                        self.__module__,
                        ".",
                        self.__class__.__name__,
                        " object bound to ",
                        repr(outer_weakref()),
                        " at ",
                        hex(id(self)),
                        ">"])
        return Wrapper

class UnboundInnerClass(_Worker):
    def _wrap(self, outer, base):
        class Wrapper(base):
            pass
        return Wrapper


# The code in this "if" statement will only execute if you run the module
# directly; it won't run if you "import" this code into your own programs.
if __name__ == "__main__":
    class Outer(object):
        @BoundInnerClass
        class Inner(object):
            def __init__(self, outer):
                self.outer = outer

        @BoundInnerClass
        class SubclassOfInner(Inner.cls):
            def __init__(self, outer):
                super(Outer.SubclassOfInner, self).__init__()
                assert self.outer == outer

        @BoundInnerClass
        class SubsubclassOfInner(SubclassOfInner.cls):
            def __init__(self, outer):
                super(Outer.SubsubclassOfInner, self).__init__()
                assert self.outer == outer

        @BoundInnerClass
        class Subclass2OfInner(Inner.cls):
            def __init__(self, outer):
                super(Outer.Subclass2OfInner, self).__init__()
                assert self.outer == outer

        class RandomUnboundInner(object):
            def __init__(self):
                super(Outer.RandomUnboundInner, self).__init__()
                pass

        @BoundInnerClass
        class MultipleInheritanceTest(SubclassOfInner.cls,
                     RandomUnboundInner,
                     Subclass2OfInner.cls):
            def __init__(self, outer):
                super(Outer.MultipleInheritanceTest, self).__init__()
                assert self.outer == outer

        @UnboundInnerClass
        class UnboundSubclassOfInner(Inner.cls):
            pass


    def tests():
        assert outer.Inner == outer.Inner
        assert isinstance(inner, outer.Inner)
        assert isinstance(inner, Outer.Inner)

        assert isinstance(subclass, Outer.SubclassOfInner)
        assert isinstance(subclass, outer.SubclassOfInner)
        assert isinstance(subclass, Outer.Inner)
        assert isinstance(subclass, outer.Inner)

        assert isinstance(subsubclass, Outer.SubsubclassOfInner)
        assert isinstance(subsubclass, outer.SubsubclassOfInner)
        assert isinstance(subsubclass, Outer.SubclassOfInner)
        assert isinstance(subsubclass, outer.SubclassOfInner)
        assert isinstance(subsubclass, Outer.Inner)
        assert isinstance(subsubclass, outer.Inner)

    import itertools

    for order in itertools.permutations([1, 2, 3]):
        outer = Outer()
        # This strange "for" statement lets us test every possible order of
        # initialization for the "inner" / "subclass" / "subsubclass" objects.
        for which in order:
            if which == 1: inner = outer.Inner()
            elif which == 2: subclass = outer.SubclassOfInner()
            elif which == 3: subsubclass = outer.SubsubclassOfInner()
        tests()

    multiple_inheritance_test = outer.MultipleInheritanceTest()
    assert outer.MultipleInheritanceTest.mro() == [
        # bound inner class, notice lowercase-o "outer"
        outer.MultipleInheritanceTest,
        # unbound inner class, notice uppercase-o "Outer"
        Outer.MultipleInheritanceTest,
        outer.SubclassOfInner, # bound
        Outer.SubclassOfInner, # unbound
        Outer.RandomUnboundInner, # etc.
        outer.Subclass2OfInner,
        Outer.Subclass2OfInner,
        outer.Inner,
        Outer.Inner,
        object
    ]

    unbound = outer.UnboundSubclassOfInner()
    assert outer.UnboundSubclassOfInner.mro() == [
        outer.UnboundSubclassOfInner,
        Outer.UnboundSubclassOfInner,
        outer.Inner,
        Outer.Inner,
        object
    ]

    class InnerChild(outer.Inner):
        pass

    inner_child = InnerChild()

    isinstance(inner_child, Outer.Inner)
    isinstance(inner_child, InnerChild)
    isinstance(inner_child, outer.Inner)

      

Overview

One minor complaint about Python is that inner classes don't have access to the outer object. Consider this Python code:

class Outer(object):
    def method(self):
        pass
    class Inner(object):
        def __init__(self):
            pass
o = Outer()
o.method()
i = o.Inner()

When o.method is called, Python automatically passes in the o object as the first parameter (generally called "self"). But there's no built-in way for the o.Inner object being constructed to automatically get a reference to the o Outer object. If you need one, you must pass it in yourself like so:

class Outer(object):
    def method(self):
        pass
    class Inner(object):
        def __init__(self, outer):
            self.outer = outer
o = Outer()
o.method()
i = o.Inner(o)

It seems redundant to have to pass in o as an argument to the Inner constructor. You don't have to pass in o explicitly to method calls; why should you have to pass it in explicitly to inner classes?

I posted a question online to see if anyone could propose a way to automatically pass the outer instance into the inner class's __init__. Alex Martelli responded with a brilliant solution: a class decorator, itself implemented as a class providing a descriptor, which internally subclasses the passed-in class and passes in a reference to the object to the class's __init__. The result has a pleasing symmetry with normal callables and "bound method objects". After trading some edits back and forth, Alex and I are pleased to present our solution below, which I've called "bound inner classes".

Using Bound Inner Classes

Let's modify the above example to use the recipe:

from boundinnerclass import BoundInnerClass
class Outer(object):
    def method(self):
        pass
    @BoundInnerClass
    class Inner(object):
        def __init__(self, outer):
            self.outer = outer
o = Outer()
o.method()
i = o.Inner()

All we did:

Import the recipe.
Decorate the inner class with BoundInnerClass.
Stop passing in the outer object o to the constructor. It's now passed in automatically!

An Example Use Case

In case you're wondering "why would I ever care about this", consider the following only-slightly-contrived example.

Let's say you have a class that represents performing some task. The class supports the __call__ method, making it directly callable to do its work. Something like:

class Worker(object):
    def __init__(self, ...):
        ...
    def __call__(self):
        # do work here
        ...

Now let's say you get more sophisticated one day, and want to wrap all that up in a thread. The most convenient way to create threads in Python is by subclassing the threading.Thread class. However, the documentation for threading.Thread tells you how to override the class's __init__ and run methods, then states:

No other methods (except for the constructor) should be overridden in a subclass. In other words, only override the __init__() and run() methods of this class.

Because of this restriction, it's generally best not to subclass threading.Thread in your own general-purpose classes. Instead, you should write a special class, whose whole purpose in life is to subclass threading.Thread and override just those two methods.

You could easily write it like this:

class Worker(object):
    ...
class WorkerThread(threading.Thread):
    def __init__(self, worker):
        super(WorkerThread, self).__init__()
        self.worker = worker
    def run(self):
        self.worker()

worker = Worker()
thread = WorkerThread(worker)

Clearly this works. But I suggest it can be improved. Conceptually, WorkerThread really has no utility apart from servicing the Worker class. So I suggest it should be an inner class:

class Worker(object):
    ...
    class Thread(threading.Thread):
        def __init__(self, worker):
            super(WorkerThread, self).__init__()
            self.worker = worker
        def run(self):
            self.worker()

But now your code for initialization looks a bit silly:

worker = Worker()
thread = worker.Thread(worker)

Here you are, calling an attribute of worker, and yet you must also pass in worker? Why doesn't it automatically know about worker?

You can hide this redundancy by making a method of Worker doing the redundant passing-in for you:

class Worker(object):
    ...
    class Thread(threading.Thread):
        ...
    def create_thread(self):
        return self.Thread(self)

worker = Worker()
thread = worker.create_thread()

But this really creates more problems than it solves. You've got two callable things when one should suffice. This clutters up the namespace of the class, not to mention cluttering your mind. If someone wants to create one of these thread objects, they must use one interface, but if they want to do an isinstance() test they must use the other.

Decorating the inner Thread class with BoundInnerClass solves this problem:

class Worker(object):
    ...
    @BoundInnerClass
    class Thread(threading.Thread):
        def __init__(self, worker):
            ...

worker = Worker()
thread = worker.Thread()

Written this way, the Thread constructor is now automatically called with the "outer" object, in this case the worker object, without anyone having to explicitly--redundantly--pass it in. And everything works as you'd expect: you can tear off the worker.Thread callable and call it as a standalone function, and you can run an isinstance() test which will report success.

Inheritance

Bound inner classes get slightly complicated when mixed with inheritance. It's not all that difficult, you merely need to obey the following rules:

A bound inner class can inherit normally from any unbound class.
To subclass from a bound inner class while inside the same outer class, or when referencing the inner class from the outer class (as opposed to an instance of the outer class), you must actually subclass from classname.cls. (This is because inside the outer class, the "class" you see is actually an instance of a BoundInnerClass object.)
All classes that inherit from a bound inner class must always call call the superclass's __init__. You don't need to pass in the outer parameter; it'll be automatically passed in to the superclass's __init__ as before.
An inner class that inherits from a bound inner class, and which also wants to be bound to the outer object, should be decorated with BoundInnerClass.
An inner class that inherits from a bound inner class, but isn't interested in being bound to the outer object, should be decorated with this recipe's other decorator, UnboundInnerClass.

Restating the last two rules: every class that descends from any BoundInnerClass should be decorated with either BoundInnerClass or UnboundInnerClass.

Here's a simple example of inheritance with bound inner classes:

from boundinnerclass import BoundInnerClass
class Outer(object):
    @BoundInnerClass
    class Inner(object):
        def __init__(self, outer):
            self.outer = outer
    @UnboundInnerClass
    class ChildOfInner(Inner.cls):
        def __init__(self):
            super(Outer.ChildOfInner, self).__init__()

o = Outer()
i = o.ChildOfInner()

We followed the rules:

Inner inherits from object; since object isn't a bound inner class, there are no special rules about inheritance Inner needs to obey.
ChildOfInner inherits from Inner.cls, not Inner.
Since ChildOfInner inherits from a BoundInnerClass, it must be decorated with either BoundInnerClass or UnboundInnerClass. It doesn't want the outer object passed in, so it's decorated with UnboundInnerClass.
ChildOfInner.__init__ calls the __init__ of its super().

Note that, because ChildOfInner is decorated with UnboundInnerClass, it doesn't take a outer argument. Nor does it pass one in when it it calls super().__init__. But when the constructor for Inner is called, the correct outer argument appears like magic. Thanks, BoundInnerClass!

If you wanted ChildOfInner to also get the outer argument passed in to its __init__, just decorate it with BoundInnerClass instead of UnboundInnerClass like so:

from boundinnerclass import BoundInnerClass
class Outer(object):
    @BoundInnerClass
    class Inner(object):
        def __init__(self, outer):
            self.outer = outer
    @BoundInnerClass
    class ChildOfInner(Inner.cls):
        def __init__(self, outer):
            super(Outer.ChildOfInner, self).__init__()
            assert self.outer == outer

o = Outer()
i = o.ChildOfInner()

You can see more complex examples of using inheritance with BoundInnerClass (and UnboundInnerClass) in the recipe's test code.

Other Notes

If you refer to a bound inner class directly from the outer class, rather than using the outer instance, you get the original class. This means that references to Outer.Inner are consistent, and it's a base class of all the bound inner classes. This also means that if you attempt to construct one without using an outer instance, you must pass in the outer parameter by hand, just as you would have to pass in the self parameter by hand when calling an unbound method.
If you refer to a bound inner class from an outer instance, you get a subclass of the original class.
Bound classes are cached in the outer object, which both provides a small speedup and ensures that isinstance relationships are consistent.
You must not rename inner classes decorated with this recipe! The implementation of BoundInnerClass looks up the bound inner class in the outer object by name in several places. Adding aliases to bound inner classes is harmless, but the original attribute name must always work.
Bound inner classes from different objects are different classes. This is symmetric with bound methods; if you have two objects "a" and "b" that are instances of the same class, a.BoundInnerClass != b.BoundInnerClass, just as a.method != b.method.
The binding only goes one level deep; if you had an inner class C inside another inner class B inside a class A, the constructor for C would be called with the B object, not the A object.
Similarly, if you have a bound inner class B inside a class A, and another bound inner class D inside a class C, and D inherits from B, the constructor for D will be called with the B object but not the A object. When D calls super().__init__ it'll have to fill in the outer parameter by hand.
There is a race condition in the implementation: if you access a bound inner class through an outer instance from two separate threads, and the bound inner class was not previously cached, the two threads may get different (but equivalent) bound inner class objects, and only one of those instances will get cached on the outer object. This could lead to confusion and possibly cause bugs. For example, you could have two objects that would be considered equal if they were instances of the same bound inner class, but would not be considered equal if instantiated by different instances of that same bound inner class. There's an easy workaround for this problem: access the bound inner class from the __init__ of the outer class, which should allow the code to cache the bound inner class instance before a second thread could ever get a reference to the outer object.

An Alternate Approach

I've contributed a second implementation of bound inner classes as a new recipe, here:

http://code.activestate.com/recipes/577623-bound-inner-classes-using-an-alternate-approach/

While that implementation is interesting, I think this original approach remains preferable for production code.

Revision History

Revision 1 was the original revision.

Revision 2 has been lost to the hoary mists of time.

Revision 3 removed the per-decorator cache mapping outer objects to the bound inner form of the decorated class. Instead, a reference to the bound inner class is stored in the object, occluding the original decorated class. This helps with object cleanup; storing this cache in the decorator meant the bound inner classes and the outer objects were effectively immortal. (They'd only get collected if you dropped all references to the outer object's class, and even then there was a long reference cycle.) Also, added warning about the surprising "bound inner subclasses don't appear to be children of the bound inner parent class" behavior.

Revision 4 cleaned up the interaction between bound inner classes and inheritance, by adding the "multiply inherit from bound inner version(s)" code and UnboundInnerSubclass. Multiple inheritance now works cleanly, passing in the "outer" argument to the bound inner base classes is now automatic, and bound inner subclasses now inherit from their bound inner parent classes (which fixes some surprising isinstance behavior from previous revisions).

Revision 5 changed BoundInnerClass to keep only a weakref to the outer object. Without this change, BoundInnerClasses created a reference cycle (o -> boundinnerclass, boundinnerclass -> o) which made object cleanup unnecessarily harder. Also, UnboundInnerSubclass was renamed to UnboundInnerClass; the former nomenclature was needlessly asymmetric. Third, all references to parent were renamed to outer. I think that's clearer, though it doesn't affect any of the interfaces. Finally, an explicit Zlib license was added to the source code. (The recipe is marked as being MIT license; ActiveState doesn't provide a choice for the Zlib license. Don't worry, Zlib is even less restrictive than MIT; it explicitly says that you don't need to preserve the copyright notice in compiled form.)

Revision 6 fixed a bug: if a bound inner class inherited from a class with the same name, the multiple-inheritance check would go into an infinite __get__ recursion loop and blow the stack. To fix it, the multiple-inheritance code now skips classes with the same name as the currently-being-bound class, as there's no way said class could be an inner class inside the same outer object.

Revision 7 made no bugfixes and added no features. It was purely a code readability edit, based on comments Alex made regarding the recipe in-person at PyCon 2011.

Revision 8 fixed an obscure bug: if class A defines bound inner class BIC, and class B descends from A but defines BIC to be something else (a method, a scalar, etc) but still uses class A's BIC (by calling super()), and b is an object of type B, then BoundInnerClass would overwrite b.BIC in with the bound inner version of A.BIC. BoundInnerClass definitely shouldn't overwrite b.BIC in this case. Unfortunately that means we have nowhere to store the bound version of A.BIC, so this will break some isinstance() uses--there's really no good solution here.

Tags: class_decorator, descriptor

◄	Python recipes (4591)	►
◄	Larry Hastings's recipes (2)	►

Bound Inner Classes (Python recipe) by Larry Hastings
ActiveState Code (http://code.activestate.com/recipes/577070/)

Overview

Using Bound Inner Classes

An Example Use Case

Inheritance

Other Notes

An Alternate Approach

Revision History

Tags

Required Modules

Other Information and Tasks

Accounts

Code Recipes

Feedback & Information

ActiveState

Bound Inner Classes (Python recipe) by Larry Hastings ActiveState Code (http://code.activestate.com/recipes/577070/)

Overview

Using Bound Inner Classes

An Example Use Case

Inheritance

Other Notes

An Alternate Approach

Revision History

Tags

Required Modules

Other Information and Tasks

Accounts

Code Recipes

Feedback & Information

ActiveState

Bound Inner Classes (Python recipe) by Larry Hastings
ActiveState Code (http://code.activestate.com/recipes/577070/)