Welcome, guest | Sign In | My Account | Store | Cart
7

Anyone who's used reload() on a module that defines a class in the interactive interpreter must have experienced the frustration of then running around and making sure that all instances are updated to be instances of the new rather than the old class.

This metaclass tries to help with this.

Python, 69 lines
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
import weakref, inspect

class MetaInstanceTracker(type):
    def __new__(cls, name, bases, ns):
        t = super(MetaInstanceTracker, cls).__new__(cls, name, bases, ns)
        t.__instance_refs__ = []
        return t
    def __instances__(self):
        instances = [(r, r()) for r in self.__instance_refs__]
        instances = filter(lambda (x,y): y is not None, instances)
        self.__instance_refs__ = [r for (r, o) in instances]
        return [o for (r, o) in instances]
    def __call__(self, *args, **kw):
        instance = super(MetaInstanceTracker, self).__call__(*args, **kw)
        self.__instance_refs__.append(weakref.ref(instance))
        return instance

class InstanceTracker:
    __metaclass__ = MetaInstanceTracker

class MetaAutoReloader(MetaInstanceTracker):
    def __new__(cls, name, bases, ns):
        new_class = super(MetaAutoReloader, cls).__new__(
            cls, name, bases, ns)
        f = inspect.currentframe().f_back
        for d in [f.f_locals, f.f_globals]:
            if d.has_key(name):
                old_class = d[name]
                for instance in old_class.__instances__():
                    instance.change_class(new_class)
                    new_class.__instance_refs__.append(
                        weakref.ref(instance))
                # this section only works in 2.3
                for subcls in old_class.__subclasses__():
                    newbases = ()
                    for base in subcls.__bases__:
                        if base is old_class:
                            newbases += (new_class,)
                        else:
                            newbases += (base,)
                    subcls.__bases__ = newbases
                break
        return new_class

class AutoReloader:
    __metaclass__ = MetaAutoReloader
    def change_class(self, new_class):
        self.__class__ = new_class

class Bar(AutoReloader):
    pass

class Baz(Bar):
    pass

b = Bar()
b2 = Baz()

class Bar(AutoReloader):
    def meth(self, arg):
       print arg

if __name__ == '__main__':
    # now b is "upgraded" to the new Bar class:
    b.meth(1)
    # new in 2.3, Baz instances join the fun:
    b2.meth(2)
    # new Baz() instances now play too:
    Baz().meth(3)

The problem this code is meant to help with is hopefully familiar.

You're editing a Python module in emacs (or vim or notepad or whatever). Let's say at some point it looks like this: <pre> -------- # mod.py -------- class Foo(object): def meth1(self, arg):

print arg

</pre> In another window, you have an interactive interpreter running to test your code: <pre>

>>> import mod
>>> f = mod.Foo()
>>> f.meth1(1)
1
</pre>
That seems to be working.  Good.  Now you edit mod.py to add another method:
<pre>
-------- # mod.py --------
class Foo(object):
    def meth1(self, arg):
        print arg
    def meth2(self, arg):
<h5>        print -arg</h5>

</pre> Head back to the test session: <pre>

>>> reload(mod)
&lt;module 'mod' from 'mod.pyc'&gt;
>>> f.meth2(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
AttributeError: 'Foo' object has no attribute 'meth2'
</pre>
Argh!  You forgot that 'f' was an instance of the *old* mod.Foo!

There are two things you can do about this: 1) regenerate the instance: <pre>

>>> f = meth.Foo()
>>> f.meth2(1)
-1
</pre>
or 2) assign to f.__class__:
<pre>
>>> f.__class__ = mod.Foo
>>> f.meth2(1)
-1
</pre>
1) works easily enough in simple situations, but can become very tedious.  2) can be automated, which is what my code does.

The first class is a metaclass that tracks instances of its instances (I hope you have your brain-splatter protection set up).

As metaclasses go, this isn't too complicated. New classes of this metatype get an extra __instance_refs__ class variable (used to store weak references to instances) and an __instances__ method (which strips dead references out of the __instance_refs__ list and returns real references to the still live instances). When a class of metatype MetaInstanceTracker is instantiated, a weak reference to the instances is stored in the __instance_refs__ list.

When the definition of a class of metatype MetaAutoReloader is executed, the namespace of the definition is examined to see if a class of the same name already exists. If it does, then it is assumed that instead of a class defintion, this is a class REdefinition, and all instances of the OLD class (MetaAutoReloader inherits from MetaInstanceTracker, so they can easily be found) are updated to the NEW class.

There should be more error checking (e.g. that old_class is an InstanceTracker, change_class can fail). This was omitted through a mixture of a desire for clarity and laziness.

11 comments

Michael Hudson (author) 13 years, 9 months ago  # | flag

This was supposed to go in the discussion... Well, I thought the "Add" button meant add a discussion item, not "Add recipe to cookbook".

Anyway...

The problem this code is meant to help with is hopefully familiar.

You're editing a Python module in emacs (or vim or notepad or whatever). Let's say at some point it looks like this:

-------- # mod.py --------
class Foo(object):
    def meth1(self, arg):
        print arg
--------------------------

In another window, you have an interactive interpreter running to test your code:

>>> import mod
>>> f = mod.Foo()
>>> f.meth1(1)
1

That seems to be working. Good. Now you edit mod.py to add another method:

-------- # mod.py --------
class Foo(object):
    def meth1(self, arg):
        print arg
    def meth2(self, arg):
        print -arg
--------------------------

Head back to the test session:

>>> reload(mod)
&lt;module 'mod' from 'mod.pyc'&gt;
>>> f.meth2(1)
Traceback (most recent call last):
  File "", line 1, in ?
AttributeError: 'Foo' object has no attribute 'meth2'

Argh! You forgot that 'f' was an instance of the old mod.Foo!

There are two things you can do about this: 1) regenerate the instance:

>>> f = meth.Foo()
>>> f.meth2(1)
-1

or 2) assign to f.__class__:

>>> f.__class__ = mod.Foo
>>> f.meth2(1)
-1

1) works easily enough in simple situations, but can become very tedious. 2) can be automated, which is what my code does.

The first class is a metaclass that tracks instances of its instances (I hope you have your brain-splatter protection set up).

As metaclasses go, this isn't too complicated. New classes of this metatype get an extra __instance_refs__ class variable (used to store weak references to instances) and an __instances__ method (which strips dead references out of the __instance_refs__ list and returns real references to the still live instances). When a class of metatype MetaInstanceTracker is instantiated, a weak reference to the instances is stored in the __instance_refs__ list.

When the definition of a class of metatype MetaAutoReloader is executed, the namespace of the definition is examined to see if a class of the same name already exists. If it does, then it is assumed that instead of a class defintion, this is a class REdefinition, and all instances of the OLD class (MetaAutoReloader inherits from MetaInstanceTracker, so they can easily be found) are updated to the NEW class.

Problems with this code:

1) there should be more error checking (e.g. that old_class is an InstanceTracker, change_class can fail). This was omitted through a mixture of a desire for clarity and laziness.

(comment continued...)

Michael Hudson (author) 13 years, 9 months ago  # | flag

(...continued from previous comment)

2) bad interaction with inheritance. Ideally, on redefinition a class should tweak subclasses of the old class to inherit from the new class. But in Python 2.2.X at least, assignment to __bases__ is not allowed, so this is impossible. This might change in Python 2.3.

If this comment could magically become the discussion part of this recipe, I'd appreciate it...

Michael Hudson (author) 13 years, 9 months ago  # | flag

Never mind... I found the edit recipe bit.

Michael Hudson (author) 13 years, 4 months ago  # | flag

__bases__ will be assignable in 2.3. I wrote a patch to allow assignment to __bases__ of new style classes in Python 2.3 and it was accepted and checked in, so it will be possible to update this recipe to play nicely with inheritance in 2.3.

Raymond Hettinger 12 years, 2 months ago  # | flag

Nice recipe. This completely takes care of the reload() wart.

Paul Du Bois 12 years, 1 month ago  # | flag

Update for 2.3. Can you show how to solve the subclass problem in 2.3? How would one find all subclasses, if the inheritance linkage is upward only?

Perhaps the solution would involve tracking all instances of MetaAutoReloader. That implies that MetaAutoReloader should have a __metaclass__ of MetaInstanceTracker. :-)

Michael Hudson (author) 11 years, 9 months ago  # | flag

nah, use __subclasses__(). I've updated the recipe (finally) to take advantage of the new-in-2.3 features.

Joakim Pettersson 11 years, 4 months ago  # | flag

patch. I got rid of some inheritance cycle problems by inserting an IF statement inside MetaAutoReloader:

:
for subcls in old_class.__subclasses__():
    if subcls.__name__ != name \
    or subcls.__module__ != ns['__module__']:
         newbases = ()
         :

This fix makes it possible to patch old instances like this:

:
>>> class Bar(Bar):
...     def mix(self, arg):
...         self.meth(arg)
...
>>> b2.mix(4)
4
Michael Hudson (author) 11 years, 3 months ago  # | flag

hmm. Interesting, but it's an extension of what I originally intended.

Mike Meyer 10 years, 8 months ago  # | flag

How about other methods of code capture? Very cool. But objects are only one way you can capture code in Python- albeit the most common one. But you get the same kind of problemsanywhere a pointer to a function or method gets saved: callbacks inGUIs, closures, etc.

Anyone got solutions for those?

Harald Armin Massa 10 years, 8 months ago  # | flag

that really crys for integration. with

http://aspn.activestate.com/ASPN/Mail/Message/python-list/907876

Thomas Hellers autoreload. That really will push up long running processes, won't it?

Harald

Add a comment

Sign in to comment