ActiveState Code

Recipe 576720: lazy property

Python does not have lazy evaluation syntax features built-in, but fortunately decorators can be used with new-style classes to emulate such a feature. There are cases where one wants foo.property to return the actual property whose calculation takes significant amount of time.

This recipe adapts the existing property to provide a lazypropery decorator that does this.

See the first comment below for an example usage.

Also see: lazy initialization

Python 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
def lazyproperty(func):
    """A decorator for lazy evaluation of properties
    """
    cache = {}
    def _get(self):
        try:
            return cache[self]
        except KeyError:
            cache[self] = value = func(self)
            return value
        
    return property(_get)

Comments

  1. 1. At 2:35 p.m. on 15 apr 2009, Sridhar Ratnakumar (the author) said:

    Example,

    class Foo(object):
    @lazyproperty
    def bar(self):
        # do some length operation
        value = calculate()
        return value

f = Foo()
for x in range(10):
    print f.bar
  2. 2. At 3:12 a.m. on 16 apr 2009, Alan Franzoni said:

    The memoize pattern is pretty useful in order to cache heavy operations, but I think this kind of usage could be misleading. When used on a pure or almost-pure function, it could be assumed that the global state has little impact on the result.

    But what if Foo class instances are not immutable, and the bar() method result does not depend just on the input value but on any of the object's attributes? E.g. something like this:

    class Foo(object):
    attr1 = None
    attr2 = None

    @lazyproperty
    def bar(self, a, b, c):
        value = calculate(a, b, c, self.attr1, self.attr2)
        return value

    I think a proper way to do this should take into account the Foo instance state BEYOND the input arguments.

  3. 3. At 6:18 a.m. on 16 apr 2009, Matteo Dell'Amico said:

    Alan: perhaps a @mutating decorator to declare that some methods should reset one of more of the cached properties?

  4. 4. At 8:14 a.m. on 16 apr 2009, Alan Franzoni said:

    Matteo: that might, or might not, be useful. If such changes happen frequently you just happen to reset your cache too often - if attr1 switches back and forth between 1 and 2, you just keep resetting the cache instead of caching two different values.

    And it would be overly complex, IMHO, since a forgotten decorator might just make an app unpredictable.

  5. 5. At 3:37 p.m. on 16 apr 2009, Sridhar Ratnakumar (the author) said:

    Alan,

    The motivation behind using lazy properties (for me) is to defer execution of the initialization code that sets member variables which is typically done in __init__(). The reason for deferring execution is that, one may want to avoid performing expensive calculation (in arriving at the values for member variables) during the object creation phase - and have that done only during the invocation of the member variable.

    Since this is a lazy property and not a lazy method, I would expect Foo.bar() to not accept any arguments; even if it did.. there is no way the dot accesor syntax - f.bar - is able to pass the arguments to bar(). The argument is thus moot.

    I'll shortly update the recipe (by removing the memoize decorator) to avoid this confusion.

  6. 6. At 2:32 a.m. on 22 apr 2009, Alan Franzoni said:

    Sridhar,

    I probably went wrong with my example.

    But my point was a bit different; the problem is that any property might access other instance attributes. Whatever way they are created, those may vary during execution. Hence you should either clearly state this recipe only works on immutable objects, or find out a different caching algorithm.

    Here's a "proper" example of the issue. Try this (there may be too much code here, but this will help with my next proposal):

    class MyMutableObject(object):
    _counter = 0

    def __init__(self, a, b, c):
        self.a = a
        self.b = b
        self.c = c

        self.__class__._counter += 1
        self.counter = self._counter

    @lazyproperty
    def complex_property(self):
        return self.a + self.b

    def __getstate__(self):
        return (self.a, self.b, self.c)

    def __repr__(self):
        return "MMO %s: a=%s, b=%s, c=%s" % (self.counter, self.a, self.b,
                self.c)

mmo1 = MyMutableObject(5, 10, 12)
mmo2 = MyMutableObject(1, 2, 3)

print "First access"
print mmo1.complex_property
print mmo2.complex_property
print "Reaccess"
print mmo1.complex_property
print mmo2.complex_property

print "Change values and reaccess."
mmo1.a = 1
mmo1.b = 2
mmo1.c = 3

print mmo1.complex_property

    Will output:

    First access
15
3
Reaccess
15
3
Change values and reaccess.
15

    The last line's result might come just unexpected, because the object attributes have been changed, but no cache invalidation pattern has been applied. What I mean is: this sort of approach might be both error-prone and sub-effective: if such complex property results depends only on the object state, it won't probably vary between instances sharing an identical state; but this kind of lazyproperty forces a recalc every time a different instance is created.

    This is my proposal:

    def statelazyproperty(func):
    """A decorator for state-based lazy evaluation of properties
    """
    cache = {}
    def _get(self):
        state = self.__getstate__()
        try:
            v = cache[state]
            print "Cache hit %s" % str(state)
            return v
        except KeyError:
            print "Cache miss %s" % str(state)
            cache[state] = value = func(self)
            return value

    return property(_get)

    Just let any instance have a __getstate__() method, and cache values upon that key. This makes the cache change-aware and lets it hit even though different instances get called.

  7. 7. At 5:38 p.m. on 26 apr 2009, Sridhar Ratnakumar (the author) said:

    Alan,

    I did understand your original comment. As I said earlier, "The motivation ... is to defer execution of the initialization code".

    See 'lazy initialization'.

    BTW, in your example using __hash__ is better than __getstate__.

Sign in to comment