Welcome, guest | Sign In | My Account | Store | Cart

Recursive version sys.getsizeof(). Extendable with custom handlers.

Try it out on your machine

Run the command below in your terminal to instantly set up a sandboxed dev environment with this recipe.
You can view the complete code in the github repository for this recipe.

Python, 55 lines
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
from __future__ import print_function
from sys import getsizeof, stderr
from itertools import chain
from collections import deque
try:
    from reprlib import repr
except ImportError:
    pass

def total_size(o, handlers={}, verbose=False):
    """ Returns the approximate memory footprint an object and all of its contents.

    Automatically finds the contents of the following builtin containers and
    their subclasses:  tuple, list, deque, dict, set and frozenset.
    To search other containers, add handlers to iterate over their contents:

        handlers = {SomeContainerClass: iter,
                    OtherContainerClass: OtherContainerClass.get_elements}

    """
    dict_handler = lambda d: chain.from_iterable(d.items())
    all_handlers = {tuple: iter,
                    list: iter,
                    deque: iter,
                    dict: dict_handler,
                    set: iter,
                    frozenset: iter,
                   }
    all_handlers.update(handlers)     # user handlers take precedence
    seen = set()                      # track which object id's have already been seen
    default_size = getsizeof(0)       # estimate sizeof object without __sizeof__

    def sizeof(o):
        if id(o) in seen:       # do not double count the same object
            return 0
        seen.add(id(o))
        s = getsizeof(o, default_size)

        if verbose:
            print(s, type(o), repr(o), file=stderr)

        for typ, handler in all_handlers.items():
            if isinstance(o, typ):
                s += sum(map(sizeof, handler(o)))
                break
        return s

    return sizeof(o)


##### Example call #####

if __name__ == '__main__':
    d = dict(a=1, b=2, c=3, d=[4,5,6,7], e='a string of chars')
    print(total_size(d, verbose=True))

By itself, the builtin function sys.getsizeof() is not helpful determining the size of a container and all of its contents.

This recipe makes it easy to find the memory footprint of a container and its context. Builtin containers and their subclasses are all fully supported. Extensions are provided for user-defined containers.

5 comments

Jean Brouwers 13 years, 2 months ago  # | flag

The following 2 changes make this recipe work for Python 2+ and 3+. Replace line 4 with:

 try:
     from reprlib import repr
 except ImportError:
     pass

and change line 36 to:

             print(s, type(o), repr(o))  ## , file=stderr)`
Martin Miller 11 years, 3 months ago  # | flag

Seems to me that when the same object is encountered the returned size should be something greater than 0 because at the very least there must be something like a pointer to the duplicated object. Not sure what that value should be, though...

Cristian Andrades 11 years, 3 months ago  # | flag

How can I add a handler to calculate memory usage of a user defined class? For example:

class userClass(object):
    def __init__(self):
        self.integer = 1
        self.array = []

myClass = userClass()

print( total_size(myClass,{**what goes here?**},verbose=True) )
Steven D'Aprano 8 years, 6 months ago  # | flag

Cristian,

I think that this will work:

def myHandler(obj):
    assert isinstance(obj, userClass)
    yield obj.integer
    yield obj.array

total_size(myClass, {userClass: myHandler}, verbose=True)
Sean Worley 6 years, 10 months ago  # | flag

Christian,

I think this will work for any user defined class, even if jt has __slots__, just insert this after line 45 (the else clause has the same indentation level as the for loop, NOT the if statement inside it):

else:
    if not hasattr(o.__class__, '__slots__'):
        if hasattr(o, '__dict__'):
            s+=sizeof(o.__dict__) # no __slots__ *usually* means a __dict__, but some special builtin classes (such as `type(None)`) have neither
        # else, `o` has no attributes at all, so sys.getsizeof() actually returned the correct value
    else:
        s+=sum(sizeof(getattr(o, x)) for x in o.__class__.__slots__ if hasattr(o, x))

You might even want to take it out of the else clause and have it execute unconditionally right before the return. Or, if your passed-in handler functions do attributes, add a parameter to control whether or not it runs (if you choose to do this, I recommend having the parameter default to True).