Recursive version sys.getsizeof(). Extendable with custom handlers.
Run the command below in your terminal to instantly set up a sandboxed dev environment with this recipe.
You can view the complete code in the github
repository for this recipe.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | from __future__ import print_function
from sys import getsizeof, stderr
from itertools import chain
from collections import deque
try:
from reprlib import repr
except ImportError:
pass
def total_size(o, handlers={}, verbose=False):
""" Returns the approximate memory footprint an object and all of its contents.
Automatically finds the contents of the following builtin containers and
their subclasses: tuple, list, deque, dict, set and frozenset.
To search other containers, add handlers to iterate over their contents:
handlers = {SomeContainerClass: iter,
OtherContainerClass: OtherContainerClass.get_elements}
"""
dict_handler = lambda d: chain.from_iterable(d.items())
all_handlers = {tuple: iter,
list: iter,
deque: iter,
dict: dict_handler,
set: iter,
frozenset: iter,
}
all_handlers.update(handlers) # user handlers take precedence
seen = set() # track which object id's have already been seen
default_size = getsizeof(0) # estimate sizeof object without __sizeof__
def sizeof(o):
if id(o) in seen: # do not double count the same object
return 0
seen.add(id(o))
s = getsizeof(o, default_size)
if verbose:
print(s, type(o), repr(o), file=stderr)
for typ, handler in all_handlers.items():
if isinstance(o, typ):
s += sum(map(sizeof, handler(o)))
break
return s
return sizeof(o)
##### Example call #####
if __name__ == '__main__':
d = dict(a=1, b=2, c=3, d=[4,5,6,7], e='a string of chars')
print(total_size(d, verbose=True))
|
By itself, the builtin function sys.getsizeof() is not helpful determining the size of a container and all of its contents.
This recipe makes it easy to find the memory footprint of a container and its context. Builtin containers and their subclasses are all fully supported. Extensions are provided for user-defined containers.
The following 2 changes make this recipe work for Python 2+ and 3+. Replace line 4 with:
and change line 36 to:
Seems to me that when the same object is encountered the returned size should be something greater than 0 because at the very least there must be something like a pointer to the duplicated object. Not sure what that value should be, though...
How can I add a handler to calculate memory usage of a user defined class? For example:
Cristian,
I think that this will work:
Christian,
I think this will work for any user defined class, even if jt has __slots__, just insert this after line 45 (the
else
clause has the same indentation level as thefor
loop, NOT theif
statement inside it):You might even want to take it out of the
else
clause and have it execute unconditionally right before thereturn
. Or, if your passed-in handler functions do attributes, add a parameter to control whether or not it runs (if you choose to do this, I recommend having the parameter default toTrue
).