This function is built on assumption, that no custom types was implemented. Function traversing the tree of given object and summing sizes of pointers and basic-types objects got with sys.getsizeof() function. For example, for this two dicts {'key': 'shortstring'} and {'key': 'veryverylongandconsumingstring'} my function will return different sizes, while sys.getsizeof() will return same. TODO - make that calculation of size covers pointers of circular references which for now is just passed by
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 | from sys import getsizeof
import types
from inspect import getmembers
from collections import (deque, defaultdict, Counter,
OrderedDict, Iterable)
types_basic = {
e[1] for e in filter(lambda x: isinstance(x[1], type),
getmembers(types))
}
types_basic.discard(types.InstanceType)
types_basic.discard(types.ObjectType)
#for method-wrappers
types_basic.add(type(types_basic.__str__))
types_kv = {dict, OrderedDict, defaultdict, types.DictProxyType, Counter}
types_listlike = {list, set, frozenset, tuple, deque}
types_basic -= types_kv
types_basic -= types_listlike
types_kv = tuple(types_kv)
types_listlike = tuple(types_listlike)
types_basic = tuple(types_basic)
def get_size_of(obj):
"""Non-recursive function, that takes Python object,
walk through its sub-objects and return a sum of size of each pointers
and basic types.
"""
size = 0
seen = set()
stack = deque()
stack.append(obj)
seen.add(id(obj))
while stack:
cur = stack.pop()
size += getsizeof(cur)
if isinstance(cur, types_basic):
seen.add(cur)
elif isinstance(cur, types_listlike):
for e in list(cur):
if id(e) not in seen:
stack.append(e)
seen.add(id(e))
elif isinstance(cur, types_kv):
for k, v in cur.items():
if id(k) not in seen:
stack.append(k)
seen.add(id(k))
if id(v) not in seen:
stack.append(v)
seen.add(id(v))
else:
for attr in dir(cur):
try:
o = getattr(cur, attr)
except AttributeError:
continue
if id(o) not in seen:
stack.append(o)
seen.add(id(o))
return size
|
This only works on Python 2.7
Did you test on any other version of Python?
You should at least test on 2.6, 2.7, 3.2 and 3.3
cheers
James
I do however like this Recipe.
Probably one of my favorite I've seen in a long time!
What you've written here might actually help us optimize memory usage in our circuits (1) framework. Do you mind if we borrow, modify and improve upon your code?
cheers
James
It's more "proof of concept" right now, I need to solve how to correctly measure "pointers" in Python, and then I suppose to expand code on other version of Python, than 2.7.
Of course you can take it, actually it would be interesting for me to look on your version if you share it (and take something back :D).
Regards