Welcome, guest | Sign In | My Account | Store | Cart

This function is built on assumption, that no custom types was implemented. Function traversing the tree of given object and summing sizes of pointers and basic-types objects got with sys.getsizeof() function. For example, for this two dicts {'key': 'shortstring'} and {'key': 'veryverylongandconsumingstring'} my function will return different sizes, while sys.getsizeof() will return same. TODO - make that calculation of size covers pointers of circular references which for now is just passed by

Python, 69 lines
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
from sys import getsizeof
import types
from inspect import getmembers
from collections import (deque, defaultdict, Counter,
                                    OrderedDict, Iterable)


types_basic = {
e[1] for e in filter(lambda x: isinstance(x[1], type),
                     getmembers(types))
}
types_basic.discard(types.InstanceType)
types_basic.discard(types.ObjectType)

#for method-wrappers
types_basic.add(type(types_basic.__str__))

types_kv = {dict, OrderedDict, defaultdict, types.DictProxyType, Counter}
types_listlike = {list, set, frozenset, tuple, deque}

types_basic -= types_kv
types_basic -= types_listlike

types_kv = tuple(types_kv)
types_listlike = tuple(types_listlike)
types_basic = tuple(types_basic)

def get_size_of(obj):
    """Non-recursive function, that takes Python object,
    walk through its sub-objects and return a sum of size of each pointers
    and basic types.

    """
    size = 0
    seen = set()
    stack = deque()

    stack.append(obj)
    seen.add(id(obj))

    while stack:
        cur = stack.pop()

        size += getsizeof(cur)
        if isinstance(cur, types_basic):
            seen.add(cur)
        elif isinstance(cur, types_listlike):
            for e in list(cur):
                if id(e) not in seen:
                    stack.append(e)
                    seen.add(id(e))
        elif isinstance(cur, types_kv):
            for k, v in cur.items():
                if id(k) not in seen:
                    stack.append(k)
                    seen.add(id(k))
                if id(v) not in seen:
                    stack.append(v)
                    seen.add(id(v))
        else:
            for attr in dir(cur):
                try:
                    o = getattr(cur, attr)
                except AttributeError:
                    continue
                if id(o) not in seen:
                    stack.append(o)
                    seen.add(id(o))
    return size

3 comments

James Mills 8 years, 10 months ago  # | flag

This only works on Python 2.7

Did you test on any other version of Python?

You should at least test on 2.6, 2.7, 3.2 and 3.3

cheers

James

James Mills 8 years, 10 months ago  # | flag

I do however like this Recipe.

Probably one of my favorite I've seen in a long time!

What you've written here might actually help us optimize memory usage in our circuits (1) framework. Do you mind if we borrow, modify and improve upon your code?

cheers

James

  1. http://pypi.python.org/pypi/circuits
Andrew (author) 8 years, 9 months ago  # | flag

It's more "proof of concept" right now, I need to solve how to correctly measure "pointers" in Python, and then I suppose to expand code on other version of Python, than 2.7.

Of course you can take it, actually it would be interesting for me to look on your version if you share it (and take something back :D).

Regards

Created by Andrew on Fri, 1 Feb 2013 (MIT)
Python recipes (4591)
Andrew's recipes (1)

Required Modules

  • (none specified)

Other Information and Tasks