Welcome, guest | Sign In | My Account | Store | Cart

help in debugging memory problems (Python recipe) by Will Ware
ActiveState Code (http://code.activestate.com/recipes/67112/)

When developing C extensions and running into memory problems, I find the typical problem is mismanagement of reference counts, particularly abuses of Py_INCREF and Py_DECREF, as well as forgetfulness of the refcount effects of functions like Py_BuildValue, PyArg_ParseTuple, PyTuple/List_SetItem/GetItem, etc. The 1.5.2 source codebase offers some help with this (search for Py_TRACE_REFS) but I found it useful to add this function in Objects/object.c, just before _Py_PrintReferences.

Unlike _Py_PrintReferences, this function will print only the total of all the refcounts in the system, so it can be used safely in loops that will repeat millions of times, where_Py_PrintReferences would print out way too much stuff to be useful. This can help you to identify errantly wandering Py_INCREFs and Py_DECREFs.

Python, 33 lines
Download
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
void _Py_CountReferences(fp)
	FILE *fp;
{
	int n;
	PyObject *op;
	for (n = 0, op = refchain._ob_next;
	     op != &refchain;
	     op = op->_ob_next, n += op->ob_refcnt);
        fprintf(fp, "%d refs\n", n);
}

/* In my C extension, I put in the following macros. */

#if defined(Py_DEBUG) || defined(DEBUG)
extern void _Py_CountReferences(FILE*);
#define CURIOUS(x) { fprintf(stderr, __FILE__ ":%d ", __LINE__); x; }
#else
#define CURIOUS(x)
#endif
#define MARKER()        CURIOUS(fprintf(stderr, "\n"))
#define DESCRIBE(x)     CURIOUS(fprintf(stderr, "  " #x "=%d\n", x))
#define DESCRIBE_HEX(x) CURIOUS(fprintf(stderr, "  " #x "=%08x\n", x))
#define COUNTREFS()     CURIOUS(_Py_CountReferences(stderr))

/*
To debug, I rebuild Python using 'make OPT="-DPy_DEBUG"', which
causes the stuff under Py_TRACE_REFS to be built. My own makefile
uses the same trick, by including these lines:

debug:
        make clean; make OPT="-g -DPy_DEBUG" all
CFLAGS = $(OPT) -fpic -O2 -I/usr/local/include -I/usr/include/python1.5
*/

When I suspect that one of my functions is responsible for memory problems, I liberally sprinkle it with calls to the COUNTREFS() macro. This allows me to keep track of exactly how many references are being created or destroyed as I go through my function. This is particularly useful in tight loops where dumb mistakes can cause reference counts to grow ridiculously fast. Also, reference counts that shrink too fast (overzealous use of Py_DECREF) can cause core dumps because the memory for objects that should still exist has been reallocated for new objects.

Tags: extending

1 comment

Alexander Pletzer 21 years, 9 months ago  # | flag

works for Python 2.2. Good stuff! I tried it out. It helped me understand the reference counting in Python.

--Alex.