Re: [pysqlite] unusual memory behavior

Michael Bayer wrote:
> We have a series of tests that
> attempt to detect memory leaks by running a block of code repeatedly,  
> performing a gc.collect() and then comparing the count of objects in  
> gc.get_objects() over time.

For APSW I detect memory leaks by using valgrind.  A big underlying
issue is that many different containers (eg lists, tuples, dicts and
even ints) behind the scenes don't actually free things.  Instead they
keep them ready for reuse in a free list, although freeing will
eventually happen if the free lists get too long.  Event things like
stack frame objects have a free list so if exceptions get thrown it can
take a while for things to get recycled.

My solution is to compile Python using normal malloc so valgrind can see
what is happening, and to disable all the free lists so things get freed
when Python or C code frees them.  Fortunately they unified the #defines
in Python 2.6+ so it is pretty easy to do.  You can see how to do it in
the comments at the top of:

http://code.google.com/p/apsw/source/browse/apsw/trunk/tools/valgrind.sh

Roger