Re: polipo memory growing apparently without bound; atoms?

On Mon, Oct 19, 2009 at 03:52:37PM -0700, Josh Triplett wrote:
> On Mon, Oct 19, 2009 at 05:45:19PM +0200, Juliusz Chroboczek wrote:
> > > Discarded all objects, 1 + 3 objects left (5 chunks and 351679 atoms used).
> > 
> > Yep, that's not good.
[...]
> > The contents of the atoms will give us an idea of where these atoms
> > come from.
> 
> We built polipo with debug symbols and started running that on our
> server, so we could investigate.  We produced the following gdb script,
> runnable with "gdb -batch -x thescript -p `pidof polipo`", which dumps
> the contents of the atom cache.  We saved a copy of the atom cache now,
> and we'll check again after polipo grows huge again.

Polipo hasn't yet grown huge enough to make the system start swapping,
but it has seen a few days of crawler traffic, and the atom cache has
grown to almost 30k atoms.  We've attached a copy of the current atom
cache.  (It includes no private information, as it only caches the
results of crawling public webpages.)  Most of the atoms look like full
HTTP headers for a particular response.

Killing polipo with SIGUSR1 didn't make any difference to the atom
cache.  SIGUSR2 did cause the atom cache to shrink by a handful of
entries, but it still held just under 30k atoms.

Please let us know if we can provide any further information.

Thanks,
Josh Triplett and Jamey Sharp

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay 
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

_______________________________________________
Polipo-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/polipo-users