Re: Memory fragmentation?

Daniel Brooks wrote:

> Applications request memory from the operating system in 4 kilobyte
> pages. Obviously we can bundle up a lot of small objects into a single
> page. When one of those objects is freed, the space it occupied does
> become available for the app to reuse, if it can find something that
> will fit. Because there's no way to move objects around in memory once
> they're allocated, the memory can become fragmented, which means that
> all of the unallocated memory exists only as small chunks strewn
> through out the allocated pages. When the memory is fragmented like
> this, there are a lot of places you can allocate small objects, but to
> create large objects you have to request more pages from the OS. In
> addition, we can't release a page back to the OS if there is anything
> still in it. As it turns out, this is a large part of why Firefox's
> memory usage doesn't go down much when you close a tab - we may have
> freed a lot of things, but because of fragmentation there is still at
> least some part of each page being used.

One thing that wasn't made super clear, though Daniel did touch on it, 
was that when we all of the memory on a page has been freed, that page 
can be reclaimed by the OS. That's why it's important to try and 
allocate things we suspect we can free together on the same page -- so 
we don't end up with one long-lived object keeping us from returning a 
page to the OS.

FWIW, Subversion (http://subversion.tigris.org/) makes extensive use of 
memory  pools, at least in their API calls. Perhaps it's worth trying to 
get in contact with them to see if they have any advice, best practices, 
etc with respect to memory pools?

-Colin