Re: Memory fragmentation?

Justin Dolske wrote:
> Here's the tricky bit: if you free those 4096 1-byte buffers, the VM 
> size of your heap won't change (still 16MB+8KB): it only shrinks from 
> the top, and is blocked by that 8K allocation. This isn't really a 
> concern; you're not touching that 16MB of memory so it'll get paged out 
> and not mapped to real memory.
 >
> If you then free that 8KB chunk, the malloc implementation could 
> potentially tell the OS to shrink the heap's VM region (which would make 
> the process's reported VM size decrease). But afaik not all 
> implementations do that, and not all OSes support it.

Still it would really useful to see if there's a way to directly tell 
the OS the page is no longer used, and can be discarded (not even stored 
in swap).

The memory management pages on MSDN
http://msdn2.microsoft.com/en-us/library/aa366779.aspx
have some interesting info.

Especially here :
http://msdn2.microsoft.com/en-us/library/aa366794.aspx
"Page State :
Free 	The page is neither committed nor reserved. The page is not 
accessible to the process. [...]
A process can use the VirtualFree or VirtualFreeEx function to release 
reserved or committed pages of its address space, returning them to the 
free state.

Reserved 	The page has been reserved for future use. [...] The page is 
not accessible and has no physical storage associated with it.

Committed 	Physical storage is allocated [...]
A process can use VirtualAlloc or VirtualAllocEx to commit physical page 
[...]
The GlobalAlloc and LocalAlloc functions allocate committed pages with 
read/write access.
"

So it seems to be a major gain to use arenas allocated with 
VirtualAlloc, and to VirtualFree them as soon as the arena is empty.

It would probably mean getting fully rid of the standard allocation 
mechanism, but be worth it. VirtualFree'd arena would no more appear as 
memory owned by the process, so the "task manager tells the app is using 
huge amount of memory" crowd would be happy too.