Re: Memory fragmentation?

Justin Dolske wrote:
> But here we're talking about 
> a page completely unused by the app, which means it would never get 
> paged in again.  So other than occupying some disk space, it should
 > be harmless

I don't think so.

The first gain is that it will not be paged out, so not written.
The second gain is that we'll tell immediately the OS it's not used, it 
will not have to wait for a while and test last access time to guess it.
The third gain is that it *will* be used again the next time we need to 
allocate memory, and without that optimization the OS will page it in 
only for it to be overwritten by the new data.

> My assumption here was that the VM system can more efficiently deal with 
> large contiguous regions of a process's address space than when it's 
> scattered with unmapped holes. To me, "I'm using 0x10000000 to 
> 0x20000000" implies simple (fast) accounting, while adding hundreds or 
> thousands of gaps to that would increase overhead.
> 
> Maybe there are clever techniques used to mitigate that... Which is why 
> I just said I'm wary about doing this. Performance problems like this 
> have a tendency to be fickle and require testing for conclusive answers.

As I reported, it's possible to play with three different options 
Free/Reserved/Committed + MEM_RESET flag set.
I think what you describe is most likely to happen if your make use only 
of the Free state, and manage a lot of independent VirtualAlloc memory 
blocks.
I think the "Reserved" or "Committed + MEM_RESET flag set" states are 
much lighter and are likely to add almost nothing to the fact that even 
if I say "I'm using 0x10000000 to 0x20000000" the OS needs to manage for 
each page in that range of it's swapped in or out.

I fully agree it require quite extensive testing to reach definitive 
conclusions about what should be used, and to see if the investment is 
worth the cost.

I think also what should really be tested is :

- reproduce a case where firefox (in my case seamonkey) uses hundreds of 
megbyte of memory and brings the OS down to it's knees. BTW in my case 
it seems easily reproduced if I leave Google Reader running for the 
night, which might be a leak problem in Google Reader.

- instrument exactly what happens.

Is all of that memory actively used? In which case it's not a leak 
problem, and leak will not solve it.
One known problem causing that might be counted memory reference. Your 
soft will tend to go all over the memory to check/update the memory 
reference counters. I think GC can have similar behaviour and *kill* 
performances by causing *all* memory to be swapped in in order to check 
if it can be released.

If it's not actively used, then it should be swapped out once and not 
reused, so what happens exactly? My feeling, that strongly influence 
what I wrote above is that Windows is not very efficient at swapping 
"out" memory. But maybe instead it is that large amount of memory appear 
to the OS to be actively used, because a small amount of actively used 
memory is heavily fragmented throughout that memory space? Or something 
else, what then ?

So test cases, instrumentation are needed to go further than the 
discussion we're currently having.