Re: ideas for returning memory to the OS

* [email protected] <[email protected]> wrote:

Hi,

> So we're doing a bunch of work to avoid allocations, fuse allocations,
> and move lifetime-correlated allocations into arenas/subheaps, all to
> avoid fragmentation. That's great. But what about returning memory to
> the OS? For that we need to unmap memory. 

Uh, tricky issue.

IMHO, memory allocation should be (exclusively) done by libc, because
it's optimized to the OS/kernel and provides an (quite;-o) portable
interface. If an certain libc isn't yet good enough, improve *this*
instead of trying to work around that in individual applications.

I don't think, unmapping is that great deal, it just saves some
pagetable entries. BUT: it's important to keep fragmentation within
pages low. The key point is trying to put as much traffic as you can 
to an as small as possible set of pages (so reuse where you can;-))
and the kernel will handle the rest. 

Unused pages lurking around and not being accessed for longer time are
no problem, since they will just silently sleep in swap.

> In fact all three of our main platforms have a way to tell the OS 
> that the contents of a committed page are no longer needed, need 
> not be paged out, so the RAM can be be used by another process, 
> but without messing around with munmap and VMA fragmentation!

Right, that avoids unnecessary swap-out (and later swap-in) for 
unused pages. An sane malloc() should be expected to already
utilize this.

> On Linux: madvise(MADV_DONTNEED) (MADV_FREE is a slightly better
> semantic fit but isn't supported yet)

AFAIK, BSD's MADV_FREE is Linux's MADV_DONTNEED. 
Be aware: MADV_DONTNEED on BSD will make the address space invalid
(so further access will end up in page faults !)

> The main question is, are these already used by platform heap
> implementations? 

AFAIK, BSD's and GNU libc already use it.

> If they are, there's really nothing for us to do.
> Otherwise, we can use them, fairly easily in the case of our arena
> implementation(s).

No, I don't think, mozilla's code should do anything in that area.
Please fix it at the source -> libc.


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------