Re: [MLton] tuning up mlton gc

On Fri, Jul 25, 2014 at 9:34 AM, Bernard Berthomieu <[email protected]> wrote:
> On 7/23/14 4:54 PM, Matthew Fluet wrote:
> > > 2. Are there some way other than playing with ratios (which could
> > > have some undesired effects as well) to have mlton always use
> > > generational gc ? ...
>
> > ...  I think that adjusting
> > copy-generational-ratio is the simplest and safest solution.
>
>
> Quite effective, and improving gc times a little further. Thanks for the
> hint !

Great!

> Since we are talking gc, I take the opportunity to mention another
> gripe with it  :-) :
>
> Another "feature" of model checking is that it eats a lot of space.
> In my experience, and whatever the OS and gc options (other than
> may-page-heap) mlton rarely succeeds in using more than approximately
> half the available RAM, even when the applications would benefit from
> more. This obviously makes gc more expensive and, worse, prevents
> some applications to complete by lack of memory (while almost half
> of RAM is unused).
>
> I understand this follows from the heap growing strategy; this
> has been discussed on the list some years ago. But an average
> 50% efficiency in storage management is quite frustrating, and
> certainly not a good point for our tools :-(

I assume that you have tried using a larger value for 'ram-slop'
(which defaults to 0.5).  That's the obvious place where the runtime
system has a 50% RAM limitation, but I suspect that you are noting the
problem described below where the initial copying GC strategy leads to
using less RAM than desired.

> On Thu, 19 Aug 2008, Matthew Fluet wrote on the mlton list:
>
>> The second thought is that I wonder if the heap would be easier to
>> predict/control if we used one contiguous heap at all times (that is, even
>> for major copying collections).  In particular, it would be nice to have
>> the behavior described above for fixed-heap --- namely, that one could use
>> fixed-heap to grab a large block of memory at the beginning of the program
>> and there would be no subsequent resizing, even if the runtime switched
>> over from major copying collections to major mark-compact collections.
>
> That would help I guess. A contiguous heap should ease resizing
> it in place, and a fixed-heap working that way would be great.

The motivation for that comment was that using 'fixed-heap X' often
results in creating two 'X / 2' heaps (due to the initial copying GC
strategy).  Once one of those 'X / 2' sized heaps is filled with live
data, the runtime tries to switch over to mark-compact GC and resize
the 'X / 2' sized heap to an 'X' sized heap.  On a 32-bit system,
though, with 'fixed-heap 3.5G', it is often the case that the virtual
memory system cannot remap a 1.25G heap to a 3.5G heap.  I don't think
this is as much of an issue on a 64-bit system, since the 40-bit
virtual address space is usually significantly larger than RAM (and
therefore significantly larger than any reasonably 'fixed-heap X') and
there is usually no difficulty remapping.

The reason that 'may-page-heap true' is able to help is that it allows
the runtime system to write the 'X / 2' sized heap to disk, thereby
making things easier for the virtual memory system to satisfy a larger
mmap request.

------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds