Re: [MLton] Crash in GC

A few (not very clever) questions about the GC state here -

 * The frontier pointer, stored in gc_state rather than in heap - does this represent the end of live allocation in the nursery? In this case it appears to be equal to the nursery pointer. Does that mean all live objects have been (or should have been) copied from nursery to old generation by the copying GC that just finished, and so the nursery is (should be) empty?

 * The gc-messages output says the nursery had size 1,093,424. My offending pointer was beyond the nursery pointer, but not by as much as that - so it was a valid pointer within the nursery before the last copying GC?

 * The error comes from the invariant scan performed at invariant.c:116 following the remark "Checking old generation" (see stack trace below). I think this is going through every object pointer it can find in the old generation heap to make sure the thing it points to is also in some valid space? So this pointer that was in the nursery, and has been overlooked by the copying GC, is pointed to by something that either already was in the old generation or has just been copied to it, hence the problem.

I am struggling slightly with the philosophical question of how the GC can see that the pointer is pointed to by something in the old generation, while also having been unable to spot that the object existed when performing the copying GC.


Chris

#0  assertIsObjptrInFromSpaceOrImmutableMutableOrRootStaticHeap (s=0x555555602b80 <gcState>, opp=0x80000027308, env=0x0)
    at gc/invariant.c:17
#1  0x00005555555c4082 in callIfIsObjptr (s=0x555555602b80 <gcState>, f=0x7fffffffe220, opp=0x80000027308) at gc/foreach.c:12
#2  0x00005555555c443c in foreachObjptrInObject (s=0x555555602b80 <gcState>, p=0x80000027308 "", f=0x7fffffffe220, 
    skipWeaks=false) at gc/foreach.c:95
#3  0x00005555555c48b8 in foreachObjptrInRange (s=0x555555602b80 <gcState>, front=0x80000027248 "", back=0x7fffffffe1f0, 
    f=0x7fffffffe220, skipWeaks=false) at gc/foreach.c:190
#4  0x00005555555ce988 in invariantForGC (s=0x555555602b80 <gcState>) at gc/invariant.c:116
#5  0x00005555555c66ef in performGC (s=0x555555602b80 <gcState>, oldGenBytesRequested=0, nurseryBytesRequested=32808, 
    forceMajor=false, mayResize=true) at gc/garbage-collection.c:194
#6  0x00005555555d2352 in GC_sequenceAllocate (s=0x555555602b80 <gcState>, ensureBytesFree=0, numElements=4098, header=211)
    at gc/sequence-allocate.c:65
#7  0x00005555555772d0 in L_nonZeroLen_135 () at /tmp/fileM2oB33.3.s:18345
#8  0x00007fffffffe470 in ?? ()
#9  0x00005555555d4afe in MLton_init (argc=2048, argv=0x8000000d078, s=0x800003d5648) at platform.c:20

On Thu, 3 Nov 2022, at 16:45, Chris Cannam wrote:
> Thanks! Trying again with some more debug output, the failure is
>
> gc.c: assertIsObjptrInFromSpace opp = 0x0000080000027308  *opp = 
> 0x0000080000346e00
>
> at a time when 
>
> 0x0000080000000000 - start of heap
> 0x00000800001da1a0 - end of old generation
> 0x00000800002e50d0 - start of nursery
> 0x00000800002e50d0 - frontier pointer in gc_state
> 0x00000800003f0000 - end of heap
>
> and so the test that fails must be "p <= s->frontier" in isPointerInNursery().
>
> The last output from gc-messages was
>
> [GC: Finished minor Cheney-copy; copied 66,808 bytes.]
> [GC: Finished gc #34; time 18 ms,]
> [GC:	heap at 0x0000080000000000 of size 4,128,768 bytes (+ 32,768 bytes 
> card/cross map),]
> [GC:	with old-gen of size 1,941,920 bytes (47.0% of heap),]
> [GC:	and nursery of size 1,093,424 bytes (26.5% of heap).]
>
> Does that suggest anything obvious? I think I would have to learn more 
> about GCs to get really into this.
>
> I've uploaded the offending program at 
> https://all-day-breakfast.com/m/program.sml.gz (as it's pretty much a 
> concatenation of publicly-available code; it just isn't something 
> currently managed in a public repo).
>
>
> Chris
>
> On Thu, 3 Nov 2022, at 01:28, Matthew Fluet wrote:
>> Agreed that, while non-default, your GC options aren't particularly 
>> unusual and there is a true GC bug there.
>>
>> You can automate compiling with `-g` and linking to the debug runtime 
>> by compiling with `-debug true`.
>>
>> The https://github.com/MLton/mlton/blob/master/runtime/gc/debug.h has a 
>> number of enums/constants that can be switched from `FALSE` to `TRUE` 
>> to turn on additional assertions and logging.  `DEBUG_GENERATIONAL` 
>> would seem to be a good candidate to enable.  It can also be helpful to 
>> simply run the program with `@MLton gc-messages`, which will print out 
>> information at the start and end of each GC, especially about the 
>> address and sizes of the heaps (including generational boundary).  That 
>> might help to understand if the bad pointer is "near" the FromSpace or 
>> in an old heap or something similar.
>>
>> Feel free to follow up with more information or to share the 
>> bug-triggering program if its still hard to track down.
>>
>> -Matthew
>>
>> On Wed, Nov 2, 2022 at 11:58 AM Chris Cannam 
>> <[email protected]> wrote:
>>> Hello!
>>> 
>>> I have a pure SML program (no FFI) which compiles and runs when compiled with the default MLton options, but crashes with a segfault when compiled with certain GC flags.
>>> 
>>> For example,
>>> 
>>> $ mlton program.sml
>>> $ ./program
>>> 
>>> runs to completion, while
>>> 
>>> $ mlton -runtime 'copy-generational-ratio 10.0' program.sml
>>> $ ./program
>>> 
>>> runs for a second or so and then crashes. The crash is a segfault in (according to gdb) forwardObjptr within performGC.
>>> 
>>> I tried emitting C and compiling that with -g linked against libmlton-dbg.a, and in this mode I get an assertion failure::
>>> 
>>> gc/invariant.c:13: assertIsObjptrInFromSpaceOrImmutableMutableOrRootStaticHeap: Assertion `isObjptrInFromSpace (s, *opp) || isObjptrInImmutableMutableOrRootStaticHeap (s, *opp)' failed.
>>> 
>>> The failure is within the invariant check made at the end of performGC just before leaveGC is called, and is within the "checking old generation" part of invariantForGC.
>>> 
>>> The same behaviour happens with the Arch Linux package of 20210117 and with a clean build from the current MLton repo, and it doesn't appear to matter which codegen I use.
>>> 
>>> Any tips or ideas for how to debug this? Besides that it seems a bug, I am afraid that a failure in GC with unusual options now has the potential to become a failure with default options in the future under different memory pressure or allocation patterns.
>>> 
>>> I can provide a test program off-list if that would help anyone get to grips with it, but although it's just a single SML file, it is quite large - the result of concatenating about 38k lines from other files - and I haven't identified a nice small test case.
>>> 
>>> 
>>> Chris
>>> 
>>> 
>>> _______________________________________________
>>> MLton-devel mailing list
>>> [email protected]; [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/mlton-devel
>>
>> _______________________________________________
>> MLton-devel mailing list
>> [email protected]; [email protected]
>> https://lists.sourceforge.net/lists/listinfo/mlton-devel
>
>
> _______________________________________________
> MLton-devel mailing list
> [email protected]; [email protected]
> https://lists.sourceforge.net/lists/listinfo/mlton-devel