Re: OpenMCL 1.0 (up to head) memory corruption

On Tue, 31 Jan 2006, Ralf Stoye wrote:

> Hi
>
>> How do you (Ralf) want to proceed
>
> I started these experiments because our backupsoftware died to often (thereby 
> preventing all client machines to do their "nightwork"). The current 
> workaround is to give each client its own server (on another port) which 
> works nice but isn't a real solution. So "doing the right thing" is on the 
> todo/wishlist for the next weeks.
>
> A harder testcase, and a workaround?
> In the code i posted, I reduced the size of the sended data (local size in 
> client-send-thing) to 1000+Random(1000), so the egc has more things to 
> shuffle around and the crash will happen much earlier (usually in the first 3 
> minutes)
> On the other side all seems to be stable when i disable egc and set the 
> LISP-HEAP-GC-THRESHOLD to some 100MB. (So after a gc there isn't a reason to 
> immediatly gc again?)
> Will test this on the real thing... (But there remains an unpleasant 
> feeling).
> Do you think this arrangements will eliminate the Problem completely?
>

No.

The problem(s) (there might be more than one of them) seem to have to
do with:

    a) stopping and restarting other threads, especially when there are
       a lot of such threads and they start allocating memory as soon
       as they wake up.  (This seems to prevent some threads from getting
       enough CPU time to wake up, and seems to cause further problems.
       I'd almost be tempted to say that there's an OS bug there, but
       saw similar behavior under both Linux and Darwin.)

    b) at least in the 64-bit lisp, there seems to be a fencepost in
       code that stack-allocates certain types of objects (lists/conses
       and vectors where DYNAMIC-EXTENT declarations are involved.)  If
       the GC interrupts a thread that's in the middle of initializing
       such an object, the GC doesn't realize that the object isn't
       fully initalized (e.g., is half-garbage) and this may cause
       problems.

Both of these problems (and anything similar to (b) that I haven't
found yet) are GC-related; there's no difference between the full
GC and the EGC as far as this stuff goes, except for the fact that
the EGC runs more frequently (often much more frequently.)

To pick a (plausible) number out of a hat, it might be the case that
one or the other of the above problems will happen every 1000 GCs (or
has a 1/1000 chance of happening on any particular GC.)  Turning off
the EGC and/or bumping up GC thresholds might increase the likelyhood
that you'll get work done before factors combine to trigger one of
the bugs, but doesn't guarantee it.

>
>> the server that
>> crashed and which had entered some non-productive state before crashing.
>> Has that been your experience, or have you seen crashes in the client
>> as well ?
>
> The real server crashes while some threads are waiting (read from net), some 
> are working (read from disk).
> I have newer seen a crashed client.
>
>
>> I think that I've fixed this (at the very least, I've been running
> Please give me that patch ;-)

After I sent that message, I started running into (b).  It wasn't until
I understood the problem that I realized that it was specific to the
64-bit port.

> Maybe you can include it allow us to turn it on/off?
> At least i could test it to ensure it really eliminates the problem.
>

What I'd like to do at the moment is:

(1) start the test running
(2) get some sleep (!)
(3) it the test runs to completion, check the changes into bleeding-edge
     CVS.
(4) if things run reliably for you (at least the test case does), figure
     out how to get them to you so that you can build your application
     with the main CVS code base + those patches
(5) if the problem seems to be fixed, check it into the main CVS tree.


The way that I'm currently trying to avoid (a) -might- interact poorly
with the functions PROCESS-SUSPEND and PROCESS-RESUME.  User code that
does PROCESS-SUSPEND is already asking to lose and has many ways of
doing so; the best solution may be to simply deprecate and eventually
remove PROCESS-SUSPEND/RESUME


> Ralf
>
>
>
>
>
>
>
>
>
>
> p.s.
> It just noticed a much simpler way to trigger these things.
> (see below; I run some of them and the best got up to 9000, so maybe you 
> still have to wait some time)
> in every case the crash followed a call to %BIGNUM-RANDOM
> I am sure it is the gc because there are no problems with code wrapped in 
> (without-gcing....).
>
> ;;; simpler testcase
> (proclaim '(optimize (speed 0) (debug 3) (safety 3)))
>
> (defun start-n-foos (n)
>  (dotimes (k n)
>    (process-run-function `(:name ,(format nil "foo ~d" k))
>              #'foo k)))
>
> (defun foo (thevar)
>   (dotimes (n 100000)
>      (when (zerop (mod n 100))
>    (format t "~%~s has ~d" (process-name *current-process*) n))
>       (setf thevar (random 1000000000000))
>       (sleep (random 0.3))
>       ))
>
>
> ; (start-n-foos 150)
>
> ->
> "foo 63" has 2500what's being allocated here ?
> ? for help
> [450] OpenMCL kernel debugger: B
> current thread: tcr = 0x3183d0,  psn = 95, native thread ID = 0x33803, 
> interrupts enabled
>
>
> (#xf71eef60) #x00016488 : _debug_backtrace + 36
> (#xf71eefa0) #x0001678C : _lisp_Debugger + 252
> (#xf71ef000) #x00016858 : _Bug + 76
> (#xf71ef240) #x00011B54 : _normalize_tcr + 48
> (#xf71ef290) #x00011CC0 : _gc_like_from_xp + 184
> (#xf71ef2e0) #x00011D8C : _gc_from_xp + 32
> (#xf71ef330) #x00011280 : _allocate_object + 108
> (#xf71ef380) #x00011430 : _handle_alloc_trap + 204
> (#xf71ef3d0) #x000121F4 : _PMCL_exception_handler + 84
> (#xf71ef430) #x00012C08 : _signal_handler + 168
> (#xf71ef6a0) #x00017BC8 : _pseudo_sigreturn + 0
> (#xF71EFC20) #x00006100 : (subprimitive _ret1valn)
> (#xF71EFC30) #x04012CEC : #<Function DO-IT #x0800c256> + 536
> (#xF71EFC40) #x00006100 : (subprimitive _ret1valn)
> (#xF71EFC50) #x04012970 : #<Function BIGNUM-REM #x0800c236> + 244
> (#xF71EFC60) #x04012AC4 : #<Function BIGNUM-REM #x0800c236> + 584
> (#xF71EFC70) #x0403CFE8 : #<Function MOD #x08025c2e> + 52
> (#xF71EFC80) #x0403F250 : #<Function %BIGNUM-RANDOM #x08026cee> + 388
>
>