Re: OpenMCL 1.0 (up to head) memory corruption

I left things running with GC integrity-checking enabled. When I returned,
the integrity-checking code had found an inconsistency, so there's still
a problem.

I've tried to reproduce a failure, and so far haven't been able to do so.
The EGC is on, and (CCL::GC-COUNT) shows that something over 8500 GCs
have occurred.  (CCL::FULL-GCCOUNT) [I don't remember why one function
name is hyphenated and the other isn't] reports 0, so (with your change
to send smaller data structures in effect) the EGC has caught everything.
That's good: nothing that the server allocates lives very long, and
suggests that the EGC is effective in this case.

While I've been writing this, the GC count has gone over 9000. If it
gets over 10000, then it might be fair to say that the problem(s) has/have
less than a 1/10000 chance of occurring.

OK, its over 10300.  I'll check the changes into the bleeding-edge tree.
The fact that it failed earlier today suggests that things aren't 100%
right here, but there's some evidence to suggest that they're at least
99.999% right.

I'll be out of town for part of the next week. If you get a chance in
the next day or so to try the test in the development version, please
let me know and I'll try to clean things up a bit and check them
into the main tree as well.

On Tue, 31 Jan 2006, Gary Byers wrote:

>
>
> On Tue, 31 Jan 2006, Ralf Stoye wrote:
>
>> Hi
>>
>>> How do you (Ralf) want to proceed
>>
>> I started these experiments because our backupsoftware died to often (thereby
>> preventing all client machines to do their "nightwork"). The current
>> workaround is to give each client its own server (on another port) which
>> works nice but isn't a real solution. So "doing the right thing" is on the
>> todo/wishlist for the next weeks.
>>
>> A harder testcase, and a workaround?
>> In the code i posted, I reduced the size of the sended data (local size in
>> client-send-thing) to 1000+Random(1000), so the egc has more things to
>> shuffle around and the crash will happen much earlier (usually in the first 3
>> minutes)
>> On the other side all seems to be stable when i disable egc and set the
>> LISP-HEAP-GC-THRESHOLD to some 100MB. (So after a gc there isn't a reason to
>> immediatly gc again?)
>> Will test this on the real thing... (But there remains an unpleasant
>> feeling).
>> Do you think this arrangements will eliminate the Problem completely?
>>
>
> No.
>
> The problem(s) (there might be more than one of them) seem to have to
> do with:
>
>    a) stopping and restarting other threads, especially when there are
>       a lot of such threads and they start allocating memory as soon
>       as they wake up.  (This seems to prevent some threads from getting
>       enough CPU time to wake up, and seems to cause further problems.
>       I'd almost be tempted to say that there's an OS bug there, but
>       saw similar behavior under both Linux and Darwin.)
>
>    b) at least in the 64-bit lisp, there seems to be a fencepost in
>       code that stack-allocates certain types of objects (lists/conses
>       and vectors where DYNAMIC-EXTENT declarations are involved.)  If
>       the GC interrupts a thread that's in the middle of initializing
>       such an object, the GC doesn't realize that the object isn't
>       fully initalized (e.g., is half-garbage) and this may cause
>       problems.
>
> Both of these problems (and anything similar to (b) that I haven't
> found yet) are GC-related; there's no difference between the full
> GC and the EGC as far as this stuff goes, except for the fact that
> the EGC runs more frequently (often much more frequently.)
>
> To pick a (plausible) number out of a hat, it might be the case that
> one or the other of the above problems will happen every 1000 GCs (or
> has a 1/1000 chance of happening on any particular GC.)  Turning off
> the EGC and/or bumping up GC thresholds might increase the likelyhood
> that you'll get work done before factors combine to trigger one of
> the bugs, but doesn't guarantee it.
>
>>
>>> the server that
>>> crashed and which had entered some non-productive state before crashing.
>>> Has that been your experience, or have you seen crashes in the client
>>> as well ?
>>
>> The real server crashes while some threads are waiting (read from net), some
>> are working (read from disk).
>> I have newer seen a crashed client.
>>
>>
>>> I think that I've fixed this (at the very least, I've been running
>> Please give me that patch ;-)
>
> After I sent that message, I started running into (b).  It wasn't until
> I understood the problem that I realized that it was specific to the
> 64-bit port.
>
>> Maybe you can include it allow us to turn it on/off?
>> At least i could test it to ensure it really eliminates the problem.
>>
>
> What I'd like to do at the moment is:
>
> (1) start the test running
> (2) get some sleep (!)
> (3) it the test runs to completion, check the changes into bleeding-edge
>     CVS.
> (4) if things run reliably for you (at least the test case does), figure
>     out how to get them to you so that you can build your application
>     with the main CVS code base + those patches
> (5) if the problem seems to be fixed, check it into the main CVS tree.
>
>
> The way that I'm currently trying to avoid (a) -might- interact poorly
> with the functions PROCESS-SUSPEND and PROCESS-RESUME.  User code that
> does PROCESS-SUSPEND is already asking to lose and has many ways of
> doing so; the best solution may be to simply deprecate and eventually
> remove PROCESS-SUSPEND/RESUME
>
>
>> Ralf
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> p.s.
>> It just noticed a much simpler way to trigger these things.
>> (see below; I run some of them and the best got up to 9000, so maybe you
>> still have to wait some time)
>> in every case the crash followed a call to %BIGNUM-RANDOM
>> I am sure it is the gc because there are no problems with code wrapped in
>> (without-gcing....).
>>
>> ;;; simpler testcase
>> (proclaim '(optimize (speed 0) (debug 3) (safety 3)))
>>
>> (defun start-n-foos (n)
>>  (dotimes (k n)
>>    (process-run-function `(:name ,(format nil "foo ~d" k))
>>              #'foo k)))
>>
>> (defun foo (thevar)
>>   (dotimes (n 100000)
>>      (when (zerop (mod n 100))
>>    (format t "~%~s has ~d" (process-name *current-process*) n))
>>       (setf thevar (random 1000000000000))
>>       (sleep (random 0.3))
>>       ))
>>
>>
>> ; (start-n-foos 150)
>>
>> ->
>> "foo 63" has 2500what's being allocated here ?
>> ? for help
>> [450] OpenMCL kernel debugger: B
>> current thread: tcr = 0x3183d0,  psn = 95, native thread ID = 0x33803,
>> interrupts enabled
>>
>>
>> (#xf71eef60) #x00016488 : _debug_backtrace + 36
>> (#xf71eefa0) #x0001678C : _lisp_Debugger + 252
>> (#xf71ef000) #x00016858 : _Bug + 76
>> (#xf71ef240) #x00011B54 : _normalize_tcr + 48
>> (#xf71ef290) #x00011CC0 : _gc_like_from_xp + 184
>> (#xf71ef2e0) #x00011D8C : _gc_from_xp + 32
>> (#xf71ef330) #x00011280 : _allocate_object + 108
>> (#xf71ef380) #x00011430 : _handle_alloc_trap + 204
>> (#xf71ef3d0) #x000121F4 : _PMCL_exception_handler + 84
>> (#xf71ef430) #x00012C08 : _signal_handler + 168
>> (#xf71ef6a0) #x00017BC8 : _pseudo_sigreturn + 0
>> (#xF71EFC20) #x00006100 : (subprimitive _ret1valn)
>> (#xF71EFC30) #x04012CEC : #<Function DO-IT #x0800c256> + 536
>> (#xF71EFC40) #x00006100 : (subprimitive _ret1valn)
>> (#xF71EFC50) #x04012970 : #<Function BIGNUM-REM #x0800c236> + 244
>> (#xF71EFC60) #x04012AC4 : #<Function BIGNUM-REM #x0800c236> + 584
>> (#xF71EFC70) #x0403CFE8 : #<Function MOD #x08025c2e> + 52
>> (#xF71EFC80) #x0403F250 : #<Function %BIGNUM-RANDOM #x08026cee> + 388
>>
>>
> _______________________________________________
> Bug-openmcl mailing list
> [email protected]
> http://clozure.com/mailman/listinfo/bug-openmcl
>
>