Re: OpenMCL 1.0 (up to head) memory corruption

I think that I've fixed this (at the very least, I've been running
the test case for 40 minutes or so and haven't yet seen any misbehavior.

The bug seems to stem from the fact that suspending a thread isn't
atomic (and needs to be); when it's finished running, the GC
decrements the "suspend count" of each other threads and, if it
becomes 0, tells the thread to wake up.  It doesn't wait around for
any sort of acknowledgement that the thread actually got the message,
so (for a brief period) there will be some threads whose suspend count
is 0 but which have not yet gotten any CPU time to respond to the
wakeup request.  Meanwhile, other threads wake up and start consing
(in your test case, the memory allocation that the worker threads do
isn't exactly "incidental": they allocate reasonably large buffers
repeatedly), and shortly after the thread that invoked the first GC
finishes issuing wakeups to all other threads some other recently-
awakened thread needs to GC.  If some thread is still in the process
of waking up after the last GC, it'll appear to the new GC as an
active, non-suspended thread and will get suspended (even though
it's already still "mostly" suspended); context information will
be lost or misinterpreted, and the GC will either get confused and
crash or foul things up and cause some misbehavior later on ...

I think that I've fixed this (basically by waiting for a newly
awoken thread to get scheduled and acknowledge that it's gotten
the wakeup message before considering it to be "awoken"),  but
I'm not entirely happy with that approach (it effectively makes
GC take longer.)   There may be some other/better way of ensuring
that the wakeup happens atomically; it clearly needs to do so,
but I'd rather that the overhead happened in the "pathological
case" (GC occurring when some threads are still waking up after
the last GC) than in the typical case (every GC) if I can find
a way to do so.


How do you (Ralf) want to proceed ? (I'm a little reluctant to
check the quick fix into the main CVS tree because I'm concerned
that it might negatively affect performance in typical cases;
I suppose that I should try to quantify that a bit ...)


On Mon, 30 Jan 2006, Gary Byers wrote:

> I was able to reproduce this once (and am on a second iteration).
>
> I still don't know exactly what's going on, but have noticed that
> even with *really-write-files* set to NIL, the clients get way
> ahead of the server; :PROC in the client indicates that all of
> the client threads are waiting for the server to read prior
> output (in other words, the server threads aren't getting much
> of any work done.)
>
> David Rager ran into something last week where a a large number
> of threads - each doing incidental memory allocation - triggered
> nearly constant GCs.  (This has to do with the way that per-thread
> memory-allocation works and with the fact that the mechanism used
> by the GC to stop threads causes blocking system calls to fail;
> retrying the system call did incidental memory allocation, which
> triggered a GC, which interrupted blocking system calls ...)
>
> As bad as that was/is, what sometimes happened is that sometimes
> newly-awakened (after GC) threads would trigger a GC before all
> of their peers had finished waking up after the previous GC.  That's
> bad (for a number of reasons), and I think that the one crash that
> I saw in your test case -may- have something to do with that.
>
> In David's case, we worked around the issue by ensuring that one
> particular primitive (TIMED-WAIT-ON-SEMAPHORE) avoided the incidental
> consing that led to the hysteria.  Another short-term workaround was
> to disable the EGC or set its thresholds high enough that it's not
> triggered as often.
>
> In the one test case that I've had fail so far, it was the server that
> crashed and which had entered some non-productive state before crashing.
> Has that been your experience, or have you seen crashes in the client
> as well ?
>
>
> On Mon, 30 Jan 2006, rs wrote:
>
>> Hi,
>> I experience crashes when mcl runs a long time or with a lot of multithreaded
>> tasks.
>> To test this, i wrote two small apps, an remote eval server and a server
>> receiving files whose code is attached to this mail.
>>
>>
>>
>>
> _______________________________________________
> Bug-openmcl mailing list
> [email protected]
> http://clozure.com/mailman/listinfo/bug-openmcl
>
>