Re: Help on JVM hang

Joseph Shraibman wrote:
> Hui Huang wrote:
> 
>> Veda N Ponnusamy wrote:
>>
>>> Hi,
>>>    We set the LD_ASSUME_KERNEL=2.2.5 and the hang goes away. But the 
>>> application
>>
>>
>>  is running at 60% speed !!
>>
>> Interesting. You could try LD_ASSUME_KERNEL=2.4.1 (the floating
>> stack version of LinuxThreads), it's generally faster than 2.2.5.
>>
>>  Do you know any patches anywhere for this problem ???
>>
>> Unfortunately, no, Redhat did not release the patch. RH9 is the
>> only platform we've seen this problem. We didn't stress test
>> Fedora Core, but AS-3 works Ok.
>>
> 
> So you do have code that can reproduce this problem?  It would help a 
> lot if I could get it.

No special test case. We run a tomcat server and have clients to
talk to it repetitively. I tried to create a small testcase when I
reported the bug, but it wasn't easy.

> 
>> thanks,
>> -hui
>>
>> P.S. Next time when you see the hang, please try the gdb trick
>> to id the bug. If the process still hangs after attach-detach
>> with gdb, it's something else.
> 
> 
> Would strace -p <pid> work as well?
> 
> From: http://bugs.mysql.com/bug.php?id=868
> 
> One interesting side note is that if you strace the right process, the 
> system
> will recover.  However, we have still had database corruption when we 
> did this.
> 

It looks like the same issue. IIRC, the problem happened when one
thread sleeps on pthread_cond_wait, and another thread in
pthread_cond_{signal,broadcast} to wake up the first thread,
sometimes kernel failed to change the first thread back to running.

Both gdb and strace use the ptrace interface, when PTRACE_DETACH
happens, kernel will wake up the process/thread no matter what its
previous state is. When gdb is detached from a process, it will
PTRACE_DETACH every thread (including the one waiting in cond_wait),
that's why the gdb trick can break the hang. Another way to break
the hang is to send a signal to the "right" process/thread.

regards,
-hui


----------------------------------------------------------------------
To UNSUBSCRIBE, email to [email protected]
with a subject of "unsubscribe". Trouble? Contact [email protected]