Re: [Fwd: tests/rr-server core dumps multi-processor]

On Thu, 2003-02-20 at 14:18, Paul Caffrey wrote:
> Jonas
> 
> Further testing of new version results below.
> 
> Once again thanks
> 
> Paul
> 
> Running on 4 cpu machine, client core dumps after ~ 3 hours
> 
> server stays up and running
> 
> (process:10205): RR-CRITICAL **: file rr-message-start.c: line 365 
> (rr_message_start_done): assertion `RR_IS_MESSAGE_START (start)' failed

Hi,

I think I have found the reason for this assertion, It is a similar
problem to the bug I found yesterday.

I've committed a patch that should fix this to the CVS. A complete
source tree is available here:

http://afs.codefactory.se/~jonas/roadrunner/roadrunner-20030220.tar.gz

*snip*

> On itchy, single cpu machine the client crashes after ~ 3 hours
> No error message, Bus Error (core dumped)
> (gdb) bt
> #0  0xff222604 in g_io_create_watch (channel=0x3cce0, condition=G_IO_OUT)
>      at giochannel.c:455
> #1  0xff375f44 in add_watch_full (channel=0x3cce0, priority=0,
>      condition=G_IO_OUT, func=0xff377388 <out_event>, user_data=0x2af98,
>      notify=0xff376634 <out_removed>) at rr-tcpconnection.c:99
> #2  0xff376b50 in enable_output (conn=0x2af98) at rr-tcpconnection.c:383
> #3  0xff37325c in rr_connection_enable_output (connection=0x2af98)
>      at rr-connection.c:430
> #4  0xff372a4c in rr_connection_register_sender (connection=0x2af98,
>      channel=0x2bb70) at rr-connection.c:201
> #5  0xff369a6c in send_helper (channel=0x2bb70, object=0x432a4,
>      error=0xfe9fbe54) at rr-channel.c:262
> #6  0xff369b78 in rr_channel_send_message (channel=0x2bb70, 
> message=0x432a4,
>      error=0xfe9fbe54) at rr-channel.c:285
> #7  0xff370040 in rr_manager_close_channel (manager=0x2bb70, 
> channel=0x432a4,
>      code=200, xml_lang=0x12628 "sv", diagnostic=0x12630 "Foo",
>      error=0xfe9fbe54) at rr-manager.c:893
> #8  0xff36b010 in rr_channel_close (channel=0x425d4, code=200,
>      xml_lang=0x12628 "sv", diagnostic=0x12630 "Foo", error=0xfe9fbe54)
>      at rr-channel.c:756
> #9  0x00011368 in do_tests (connection=0x425d4) at rr-client.c:44
> #10 0x000113d0 in my_thread (data=0x2af98) at rr-client.c:55
> #11 0xff246c6c in g_thread_create_proxy (data=0x0) at gthread.c:551
> 

I've never managed to reproduce this one. Can you recompile
glib2 with debug symbols so we can get a better traceback 
(CFLAGS="-g -O0"). It might be some thread related problem
with the glib event loop.

After the change I mentioned above the only bug I can reproduce
is that after an hour or so a read() operation on the socket
might fail with:

errno = ENVAL and strerror returns:

"Connection reset by peer"

I have no idea why, because read() and write() are the only
operations that are done on that socket and the socket
is on localhost. And only one thread is used to read/write
from/to the socket.

This only happens when running on a 4 cpu solaris box. I haven't
managed to reproduce any of this on a 2 cpu linux 2.2 box or
on my linux laptop.

> PS
> 
> On third machine, single cpu machine also, still running no problems.
> 
Well, it might sooner or later hit the bug I fixed above :)

/ Jonas
-- 
Jonas Borgström                  [email protected]
CodeFactory AB                   http://www.codefactory.se/
Office: +46 (0)90 71 86 14