Re: [Fwd: tests/rr-server core dumps multi-processor]

Hi Jonas

Got the following client failure on the 4 cpu machine after about 2.5 hours

This resulted in the server crashing

The test continues to run on single cpu machines

Regards

Paul

** ERROR **: rr_connection_start failed: disconnect()

aborting...

** ERROR **: rr_connection_start failed: disconnect()

aborting...

** ERROR **: rr_channel_close: disconnect()

r/lib/libthread.so.1
(gdb) bt
#0  0xff079794 in __sigprocmask () from /usr/lib/libthread.so.1
#1  0xff06e9a8 in _resetsig () from /usr/lib/libthread.so.1
#2  0xff06e148 in _sigon () from /usr/lib/libthread.so.1
#3  0xff071188 in _thrp_kill () from /usr/lib/libthread.so.1
#4  0xfeecb760 in raise () from /usr/lib/libc.so.1
#5  0xfeeb5a84 in abort () from /usr/lib/libc.so.1
#6  0xff233de8 in g_logv (log_domain=0x0, log_level=G_LOG_LEVEL_ERROR,
     format=0xfe5016d8 "rr_connection_start failed: disconnect()\n",
     args1=0x3d1bc) at gmessages.c:508
#7  0xff233e90 in g_log (log_domain=0x0, log_level=G_LOG_LEVEL_ERROR,
     format=0x125a8 "rr_connection_start failed: %s\n") at gmessages.c:527
#8  0x0001129c in do_tests (connection=0x0) at rr-client.c:31
#9  0x000113d0 in my_thread (data=0x2b230) at rr-client.c:55
#10 0xff246c6c in g_thread_create_proxy (data=0x0) at gthread.c:551

Jonas Borgström wrote:
> On Thu, 2003-02-20 at 14:18, Paul Caffrey wrote:
> 
>>Jonas
>>
>>Further testing of new version results below.
>>
>>Once again thanks
>>
>>Paul
>>
>>Running on 4 cpu machine, client core dumps after ~ 3 hours
>>
>>server stays up and running
>>
>>(process:10205): RR-CRITICAL **: file rr-message-start.c: line 365 
>>(rr_message_start_done): assertion `RR_IS_MESSAGE_START (start)' failed
> 
> 
> Hi,
> 
> I think I have found the reason for this assertion, It is a similar
> problem to the bug I found yesterday.
> 
> I've committed a patch that should fix this to the CVS. A complete
> source tree is available here:
> 
> http://afs.codefactory.se/~jonas/roadrunner/roadrunner-20030220.tar.gz
> 
> *snip*
> 
> 
>>On itchy, single cpu machine the client crashes after ~ 3 hours
>>No error message, Bus Error (core dumped)
>>(gdb) bt
>>#0  0xff222604 in g_io_create_watch (channel=0x3cce0, condition=G_IO_OUT)
>>     at giochannel.c:455
>>#1  0xff375f44 in add_watch_full (channel=0x3cce0, priority=0,
>>     condition=G_IO_OUT, func=0xff377388 <out_event>, user_data=0x2af98,
>>     notify=0xff376634 <out_removed>) at rr-tcpconnection.c:99
>>#2  0xff376b50 in enable_output (conn=0x2af98) at rr-tcpconnection.c:383
>>#3  0xff37325c in rr_connection_enable_output (connection=0x2af98)
>>     at rr-connection.c:430
>>#4  0xff372a4c in rr_connection_register_sender (connection=0x2af98,
>>     channel=0x2bb70) at rr-connection.c:201
>>#5  0xff369a6c in send_helper (channel=0x2bb70, object=0x432a4,
>>     error=0xfe9fbe54) at rr-channel.c:262
>>#6  0xff369b78 in rr_channel_send_message (channel=0x2bb70, 
>>message=0x432a4,
>>     error=0xfe9fbe54) at rr-channel.c:285
>>#7  0xff370040 in rr_manager_close_channel (manager=0x2bb70, 
>>channel=0x432a4,
>>     code=200, xml_lang=0x12628 "sv", diagnostic=0x12630 "Foo",
>>     error=0xfe9fbe54) at rr-manager.c:893
>>#8  0xff36b010 in rr_channel_close (channel=0x425d4, code=200,
>>     xml_lang=0x12628 "sv", diagnostic=0x12630 "Foo", error=0xfe9fbe54)
>>     at rr-channel.c:756
>>#9  0x00011368 in do_tests (connection=0x425d4) at rr-client.c:44
>>#10 0x000113d0 in my_thread (data=0x2af98) at rr-client.c:55
>>#11 0xff246c6c in g_thread_create_proxy (data=0x0) at gthread.c:551
>>
> 
> 
> I've never managed to reproduce this one. Can you recompile
> glib2 with debug symbols so we can get a better traceback 
> (CFLAGS="-g -O0"). It might be some thread related problem
> with the glib event loop.
> 
> After the change I mentioned above the only bug I can reproduce
> is that after an hour or so a read() operation on the socket
> might fail with:
> 
> errno = ENVAL and strerror returns:
> 
> "Connection reset by peer"
> 
> I have no idea why, because read() and write() are the only
> operations that are done on that socket and the socket
> is on localhost. And only one thread is used to read/write
> from/to the socket.
> 
> This only happens when running on a 4 cpu solaris box. I haven't
> managed to reproduce any of this on a 2 cpu linux 2.2 box or
> on my linux laptop.
> 
> 
>>PS
>>
>>On third machine, single cpu machine also, still running no problems.
>>
> 
> Well, it might sooner or later hit the bug I fixed above :)
> 
> / Jonas