Re: Reception of Multicast message stopped after network restart

Hi Bela,
	We did the test again using 2.8 GA with UDP as transport (using udp.xml shipped with JGroups 2.8 GA) and it failed 100% of the time. 

    * OS / version : CentOS release 5.3 (Final)
    * JGroups version : 2.8 GA 
    * Your config XML file used : udp.xml with shipped with JGroups only Multicast port pour was changed to use 55240
	
Result:
    A and B form a cluster
    network restart on A server @ ~ 2010-09-21 11:45:42
    
    FROM B: A was suspected by B and excluded A
    FROM A: B was suspected by A and A start an infinite Suspect/Merge loop, A will never exclude B	
    From this point A do not receive B multicast messages from B, A is looping on SUSPECT/MERGE. 

    I did a tcpdump and Multicast from B are received on server A but not handle but JGroups (at least not log on TP/UDP). 
    I also did some threadump on A and Multicast Receiver thread is still running.
    
    I attached A and B log with trace log level. 

    PS: Same behavior was observed without manual network restart with same result, we are still working to find reproducible scenarios. 

Thanks 
    	

David Forget 


-----Original Message-----
From: ext Bela Ban [mailto:[email protected]] 
Sent: Tuesday, September 21, 2010 8:00 AM
To: Forget David (Nokia-MS/Montreal)
Cc: [email protected]; [email protected]
Subject: Re: Reception of Multicast message stopped after network restart



[email protected] wrote:
> How would nodes which stop receiving messages trigger a UDP storm ?
>
>
> DF> This is still under investigation, but quick look at tcpdump show 
> enndless NAKACK retransmission and
> lots of SUSPECT from all other nodes that will never stop until all 
> nodes on the server is restarted.
>
>
>> We have been able to reproduce the issue with this very simple
>> scenario on Linux 5.3 64bits.
>> Scenario: A and B join the same cluster, when everything is stable, We
>> perform a network restart on server A {/sbin/service network restart}.

I ran this scenario on my Fedora based cluster as well: service network 
start / stop on A (coord of cluster {A,B}). What happened was:

    * A was suspected by B and excluded
    * A came back
    * A and B merged back into 1 cluster

This was done with UDP as transport (using udp.xml shipped with JGroups)


> I don't think this works: an ifconfig down/up is not something that a
> socket would survive ! Can you use IP Bonding ?

I was wrong: apparently this works, at least on Linux and with UDP as 
transport !


>> For an unknown raison node A stop receiving Multicast from B resulting
>> on an unstable cluster were members could not join the same cluster
>> anymore. We are reproducing this issue 100% of the time.


Well, maybe you need to write down the exact steps, and the following info:

    * OS / version
    * JGroups version
    * Your config XML file used



-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss

------------------------------------------------------------------------------
Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1,  ECMAScript5, and DOM L2 & L3.
Spend less time writing and  rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users