Re: Reception of Multicast message stopped after network restart

Hi Bela,
	I double checked this morning, I've downloaded 2.8 GA from your web site and loopback is default to FALSE. It is default to true only starting from 2.8.1 GA. 
I will re-run same test with 2.8 GA with loopback=true and also 2.8.1 and 2.11.Alpha1 with their default configuration.

Thx

David Forget 

>-----Original Message-----
>From: ext Bela Ban [mailto:[email protected]]
>Sent: Tuesday, September 21, 2010 4:58 PM
>To: Forget David (Nokia-MS/Montreal)
>Cc: [email protected]; javagroups-
>[email protected]
>Subject: Re: Reception of Multicast message stopped after network
>restart
>
>Hmm, UDP.loopback *is* actually set to true in udp.xml shipped with 2.8
>(see [1]), but in your config it's false...
>
>
>[1]
>http://javagroups.cvs.sourceforge.net/viewvc/javagroups/JGroups/conf/udp
>.xml?revision=1.39.2.1&view=markup&pathrev=Branch_JGroups_2_8
>
>[email protected] wrote:
>> Hi Bela,
>> 	We did the test again using 2.8 GA with UDP as transport (using
>udp.xml shipped with JGroups 2.8 GA) and it failed 100% of the time.
>>
>>     * OS / version : CentOS release 5.3 (Final)
>>     * JGroups version : 2.8 GA
>>     * Your config XML file used : udp.xml with shipped with JGroups
>only Multicast port pour was changed to use 55240
>>
>> Result:
>>     A and B form a cluster
>>     network restart on A server @ ~ 2010-09-21 11:45:42
>>
>>     FROM B: A was suspected by B and excluded A
>>     FROM A: B was suspected by A and A start an infinite Suspect/Merge
>loop, A will never exclude B
>>     From this point A do not receive B multicast messages from B, A is
>looping on SUSPECT/MERGE.
>>
>>     I did a tcpdump and Multicast from B are received on server A but
>not handle but JGroups (at least not log on TP/UDP).
>>     I also did some threadump on A and Multicast Receiver thread is
>still running.
>>
>>     I attached A and B log with trace log level.
>>
>>     PS: Same behavior was observed without manual network restart with
>same result, we are still working to find reproducible scenarios.
>>
>> Thanks
>>
>>
>> David Forget
>>
>>
>> -----Original Message-----
>> From: ext Bela Ban [mailto:[email protected]]
>> Sent: Tuesday, September 21, 2010 8:00 AM
>> To: Forget David (Nokia-MS/Montreal)
>> Cc: [email protected]; javagroups-
>[email protected]
>> Subject: Re: Reception of Multicast message stopped after network
>restart
>>
>>
>>
>> [email protected] wrote:
>>
>>> How would nodes which stop receiving messages trigger a UDP storm ?
>>>
>>>
>>> DF> This is still under investigation, but quick look at tcpdump show
>>> enndless NAKACK retransmission and
>>> lots of SUSPECT from all other nodes that will never stop until all
>>> nodes on the server is restarted.
>>>
>>>
>>>
>>>> We have been able to reproduce the issue with this very simple
>>>> scenario on Linux 5.3 64bits.
>>>> Scenario: A and B join the same cluster, when everything is stable,
>We
>>>> perform a network restart on server A {/sbin/service network
>restart}.
>>>>
>>
>> I ran this scenario on my Fedora based cluster as well: service
>network
>> start / stop on A (coord of cluster {A,B}). What happened was:
>>
>>     * A was suspected by B and excluded
>>     * A came back
>>     * A and B merged back into 1 cluster
>>
>> This was done with UDP as transport (using udp.xml shipped with
>JGroups)
>>
>>
>>
>>> I don't think this works: an ifconfig down/up is not something that a
>>> socket would survive ! Can you use IP Bonding ?
>>>
>>
>> I was wrong: apparently this works, at least on Linux and with UDP as
>> transport !
>>
>>
>>
>>>> For an unknown raison node A stop receiving Multicast from B
>resulting
>>>> on an unstable cluster were members could not join the same cluster
>>>> anymore. We are reproducing this issue 100% of the time.
>>>>
>>
>>
>> Well, maybe you need to write down the exact steps, and the following
>info:
>>
>>     * OS / version
>>     * JGroups version
>>     * Your config XML file used
>>
>>
>>
>>
>
>--
>Bela Ban
>Lead JGroups / Clustering Team
>JBoss


------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev