Re: Reception of Multicast message stopped after network restart

-----Original Message-----
From: ext Bela Ban [mailto:[email protected]] 
Sent: Monday, September 20, 2010 10:11 AM
To: Forget David (Nokia-MS/Montreal)
Cc: [email protected]; [email protected]
Subject: Re: Reception of Multicast message stopped after network restart



[email protected] wrote:
> Hi,
> We are investigating an issue that appear on JGroups 2.8 GA using 
> standard udp.xml configuration look like nodes stop receiving/handling 
> Multicast messages without exception/logs resulting on a UDP storm on 
> large cluster.

How would nodes which stop receiving messages trigger a UDP storm ?


DF> This is still under investigation, but quick look at tcpdump show enndless NAKACK retransmission and
    lots of SUSPECT from all other nodes that will never stop until all nodes on the server is restarted. 


> We have been able to reproduce the issue with this very simple 
> scenario on Linux 5.3 64bits.
> Scenario: A and B join the same cluster, when everything is stable, We 
> perform a network restart on server A {/sbin/service network restart}.

I don't think this works: an ifconfig down/up is not something that a 
socket would survive ! Can you use IP Bonding ?

DF> What I don't understand is that only multicast receiver is not handling messages after the network restart
    the unicast is still working find. 

I'll try this out on my Linux cluster though to see what the behavior is

> For an unknown raison node A stop receiving Multicast from B resulting 
> on an unstable cluster were members could not join the same cluster 
> anymore. We are reproducing this issue 100% of the time.

How ? By simply doing a network restart ?

DF> Yes a simple network restart {on RH/Centos as root run (/sbin/service network restart) on one server }

-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss

------------------------------------------------------------------------------
Start uncovering the many advantages of virtual appliances
and start using them to simplify application deployment and
accelerate your shift to cloud computing.
http://p.sf.net/sfu/novell-sfdev2dev