Re: [jgroups-dev] Issue with members joining clusters

On 12/1/12 6:45 AM, Sundararajan, Mukund wrote:
> Hi Bela,
>
> We are using jgroups 2.12.Final. Our cluster consists of 25 + nodes.
> Each node is a RHEL box.We use TCP with a static host list.


You seem to only list 2 hosts in the static list: 25.221.12.39:9802 and 
25.221.12.38:9802. OTOH, further down the GET_MBRS_REQs are sent to 
25.221.5.229:9802 and 25.221.6.100:9802, which doesn't see right.

With TCPPING, you need to list *all* cluster nodes, or else this won't 
work ! If you cannot list all nodes, the  use a dynamic discovery 
protocol such as MPING, TCPGOSSIP, FILE_PING, S3_PING etc...

You could use PDC [1] in conjunction with TCPPING and then only list 
some bootstrap members, but that's available only in later versions. 
BTW, note that I don't support 2.12.x, see [2] for details.


> We are
> seeing at some of our customer sites that a node will start and
> establishes TCP connections successfully to other nodes but doesn’t join
> the cluster. After a few minutes a Merge View happens and then the node
> joins the cluster. The delay is several minutes in some cases. Do you
> have any idea what might be the issue? Netstat shows established
> connections between the node and its peers.  I see the following
> messages (in blue below) in the jgroups log which suggests the node is
> sending GET_MBRS_REQ to other nodes and is timing out. The network is
> good. No issues there. Our protocol stack contains ENCRYPT. My
> understanding is that ENCRYPT only encrypts application payload. Do GMS
> packets also get encrypted?


It depends on where in the stack ENCRYPT is (what's your config ?), e.g. 
if it's above GMS, then GMS headers will also get encrypted, see 
EncryptNoKeyStore.xml as an example.


> Could there be an ENCRYPT issue?

I don't think so, but you can try and run the system without ENCRYPT, to 
see if it is the culprit.

> But that
> doesn’t explain the merge that happens minutes later. Is this likely to
> happen if some members in the static host list are down?


Yes, this is exactly the downside of TCPPING: if all members listed in 
TCPPING.initial_hosts are down, then a new member will form a singleton 
cluster and not ever merge with the other cluster members.


> The cluster is fine and communicating at the time that this issue happens. It is fairly
> easily reproducible  in our customer networks although I am unable to
> reproduce in a lab environment.



[1] 
http://belaban.blogspot.ch/2012/11/persisting-discovery-responses-with.html
[2] https://community.jboss.org/wiki/Support

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)

------------------------------------------------------------------------------
Keep yourself connected to Go Parallel: 
INSIGHTS What's next for parallel hardware, programming and related areas?
Interviews and blogs by thought leaders keep you ahead of the curve.
http://goparallel.sourceforge.net
_______________________________________________
Javagroups-development mailing list