Re: [jgroups-dev] TCPPING merge issue

The issue here is that TCPPING is meant to be used for clusters with 
static membership, so it should always contain a list of *all* nodes.

If you can't do this, and have to use TCP, then use either MPING (when 
IP multicasting is available for discovery only), TCPGOSSIP (for dynamic 
discovery, requires an external process), or FILE_PING (requires a 
shared file system).

Having said that, I found a small glitch in TCPPING: dynamic_hosts is 
updated only on view changes. I added code to update the list when a 
GET_MBRS_REQ is received, too.

I've actually gone through your scenario and this works fine now.

My config was TCP:TCPPING(initial_hosts=A,B), with DISCARD(use_gui=true) 
on top of TCP.

- I started A and B, then made them discard all of their messages
- Then I started C
- Now A and B toggled back from discard mode and A,B and C would merge 
back into 1 cluster.

This is in CVS head (2.10). I'll release 2.10 final some time this week.


Anthony Tong wrote:
> Hi,
>
> We are having issues with forming jgroups clusters. A joiner may get
> 'stuck' indefinitely. We are using a TCPPING based jgroups stack,
> which is fairly similar to tcp.xml with jgroups 2.9.0GA (mainly longer
> timeouts).
>
> As best as I can analyze it, here is my explaination of the scenario
> that is affecting us.
>
> {A,B} are happily running; their protocol stacks have
> TCPPING(initial_hosts=A,B)
>
> C wants to join the cluster, it has TCPPING(initial_hosts=A,B,C).
>
> On C creating its JChannels(), at this specific point, jgroups is
> susceptable to timing issues. Normally, C discovers A or B, and the
> procedes with a JOIN_REQ. However, if TCPPING's first FIND_INITIAL_MBRS
> operation by C times out with no members discovered, C will become
> a singleton group {C}.
>
> "no initial members discovered: creating group as first member"
>
> Say C's timing issues then get resolved, communication between the
> hosts are at a reasonable state. Typically, it then leads to:
>
> - {C} resending GET_MBRS_REQ to A and B
>
> - A and B happily reply GET_MBRS_RSP
>
> - based on the response, C rightfully decides it needs to merge
> "found different views : .... sending up MERGE event with merge
> participants.."
>
> - here's where C run into issues. C is the coordinator for {C}. It
> defers to the {A,B} coordinator to initiate the merge.
> "determining merge leader" .. "I am not the merge leader, waiting
> for merge leader .. to initiate merge"
>
> - however coordinator in {A,B} never initiates the merge, because
> C is not in their initial member list, nor does C get added to the
> (dynamic) member list despite their MBRS dialog
>
> - so, C will then start over, sending GET_MBRS_REQ to same outcome,
> never joining {A,B}
>
>
> I dont have enough jgroups experience yet to know what the fix should
> be. Some thoughts:
>
> It makes sense for C to become singleton group, but should it
> periodically send JOIN_REQ if it remains singleton? (but what happens
> if we get D in the mix?)
>
> Or should the coordinator in {A,B} include C in its member list,
> via knowledge from their GET_MBRS exchange. Is the the logic (or
> actual implementation) as commented in Discovery.up() correct? Maybe
> a logic tweak if GET_MBRS_REQ is received and it is from an unknown 
> member?
> Quoted here:
>
> > * If the header is PingHeader.GET_MBRS_REQ then the PING protocol
> > * will PassDown a PingRequest message
> > * If the header is PingHeader.GET_MBRS_RSP we will add the
> > * message to the initial members
> > * vector and wake up any waiting threads.
>
> I have full logs with 3 nodes upon request.

-- 
Bela Ban
Lead JGroups / Clustering Team
JBoss

------------------------------------------------------------------------------
This SF.net email is sponsored by Sprint
What will you do first with EVO, the first 4G phone?
Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
_______________________________________________
Javagroups-development mailing list