[jgroups-dev] TCPPING merge issue

Hi,

We are having issues with forming jgroups clusters. A joiner may get
'stuck' indefinitely.  We are using a TCPPING based jgroups stack,
which is fairly similar to tcp.xml with jgroups 2.9.0GA (mainly longer
timeouts).

As best as I can analyze it, here is my explaination of the scenario
that is affecting us.

{A,B} are happily running; their protocol stacks have
TCPPING(initial_hosts=A,B)

C wants to join the cluster, it has TCPPING(initial_hosts=A,B,C).

On C creating its JChannels(), at this specific point, jgroups is
susceptable to timing issues. Normally, C discovers A or B, and the
procedes with a JOIN_REQ. However, if TCPPING's first FIND_INITIAL_MBRS
operation by C times out with no members discovered, C will become
a singleton group {C}.

   "no initial members discovered: creating group as first member"

Say C's timing issues then get resolved, communication between the
hosts are at a reasonable state. Typically, it then leads to:

 - {C} resending GET_MBRS_REQ to A and B

 - A and B happily reply GET_MBRS_RSP

 - based on the response, C rightfully decides it needs to merge
   "found different views : ....  sending up MERGE event with merge
   participants.."

 - here's where C run into issues. C is the coordinator for {C}. It
defers to the {A,B} coordinator to initiate the merge.
   "determining merge leader" ..  "I am not the merge leader, waiting
   for merge leader .. to initiate merge"

 - however coordinator in {A,B} never initiates the merge, because
C is not in their initial member list, nor does C get added to the
(dynamic) member list despite their MBRS dialog

 - so, C will then start over, sending GET_MBRS_REQ to same outcome,
never joining {A,B}


I dont have enough jgroups experience yet to know what the fix should
be. Some thoughts:

It makes sense for C to become singleton group, but should it
periodically send JOIN_REQ if it remains singleton? (but what happens
if we get D in the mix?)

Or should the coordinator in {A,B} include C in its member list,
via knowledge from their GET_MBRS exchange. Is the the logic (or
actual implementation) as commented in Discovery.up() correct? Maybe
a logic tweak if GET_MBRS_REQ is received and it is from an unknown member?
Quoted here:

 >   * If the header is PingHeader.GET_MBRS_REQ then the PING protocol
 >   * will PassDown a PingRequest message
 >   * If the header is PingHeader.GET_MBRS_RSP we will add the
 >   * message to the initial members
 >   * vector and wake up any waiting threads.

I have full logs with 3 nodes upon request.

Thanks,
-at

------------------------------------------------------------------------------
This SF.net email is sponsored by Sprint
What will you do first with EVO, the first 4G phone?
Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
_______________________________________________
Javagroups-development mailing list