[jgroups-users] Node hanging when no coordinator set

Hi all,

I have been investigating a problem we are experiencing with JGroups within our application, the application uses JDBC_PING
for discovery.

I may have stumbled upon an edge case and I am curious to know what you guys think of it.

There two nodes which are about to start-up, the first node is killed after having written its record in the DB, which reports
its existence, but *before* having set itself as the cluster's coordinator in the DB. When the second node starts up it hangs in
the while loop at line 63 of ClientGmsImpl, joinInternal()

https://github.com/belaban/JGroups/blob/master/src/org/jgroups/protocols/pbcast/ClientGmsImpl.java


// We didn't get any coord responses; all responses were clients. If I'm the first of the sorted clients
// I'll become coordinator. The others will wait and then retry the discovery and join process
if(coords == null) { // e.g. because we have all clients only
   if(firstOfAllClients(mbr, responses))
        return;
   continue;
}

In my scenario the sorting algorithm in firstOfAllClients()'s client's set contains the first node at the first
position therefore inferring that the node will soon change the state in the DB to reflect the fact that it is
a coordinator.

This is not happening as the first node has been brutally shutdown (kill -9).

By "continuing" the loop will start off again from the beginning never incrementing the join_attempts variable
and never hitting the check

if(gms.max_join_attempts != 0 && join_attempts >= gms.max_join_attempts) {
  log.warn("%s: too many JOIN attempts (%d): becoming singleton", gms.local_addr, join_attempts);
  becomeSingletonMember(mbr);
  return;
}


The second node will then hang forever, I wonder if it feasible to move up the increment of the join_attempts
and remove the continue statement., we could do something along those lines:

while(!leaving) {
[..]

if(installViewIfValidJoinRsp(join_promise, false))
       return;

join_attempts++; // Move the increment up here
// I'll become coordinator. The others will wait and then retry the discovery and join process
if(coords == null) { // e.g. because we have all clients only
     if(firstOfAllClients(mbr, responses))
            return;
}

// Check, it may be null
if(coords != null && coords.size() > 1)
   log.debug("%s: found multiple coords: %s", gms.local_addr, coords);

   for(Address coord: coords) {
       log.debug("%s: sending JOIN(%s) to %s", gms.local_addr, mbr, coord);
        sendJoinMessage(coord, mbr, joinWithStateTransfer, useFlushIfPresent);
        if(installViewIfValidJoinRsp(join_promise, true))
              return;
         log.warn("%s: JOIN(%s) sent to %s timed out (after %d ms), on try %d",
                   gms.local_addr, mbr, coord, gms.join_timeout, join_attempts);
    }

    if(gms.max_join_attempts != 0 && join_attempts >= gms.max_join_attempts) {
          log.warn("%s: too many JOIN attempts (%d): becoming singleton", gms.local_addr, join_attempts);
           becomeSingletonMember(mbr);
           return;
    }





Pietro Paolini
Consultant

Alfa
________________________________
e: [email protected] | w: alfasystems.com<https://www.alfasystems.com>
t: +44 (0) 20 7920-2643 | Moor Place, 1 Fore Street Avenue, London, EC2Y 9DT, GB
________________________________

The contents of this communication are not intended to be binding or constitute any form of offer or acceptance or give rise to any legal obligations on behalf of the sender or Alfa. The views or opinions expressed represent those of the author and not necessarily those of Alfa. This email and any attachments are strictly confidential and are intended solely for use by the individual or entity to whom it is addressed. If you are not the addressee (or responsible for delivery of the message to the addressee) you may not copy, forward, disclose or use any part of the message or its attachments. At present the integrity of email across the internet cannot be guaranteed and messages sent via this medium are potentially at risk. All liability is excluded to the extent permitted by law for any claims arising as a result of the use of this medium to transmit information by or to Alfa or its affiliates.

Alfa Financial Software Ltd
Reg. in England No: 0248 2325

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users