Re: [jgroups-users] Strange Behaviour

<< I assume this is sent as a mulicast message: msg.dest == null, correct? 

Sorry for not mentioned that. 
No, I only use unicast message, no a mulicast message at all.

More clues:

1. The problem described here occurs very rare (about 5 proboasls out of 1
million propolsal) in 3-server ensemble and happens only in F2 (The last
replica is received unicast message, sent by leader) (see the scenario
explained in previous email). However, as ensemble size increases to 5,7 or
9, the problem appears very often especially in 7 and 9 ensemble size and
happen very often in last replica that leader sent unicast to it. 

2. The missing proposal will eventually receive but receive later than its
commit.  

The following shows the code for sending proposal, ACK and commit messages:

1. Leader prepares for proposal and send to all followers. Note that leader
proposal its proposal locally, so no need to send to itself:

long new_zxid = getNewZxid();
MessageOrderInfo messageOrderInfo = hdrReq.getMessageOrderInfo();
messageOrderInfo.setOrdering(new_zxid);
ZabHeader hdrProposal = new ZabHeader(ZabHeader.PROPOSAL, messageOrderInfo);
Message proposalMessage = new Message().putHeader(this.id,hdrProposal);
proposalMessage.setBuffer(new byte[1000]);
proposalMessage.setSrc(local_addr);
Proposal p = new Proposal();
p.setMessageOrderInfo(hdrReq.getMessageOrderInfo());
p.AckCount++;
outstandingProposals.put(new_zxid, p);
queuedProposalMessage.put(new_zxid, hdrProposal);
 try {
      for (Address address : zabMembers) {
	  if (address.equals(leader))
	     continue;
	     Message cpy = proposalMessage.copy();
	     cpy.setDest(address);
	     down_prot.down(new Event(Event.MSG, cpy));
	  }
} catch (Exception ex) {
	log.error("failed proposing message to members");
   }

2. Follower receives a proposal, store it locally, and send an ACK to
leader:

private void sendACK(ZabHeader hdrAck) {
	MessageOrderInfo messageOrderInfo = hdrAck.getMessageOrderInfo();
	queuedProposalMessage.put(messageOrderInfo.getOrdering(), hdrAck);
	ZabHeader hdrACK = new ZabHeader(ZabHeader.ACK,
messageOrderInfo.getOrdering());
	Message ACKMessage = new Message(leader).putHeader(this.id, hdrACK);
	try {
	     down_prot.down(new Event(Event.MSG, ACKMessage));
	} catch (Exception ex) {
		log.error("failed sending ACK message to Leader");
	}
}

3. Leader sends commit message to all, including itself:

private synchronized void commit(long zxidd) {
	if (zxidd != lastZxidCommitted + 1) {
		if (log.isDebugEnabled()){
		    log.debug("delivering Zxid out of order "+zxidd + " should be " +
lastZxidCommitted + 1);
		}
	}
	ZabHeader hdrCommit = new ZabHeader(ZabHeader.COMMIT, zxidd);
	Message commitMessage = new Message().putHeader(this.id, hdrCommit);
	for (Address address : zabMembers) {
	     Message cpy = commitMessage.copy();
	     cpy.setDest(address);			
	     down_prot.down(new Event(Event.MSG, cpy));
	}
}


I look forward to hearing from you.

Thank you indeed

Ibrahim





--
View this message in context: http://jgroups.1086181.n5.nabble.com/Strange-Behaviour-tp11141p11143.html
Sent from the JGroups - General mailing list archive at Nabble.com.

------------------------------------------------------------------------------