Re: [jgroups-users] Strange Behaviour

On 29/07/16 13:18, Questions/problems related to using JGroups wrote:
> << I assume this is sent as a mulicast message: msg.dest == null, correct?
>
> Sorry for not mentioned that.
> No, I only use unicast message, no a multicast message at all.

This means that leader L sends proposal P as a unicast message to 
follower F1, then another unicast message to F2.
When a majority of ACK have been received, L sends commit message C as 
another unicast to F1, then another unicast to F2.

Correct?

When those messages are received, is there some custom protocol (I see 
you have Myprotocol in the config) that can potentially change the 
order? Does Myprotocol for example dispatch the incoming messages to a 
thread pool for them to get processed? That would lead to ordering issues...


> More clues:
>
> 1. The problem described here occurs very rare (about 5 proboasls out of 1
> million propolsal) in 3-server ensemble and happens only in F2 (The last
> replica is received unicast message, sent by leader) (see the scenario
> explained in previous email). However, as ensemble size increases to 5,7 or
> 9, the problem appears very often especially in 7 and 9 ensemble size and
> happen very often in last replica that leader sent unicast to it.

If this happens very frequently, you should be able to come up with a 
simple prototype that has a leader continually send proposals and 
commits and stops on the first ordering violation.

I'll be happy to look into this if you can reproduce it.

************************************************************
Ah, OK, I found the problem in your code (looking at the code below):
************************************************************

- You have members A,B,C,D,E. The majority is 3
- A (leader) sends the proposal to B
- A gets the ACK from itself (1 ack)
- A sends the proposal to C
- A gets the ACK from B (2 acks)
- A sends the proposal to D
- A gets the ACK from C (3 acks)
- A now has 3 acks as therefore starts sending COMMIT messages
- A sends the commit message to B, C,D,E
- E receives the COMMIT message
- A only now sends the proposal to E
==> E can get the proposal *after* the commit!

This requires a change in your design. Possible alternatives:

- When getting a COMMIT(P) for which no proposal P has been received, 
queue P. When P is received, commit it immediately. This might lead to 
an incorrect history of commits though...

- Assign monotonically increasing sequence numbers to proposals and 
apply them only in order of their seqnos. This is similar to what Raft does.
Cheers,



> 2. The missing proposal will eventually receive but receive later than its
> commit.
>
> The following shows the code for sending proposal, ACK and commit messages:
>
> 1. Leader prepares for proposal and send to all followers. Note that leader
> proposal its proposal locally, so no need to send to itself:
>
> long new_zxid = getNewZxid();
> MessageOrderInfo messageOrderInfo = hdrReq.getMessageOrderInfo();
> messageOrderInfo.setOrdering(new_zxid);
> ZabHeader hdrProposal = new ZabHeader(ZabHeader.PROPOSAL, messageOrderInfo);
> Message proposalMessage = new Message().putHeader(this.id,hdrProposal);
> proposalMessage.setBuffer(new byte[1000]);
> proposalMessage.setSrc(local_addr);
> Proposal p = new Proposal();
> p.setMessageOrderInfo(hdrReq.getMessageOrderInfo());
> p.AckCount++;
> outstandingProposals.put(new_zxid, p);
> queuedProposalMessage.put(new_zxid, hdrProposal);
>   try {
>        for (Address address : zabMembers) {
> 	  if (address.equals(leader))
> 	     continue;
> 	     Message cpy = proposalMessage.copy();
> 	     cpy.setDest(address);
> 	     down_prot.down(new Event(Event.MSG, cpy));
> 	  }
> } catch (Exception ex) {
> 	log.error("failed proposing message to members");
>     }
>
> 2. Follower receives a proposal, store it locally, and send an ACK to
> leader:
>
> private void sendACK(ZabHeader hdrAck) {
> 	MessageOrderInfo messageOrderInfo = hdrAck.getMessageOrderInfo();
> 	queuedProposalMessage.put(messageOrderInfo.getOrdering(), hdrAck);
> 	ZabHeader hdrACK = new ZabHeader(ZabHeader.ACK,
> messageOrderInfo.getOrdering());
> 	Message ACKMessage = new Message(leader).putHeader(this.id, hdrACK);
> 	try {
> 	     down_prot.down(new Event(Event.MSG, ACKMessage));
> 	} catch (Exception ex) {
> 		log.error("failed sending ACK message to Leader");
> 	}
> }
>
> 3. Leader sends commit message to all, including itself:
>
> private synchronized void commit(long zxidd) {
> 	if (zxidd != lastZxidCommitted + 1) {
> 		if (log.isDebugEnabled()){
> 		    log.debug("delivering Zxid out of order "+zxidd + " should be " +
> lastZxidCommitted + 1);
> 		}
> 	}
> 	ZabHeader hdrCommit = new ZabHeader(ZabHeader.COMMIT, zxidd);
> 	Message commitMessage = new Message().putHeader(this.id, hdrCommit);
> 	for (Address address : zabMembers) {
> 	     Message cpy = commitMessage.copy();
> 	     cpy.setDest(address);			
> 	     down_prot.down(new Event(Event.MSG, cpy));
> 	}
> }
>
>
> I look forward to hearing from you.
>
> Thank you indeed
>
> Ibrahim
>
>
>
>
>
> --
> View this message in context: http://jgroups.1086181.n5.nabble.com/Strange-Behaviour-tp11141p11143.html
> Sent from the JGroups - General mailing list archive at Nabble.com.
>
> ------------------------------------------------------------------------------
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)


------------------------------------------------------------------------------