Re: [jgroups-users] Strange Behaviour

On 29/07/16 16:25, Questions/problems related to using JGroups wrote:
> << ************************************************************
> << Ah, OK, I found the problem in your code (looking at the code below):
> << ************************************************************
>
> << - You have members A,B,C,D,E. The majority is 3
> <<- A (leader) sends the proposal to B
> << - A gets the ACK from itself (1 ack)
> <<- A sends the proposal to C
> <<- A gets the ACK from B (2 acks)
> <<- A sends the proposal to D
> <<- A gets the ACK from C (3 acks)
> <<- A now has 3 acks as therefore starts sending COMMIT messages
> <<- A sends the commit message to B, C,D,E
> <<- E receives the COMMIT message
> <<- A only now sends the proposal to E
>
> A great catch.
>
> Questions arise:
> 1. I assume you meant the above steps are possible to occur (do not always
> happen) is it correct?

Yes. I just wanted to find *one* sequence of events that leads to 
incorrect behavior and breaks your algorithm.

> 2. As leader sends proposal, P1, to A,B,C,D,E, and doing this locally in
> same for loop, how can an ACK for proposal, P1, for  replica C receives
> before leader finishing sending P1 to all?

Well, L sends P1 to C *asynchronously*. This means L doesn't wait for 
C's ACK until it sends P1 to D. So when JChannel.send() returns, that 
means P1 was put on the wire, but it does not mean that C already 
received P1.

If you want synchronous behavior, you'd have to use RpcDispatcher. 
However, I don't recommend this as it will kill performance.

> Does that occur because thread-pool is blocked from sending subsequent
> unicast message until current unicast message arrives at the bottom of the
> protocol stack (UDP).

Not at all. Sending is completely concurrent, it could even be that P2 
from L is received before P1 from L by some members. What's important is 
that *delivery* of P1 occurs before delivery of P2 everywhere.

> <<- When getting a COMMIT(P) for which no proposal P has been received,
> << queue P. When P is received, commit it immediately. This might lead to
> << an incorrect history of commits though...
>
> Yes, it is good solution, and it works fine in 3 and 5-servers ensemble but
> it causes deliver order violation in 7 and 9-servers ensemble.
>
> I am thinking for another solution which is that leader processes proposal
> and commit messages from one thread pool. This avoids commit message to
> arrive before its proposal message. But it may cause delay in latency as it
> is possible to have many proposals ahead in queue to send before sending
> commit message.

What do you mean by "one thread pool"? As long as a thread pool has more 
than 1 threads in it, things are asynchronous.

The flaw in your proposal is that you start sending COMMIT messages as 
soon as you've received a majority of acks for a given proposal. This is 
good performance-wise, as you don't want to wait for *all* acks, but 
means acks for a given proposal can cross COMMITs for the same proposal, 
and thus some members (at the end of the list) may receive COMMITs 
before proposals.

In Raft, this is solved by always sending the previous index, current 
index and term with an append/commit message, so all changes are 
appended/committed in order of index (which is like a sequence number).

As you're apparently implementing ZAB in JGroups, I suggest take a look 
at how ZAB (with which I'm not familiar) does it.
Cheers,

> What do you think?
>
> Thank you indeed
>
> Ibrahim
>
>
>
>
> --
> View this message in context: http://jgroups.1086181.n5.nabble.com/Strange-Behaviour-tp11141p11145.html
> Sent from the JGroups - General mailing list archive at Nabble.com.
>
> ------------------------------------------------------------------------------
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)


------------------------------------------------------------------------------