Re: [jgroups-users] Strange Behaviour
Questions/problems related to using JGroups <[email protected]>
| Newsgroups | gmane.comp.java.javagroups.general |
|---|---|
| Message-ID | <[email protected]> |
On 29/07/16 00:00, Questions/problems related to using JGroups wrote:
> Hi Folks,
>
> I build a protocol on the top of JGroups. It is just simple leader-based
> state machine replication which uses following steps to process state change
> request:
>
> 1. Leader sends a proposal (including state change request) to all Followers
> (replicas).
I assume this is sent as a mulicast message: msg.dest == null, correct?
> 2. Upon receiving proposal, replicas send ACK to the leader.
OK
> 3. Upon receiving ACKs from a majority of replicas, leader sends commit
> message to all, informing all replicas to deliver the request.
Is this message also sent as a multicast (dest == null) message?
FIFO order is guaranteed by JGroups for (1) multicasts and (2) between
individual peers.
****
There are NO ordering guarantees between multicasts and unicasts! If you
send multicast M1 followed by unicast U2 (to an individual member), then
the delivery order can be M1 -> U1 or U1 -> M1!
****
This means that you need to send both proposals and commits as
multicasts, or send both as unicasts, but don't mix.
If this doesn't fix your problem, send me a small prototype that
reproduces the problem, and I'll take a look.
> 4. Upon receiving commit message, all replicas deliver the state change
> locally.
>
> Testbed:
> I have a cluster of 3 machines (one machine dedicates for leader and two
> machines dedicate for followers (replicas)). Also, I have 10 machines
> operate as clients, sending 1 million request (state change requests) in
> total. The request size is 1000 bytes. To send a request between nodes, I
> use the JGroups regular message (the default) which is essential for
> protocol requirement, the order.
>
> Problem:
> There are some state change requests, its commit messages receives before
> corresponding proposals. Taking this scenario:
> 1. Leader, L, sends proposal, P1 (it consists of state change request of
> size 1000 bytes) to followers F1 and F2 respectively.
> 2. Upon receiving proposal, F1 sends ACK for P1 to L while F2 has not
> received P1 yet.
> 3. Upon receiving ACK from F1, L sends commit message for P1 to all because
> leader has received ACKs from majority of replicas (from itself and from
> F1).
> 4. Upon receiving commit message, F1 delivers P1 because it knows about the
> state change (F1 receives P1), BUT F2 receives commit for P1 but not yet
> receiving P1!!!.
>
> Observations:
> 1. The problem is always appeared in F2 not in F1. A possible explanation
> for this might be that as a Leader always sends proposal in order of F1 then
> F2 which means F2 is the last replica who receives messages from the leader
> !!!.
> 2. Proposal message payload size is 1000 bytes whereas commit message
> payload size is just proposal id, which equals to java long primitive type.
> Although proposal message is always sent before commit message, the larger
> payload of proposal message leads the commit message arrives before the
> proposal message !!!.
>
> BUT the above observation should not occur at all because we use JGroups
> regular message (the default) which means the recipient should receive the
> message in order they have been sent.
>
> Any thought?
>
> Thank you indeed
>
> Ibrahim
>
> The following is the protocol stack that I use.
>
>
>
>
> <config xmlns="urn:org:jgroups"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="urn:org:jgroups
> http://www.jgroups.org/schema/JGroups-3.3.xsd">
> <UDP
> mcast_port="${jgroups.udp.mcast_port:45588}"
> tos="8"
> max_bundle_size="64K"
> max_bundle_timeout="30"
> enable_diagnostics="true"
> thread_naming_pattern="cl"
> ip_mcast="false"
> timer_type="new3"
> timer.min_threads="10"
> timer.max_threads="10"
> ignore_dont_bundle="false"
> timer.keep_alive_time="3000"
> timer.queue_max_size="500"
> timer.rejection_policy="abort"
> thread_pool.enabled="true"
> thread_pool.min_threads="2"
> thread_pool.max_threads="8"
> thread_pool.keep_alive_time="5000"
> thread_pool.queue_enabled="true"
> thread_pool.queue_max_size="10000"
> thread_pool.rejection_policy="discard"
> oob_thread_pool.enabled="true"
> oob_thread_pool.min_threads="1"
> oob_thread_pool.max_threads="8"
> oob_thread_pool.keep_alive_time="5000"
> oob_thread_pool.queue_enabled="false"
> oob_thread_pool.queue_max_size="100"
> oob_thread_pool.rejection_policy="discard"/>
>
>
> <FILE_PING
> timeout="2000"
> num_initial_members="20"
> location="/home/pg/p13/a6915654/ "/>
> <MERGE2 max_interval="30000"
> min_interval="10000"/>
> <FD_SOCK/>
> <FD_ALL/>
> <VERIFY_SUSPECT timeout="1500" />
> <BARRIER/>
> <pbcast.NAKACK2 xmit_interval="1000"
> xmit_table_num_rows="100"
> xmit_table_msgs_per_row="2000"
> xmit_table_max_compaction_time="30000"
> max_msg_batch_size="500"
> use_mcast_xmit="false"
> discard_delivered_msgs="true"/>
> <UNICAST3 xmit_interval="500"
> xmit_table_num_rows="100"
> xmit_table_msgs_per_row="2000"
> xmit_table_max_compaction_time="60000"
> conn_expiry_timeout="0"
> max_msg_batch_size="500"/>
> <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
> max_bytes="4M"/>
> <pbcast.GMS print_local_addr="true" join_timeout="3000"
> view_bundling="true"/>
> <UFC/>
> <MFC/>
> <RSVP resend_interval="2000" timeout="10000"/>
> <Myprotocol/>
> <FRAG2 frag_size="60K" />
> <pbcast.STATE_TRANSFER />
> </config>
>
>
>
>
>
> --
> View this message in context: http://jgroups.1086181.n5.nabble.com/Strange-Behaviour-tp11141.html
> Sent from the JGroups - General mailing list archive at Nabble.com.
>
> ------------------------------------------------------------------------------
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>
--
Bela Ban, JGroups lead (http://www.jgroups.org)
------------------------------------------------------------------------------