[jgroups-users] Strange Behaviour

Hi Folks,

I build a protocol on the top of JGroups. It is just simple leader-based
state machine replication which uses following steps to process state change
request:

1. Leader sends a proposal (including state change request) to all Followers
(replicas).
2. Upon receiving proposal, replicas send ACK to the leader.
3. Upon receiving  ACKs from a majority of replicas, leader sends commit
message to all, informing all replicas to deliver the request.
4. Upon receiving commit message, all replicas deliver the state change
locally.

Testbed:
I have a cluster of 3 machines (one machine dedicates for leader and two
machines dedicate for followers (replicas)).  Also, I have 10 machines
operate as clients, sending 1 million request (state change requests) in
total. The request size is 1000 bytes. To send a request between nodes, I
use the JGroups regular message (the default) which is essential for
protocol requirement, the order.

Problem:
There are some state change requests, its commit messages receives before
corresponding proposals. Taking this scenario:
1. Leader, L, sends proposal, P1 (it consists of state change request of
size 1000 bytes) to followers F1 and F2 respectively.
2. Upon receiving proposal, F1 sends ACK for P1 to L while F2 has not
received P1 yet.
3. Upon receiving ACK from F1, L sends commit message for P1 to all because
leader has received ACKs from majority of replicas (from itself and from
F1).
4. Upon receiving commit message, F1 delivers  P1 because it knows about the
state change (F1 receives P1), BUT F2 receives commit for P1 but not yet
receiving P1!!!.

Observations:
1. The problem is always appeared in F2 not in F1. A possible explanation
for this might be that as a Leader always sends proposal in order of F1 then
F2 which means F2 is the last replica who receives messages from the leader
!!!. 
2. Proposal message payload size is 1000 bytes whereas commit message
payload size is just proposal id, which equals to java long primitive type.
Although proposal message is always sent before commit message, the larger
payload of proposal message leads the commit message arrives before the
proposal message !!!.

BUT  the above observation should not occur at all because we use JGroups
regular message (the default) which means the recipient should receive the
message in order they have been sent.

Any thought?

Thank you indeed

Ibrahim

The following is the protocol stack that I use.




<config xmlns="urn:org:jgroups"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:org:jgroups
http://www.jgroups.org/schema/JGroups-3.3.xsd">
   <UDP
            mcast_port="${jgroups.udp.mcast_port:45588}"
            tos="8"
            max_bundle_size="64K"
            max_bundle_timeout="30"
            enable_diagnostics="true"
            thread_naming_pattern="cl"
            ip_mcast="false"
            timer_type="new3"
            timer.min_threads="10"
            timer.max_threads="10"
            ignore_dont_bundle="false"
            timer.keep_alive_time="3000"
            timer.queue_max_size="500"
            timer.rejection_policy="abort"
            thread_pool.enabled="true"
            thread_pool.min_threads="2"
            thread_pool.max_threads="8"
            thread_pool.keep_alive_time="5000"
            thread_pool.queue_enabled="true"
            thread_pool.queue_max_size="10000"
            thread_pool.rejection_policy="discard"
            oob_thread_pool.enabled="true"
            oob_thread_pool.min_threads="1"
            oob_thread_pool.max_threads="8"
            oob_thread_pool.keep_alive_time="5000"
            oob_thread_pool.queue_enabled="false"
            oob_thread_pool.queue_max_size="100"
            oob_thread_pool.rejection_policy="discard"/> 
            

    <FILE_PING
            timeout="2000"
            num_initial_members="20"
            location="/home/pg/p13/a6915654/ "/>
    <MERGE2 max_interval="30000"
            min_interval="10000"/>
    <FD_SOCK/>
    <FD_ALL/>
    <VERIFY_SUSPECT timeout="1500"  />
    <BARRIER/>
    <pbcast.NAKACK2 xmit_interval="1000"
                    xmit_table_num_rows="100"
                    xmit_table_msgs_per_row="2000"
                    xmit_table_max_compaction_time="30000"
                    max_msg_batch_size="500"
                    use_mcast_xmit="false"
                    discard_delivered_msgs="true"/>
    <UNICAST3 xmit_interval="500"
              xmit_table_num_rows="100"
              xmit_table_msgs_per_row="2000"
              xmit_table_max_compaction_time="60000"
              conn_expiry_timeout="0"
              max_msg_batch_size="500"/>
    <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
                   max_bytes="4M"/>
     <pbcast.GMS print_local_addr="true" join_timeout="3000"
                view_bundling="true"/>  
     <UFC/>
     <MFC/>
    <RSVP resend_interval="2000" timeout="10000"/>
     <Myprotocol/>
    <FRAG2 frag_size="60K"  />
    <pbcast.STATE_TRANSFER />
</config>





--
View this message in context: http://jgroups.1086181.n5.nabble.com/Strange-Behaviour-tp11141.html
Sent from the JGroups - General mailing list archive at Nabble.com.

------------------------------------------------------------------------------