[jgroups-users] Cluster configuration suggestions.

Hi.

I’m using jgroups 3.6.4 within Infinispan 8.1.0 (embedded in my application).  I’ve got a 5 node cluster (running in AWS) and I’m having some issues getting it performing correctly.  I’ve been reading a lot of the documentation on jgroups, trying to understand the configuration options possible.  While I think I’m getting a grasp of it, trying to get a good set of values for this configuration is not that easy, especially when I’m unsure just what values I should use and why.

In the happy case, the cluster can see each other ok and entries go into the cluster fine.   For discussion purposes, the number of entries in the cluster is in the 100 to 200K, with the data varying in size of a few K, to maybe 1M, updating at around 250+ entries per sec across the cluster.  For production, the number of entries will be at least 10 to 100 times this figure and I’d really like to increase the update rate as well.   I’ve yet to really test concurrent read/writes from the system, but read performance with no updates was not a problem.

My main issue is when the cluster splits for whatever reason, e.g. if 1 node is killed.  If this happens, all the (remaining) nodes fall out of the cluster and become isolated islands, never to merge / rejoin each other.   A new node spun up also seems to remain isolated.   This is not ideal obviously!

Hence, I think I’ve got some configuration wrong or I need to do something more for these situations.

Here’s the jgroups configuration I’m using:


<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

        xmlns="urn:org:jgroups"

        xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">

    <TCP_NIO bind_port="7800"

         bind_addr="GLOBAL"

         recv_buf_size="${tcp.recv_buf_size:5M}"

         send_buf_size="${tcp.send_buf_size:5M}"

         max_bundle_size="64K"

         max_bundle_timeout="10"

         use_send_queues="true"

         sock_conn_timeout="10000"

         conn_expire_time="15000"

         reaper_interval="30000"

         timer_type="new3"

         timer.min_threads="4"

         timer.max_threads="10"

         timer.keep_alive_time="3000"

         timer.queue_max_size="5000"

         log_discard_msgs="true"

         log_discard_msgs_version="true"

         loopback="true"

         internal_thread_pool.enabled="true"

         internal_thread_pool.min_threads="2"

         internal_thread_pool.max_threads="500"

         internal_thread_pool.keep_alive_time="15000"

         internal_thread_pool.queue_enabled="true"

         internal_thread_pool.queue_max_size="50000"

         internal_thread_pool.rejection_policy="discard"

         thread_pool.enabled="true"

         thread_pool.min_threads="2"

         thread_pool.max_threads="500"

         thread_pool.keep_alive_time="15000"

         thread_pool.queue_enabled="true"

         thread_pool.queue_max_size="50000"

         thread_pool.rejection_policy="discard"

         oob_thread_pool.enabled="true"

         oob_thread_pool.min_threads="1"

         oob_thread_pool.max_threads="500"

         oob_thread_pool.keep_alive_time="30000"

         oob_thread_pool.queue_enabled="true"

         oob_thread_pool.queue_max_size="10000"

         oob_thread_pool.rejection_policy="discard"/>


<com.meltmedia.jgroups.aws.AWS_PING

    async_discovery="true"

    async_discovery_use_separate_thread_per_request="true"

    always_send_physical_addr_with_discovery_request="true"

    return_entire_cache="true"

    send_cache_on_join="true"

    break_on_coord_rsp="true"

    stagger_timeout="500"

    discovery_rsp_expiry_time="60000"

timeout="60000"

port_number="7800"

port_range="0"

tags="Name,Environment" />


    <MERGE3 min_interval="5000"

            max_interval="10000"

            check_interval="15000"/>


    <FD_SOCK bind_addr="GLOBAL" num_tries="3" get_cache_timeout="10000" sock_conn_timeout="10000" suspect_msg_interval="15000"/>

    <!-- <FD timeout="3000" max_tries="3"/>  -->

    <FD_ALL timeout="60000" interval="3000" timeout_check_interval="2000"/>

    <VERIFY_SUSPECT timeout="15000" num_msgs="3" bind_addr="GLOBAL"/>

    <BARRIER/>


    <pbcast.NAKACK2 use_mcast_xmit="false"

                    xmit_interval="500"

                    xmit_table_num_rows="100"

                    xmit_table_msgs_per_row="2000"

                    xmit_table_max_compaction_time="30000"

                    max_msg_batch_size="500"

                    discard_delivered_msgs="true"/>


    <UNICAST3 xmit_table_num_rows="100"

              xmit_table_msgs_per_row="1000"

              xmit_table_max_compaction_time="30000"

              max_msg_batch_size="500"/>


    <pbcast.STABLE stability_delay="2000" desired_avg_gossip="60000"

                   max_bytes="2M"/>

    <pbcast.GMS print_local_addr="true" join_timeout="60000" leave_timeout="15000" merge_timeout="60000" resume_task_timeout="15000"

                view_bundling="true" max_join_attempts="0"/>

    <MFC max_credits="1M"

         min_threshold="0.25"/>

    <UFC max_credits="1M"

         min_threshold="0.25"/>

    <FRAG2 frag_size="60K"/>

    <!--  RSVP resend_interval="2000" timeout="10000"/ -->

    <pbcast.STATE_SOCK bind_port="7899" bind_addr="GLOBAL" buffer_size="100000" max_pool="500" pool_thread_keep_alive="60000"/>


</config>



In the logs I get various exceptions, including these after the incident has happened:

DEBUG [2016-01-27 04:18:47,750] unknown.jul.logger: Read operation on socket failed
! java.io.IOException: Peer closed socket
! at org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:855)
! at org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:808)
! at org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:741)
! at java.lang.Thread.run(Thread.java:745)

And:

DEBUG [2016-01-27 04:18:43,591] unknown.jul.logger: Read operation on socket failed
! java.io.IOException: Connection reset by peer
! at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
! at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
! at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
! at sun.nio.ch.IOUtil.read(IOUtil.java:197)
! at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
! at org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:852)
! at org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:808)
! at org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:741)
! at java.lang.Thread.run(Thread.java:745)


Any suggestions would be greatly appreciated!  If you need more information from me, please ask.

Thanks,

Chris

P.S. In case this is a problem in the discovery mechanism being used, I did have to update/change the AWS_PING code to make it work with the current version of jgroups.  The major piece of work required was to implement the findMembers method.  I did this by significantly ripping off the TCPPING class version. :)

Here’s the method:


protected void findMembers(List<Address> members, boolean initial_discovery, Responses responses) {

if (physical_addr == null) {

physical_addr = (PhysicalAddress) down(new Event(Event.GET_PHYSICAL_ADDRESS, local_addr));

}


// https://issues.jboss.org/browse/JGRP-1670

PingData data = new PingData(local_addr, false, org.jgroups.util.UUID.get(local_addr), physical_addr);

PingHeader hdr = new PingHeader(PingHeader.GET_MBRS_REQ).clusterName(cluster_name);


List<IpAddress> clusterMembers = fetchClusterMembers(initial_discovery);

clusterMembers.forEach(member -> {

if (!member.equals(physical_addr)) {

final Message msg = new Message(member)

.setFlag(Message.Flag.INTERNAL, Message.Flag.DONT_BUNDLE, Message.Flag.OOB)

.putHeader(this.id, hdr).setBuffer(marshal(data));


if (async_discovery_use_separate_thread_per_request) {

timer.execute(new Runnable() {

public void run() {

if (log.isDebugEnabled()) {

log.debug("From " + local_addr + ": sending discovery request to " + msg.getDest());

}

down_prot.down(new Event(Event.MSG, msg));

}

});

} else {

if (log.isDebugEnabled()) {

log.debug("From " + local_addr + ": sending discovery request to " + msg.getDest());

}

down_prot.down(new Event(Event.MSG, msg));

}

}

});

}

I can see in the logs that the AWS_PING seems to work, e.g.

DEBUG [2016-01-27 04:17:43,507] com.meltmedia.jgroups.aws.AWS_PING: Describing AWS instances with the following filters [[{Name: tag:Name,Values: [XXXX]}, {Name: tag:Environment,Values: [XXXX]}]]
DEBUG [2016-01-27 04:17:43,507] com.meltmedia.jgroups.aws.AWS_PING: Making AWS Request {{InstanceIds: [],Filters: [{Name: tag:Name,Values: [XXXX]}, {Name: tag:Environment,Values: [XXXX]}],}}
DEBUG [2016-01-27 04:17:43,586] com.meltmedia.jgroups.aws.AWS_PING: Instances found [[10.2.89.149, 10.2.89.145, 10.2.89.144, 10.2.89.146, 10.2.90.171]]
INFO  [2016-01-27 04:17:43,586] com.meltmedia.jgroups.aws.AWS_PING: Instances found [[10.2.89.149, 10.2.89.145, 10.2.89.144, 10.2.89.146, 10.2.90.171]]
INFO  [2016-01-27 04:17:43,586] com.meltmedia.jgroups.aws.AWS_PING: Adding cluster member 10.2.89.149:7800
INFO  [2016-01-27 04:17:43,586] com.meltmedia.jgroups.aws.AWS_PING: Adding cluster member 10.2.89.145:7800
INFO  [2016-01-27 04:17:43,586] com.meltmedia.jgroups.aws.AWS_PING: Adding cluster member 10.2.89.144:7800
INFO  [2016-01-27 04:17:43,586] com.meltmedia.jgroups.aws.AWS_PING: Adding cluster member 10.2.89.146:7800
INFO  [2016-01-27 04:17:43,586] com.meltmedia.jgroups.aws.AWS_PING: Adding cluster member 10.2.90.171:7800
DEBUG [2016-01-27 04:17:43,586] com.meltmedia.jgroups.aws.AWS_PING: From ip-10-2-90-171-44825: sending discovery request to 10.2.89.145:7800
DEBUG [2016-01-27 04:17:43,586] com.meltmedia.jgroups.aws.AWS_PING: From ip-10-2-90-171-44825: sending discovery request to 10.2.89.149:7800
DEBUG [2016-01-27 04:17:43,586] com.meltmedia.jgroups.aws.AWS_PING: From ip-10-2-90-171-44825: sending discovery request to 10.2.89.144:7800

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users