Re: [jgroups-users] Cluster configuration suggestions.

Hi Chris,

comments inline

On 27/01/16 05:28, Questions/problems related to using JGroups wrote:
> Hi.
>
> I’m using jgroups 3.6.4 within Infinispan 8.1.0 (embedded in my
> application).  I’ve got a 5 node cluster (running in AWS) and I’m having
> some issues getting it performing correctly.  I’ve been reading a lot of
> the documentation on jgroups, trying to understand the configuration
> options possible.  While I think I’m getting a grasp of it, trying to
> get a good set of values for this configuration is not that easy,
> especially when I’m unsure just what values I should use and why.
>
> In the happy case, the cluster can see each other ok and entries go into
> the cluster fine.   For discussion purposes, the number of entries in
> the cluster is in the 100 to 200K, with the data varying in size of a
> few K, to maybe 1M,

As an aside and unrelated to this problem, adding a few very large 
values can lead to an uneven use of memory across the grid: based on the 
consistent hash of the key, a key/value pair is assigned a server (and a 
backup server) to host it. If we have a cluster {A,B,C,D,E} and 3 very 
large values x1,x2,x3, then with a bit of bad luck (and a bad CH impl), 
x1,x2,x3 could also end up on B and C. possibly leading to an OOME on 
B,C, while the other nodes are fine.

I suggest chunking your data so all values have more or less the same 
size. By more or less, I mean stay within a couple of orders of 
magnitudes... :-)

There's also a GridFilesystem, which provides an I/O abstraction to a 
grid and internally uses chunking.

> updating at around 250+ entries per sec across the
> cluster.  For production, the number of entries will be at least 10 to
> 100 times this figure and I’d really like to increase the update rate as
> well.   I’ve yet to really test concurrent read/writes from the system,
> but read performance with no updates was not a problem.

OK. I suggest run a few perftests such as IspnPerfTest (Infinispan 
related) or UPerf (JGroups) to get a feeling of what perf you're likely 
to get on your N-node cluster, with varying number of reader/writer 
threads and data sizes.

> My main issue is when the cluster splits for whatever reason, e.g. if 1 node is killed.

What do you mean here? A killed node is removed from the view and there 
should not be a split.

>If this happens, all the (remaining) nodes fall out of
> the cluster and become isolated islands, never to merge / rejoin each
> other.   A new node spun up also seems to remain isolated.   This is not
> ideal obviously!

Yes, obviously!

Killing a single member should NOT lead to a split cluster! That's 
problem #1 that we need to tackle

To find out what's going on, I suggest enable tracing on
- FD_SOCK/FD_ALL/FD: find out why nodes are suspected / excluded
- MERGE3: this should merge split clusters back into a single cluster

If you can debug this, set a breakpoint into MERGE3.InfoSender.run() 
[line 388] and see which responses you get.

So here are my top suspects which may cause this:

#1 AWS_PING
- If tracing above shows that the modified AWS_PING doesn't return all 
members, the merge will never succeed. To confirm / reject this, replace 
AWS_PING with S3_PING
- I assume if you run this system in a local network, not in AWS, a 
member leaving doesn't cause a split and if there was a split, it would 
heal again? This would also point to AWS/AWS_PING/TCP_NIO

#2 TCP_NIO
- This is very old and deprecated
- I suggest switch to TCP (or TCP_NIO2 in a later JGroups version)

#3 AWS security policy
- Can you open all ports, just to see if this fixes things?

#4 Your configuration
- Thread pool sizing etc, I'll comment on this further down



<TCP_NIO bind_port="7800"

Replace this with TCP or TCP_NIO2



> sock_conn_timeout="10000"

Reduce to 300
(In general I suggest use the stock tcp.xml file and modify it slightly)


> internal_thread_pool.enabled="true"
>
> internal_thread_pool.min_threads="2"
>
> internal_thread_pool.max_threads="500"
>
> internal_thread_pool.keep_alive_time="15000"
>
> internal_thread_pool.queue_enabled="true"
>
> internal_thread_pool.queue_max_size="50000"
>
> internal_thread_pool.rejection_policy="discard"


Set queue_enabled to false



> thread_pool.enabled="true"
>
> thread_pool.min_threads="2"
>
> thread_pool.max_threads="500"
>
> thread_pool.keep_alive_time="15000"
>
> thread_pool.queue_enabled="true"
>
> thread_pool.queue_max_size="50000"
>
> thread_pool.rejection_policy="discard"



Set queue_enabled to false



> oob_thread_pool.enabled="true"
>
> oob_thread_pool.min_threads="1"
>
> oob_thread_pool.max_threads="500"
>
> oob_thread_pool.keep_alive_time="30000"
>
> oob_thread_pool.queue_enabled="true"
>
> oob_thread_pool.queue_max_size="10000"
>
> oob_thread_pool.rejection_policy="discard"/>


Set queue_enabled to false


> <com.meltmedia.jgroups.aws.AWS_PING


Try S3_PING or TCPGOSSIP. The latter requires a GossipRouter, but if 
this works, then it's cler that AWS_PING has an issue



<FD_SOCKbind_addr="GLOBAL"num_tries="3"get_cache_timeout="10000"sock_conn_timeout="10000"suspect_msg_interval="15000"/>

Use the defaults, e.g. sock_conn_timeout="1000"



> <pbcast.NAKACK2use_mcast_xmit="false"
>
> xmit_interval="500"

1000 would be sufficient



<pbcast.GMSprint_local_addr="true"join_timeout="60000"leave_timeout="15000"merge_timeout="60000"resume_task_timeout="15000"


join_timeout of 60s? This means that first member will take 60s to 
join!!! BAD! Set it to 3000.



> In the logs I get various exceptions, including these after the incident has happened:
>
> DEBUG [2016-01-27 04:18:47,750] unknown.jul.logger: Read operation on socket failed
> ! java.io.IOException: Peer closed socket
> ! at org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:855)
> ! at org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:808)
> ! at org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:741)
> ! at java.lang.Thread.run(Thread.java:745)


That may be normal, but - again - don't use TCP_NIO!



> DEBUG [2016-01-27 04:18:43,591] unknown.jul.logger: Read operation on socket failed
> ! java.io.IOException: Connection reset by peer
> ! at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> ! at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> ! at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> ! at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> ! at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
> ! at org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:852)
> ! at org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:808)
> ! at org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:741)
> ! at java.lang.Thread.run(Thread.java:745)

Same thing



-- 
Bela Ban, JGroups lead (http://www.jgroups.org)


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140