Re: [jgroups-users] Cluster configuration suggestions.

Bela,

Thanks for the reply.  It will take me some time to go through and test
everything with the changes.

I have had a lot of logging on in past tests (TRACE level) so I¹ll go back
through that to see if it shows anything.
I¹m 99% certain AWS_PING is returning the members correctly - the original
members, the reduced member set when a node is killed, then the new member
set when the new node is brought online.  I can connect to each node with
jconsole and review the JMX data, so I could see the state of membership
at any point.

My first pass at this will be:

* upgrade to latest jgroups (3.6.7 it seems?)
* change configuration as suggested
* re-run my current tests

Depending on the outcome I¹ll proceed from there (probably to add the
tests you mention).

I note your statements about the uneven memory sizes, but at least in my
case I have more than enough memory head room to avoid OOME.  Still this
is something I should probably revisit in the future to improve the system.

Thanks,

Chris


On 27/01/2016 20:30, "Questions/problems related to using JGroups"
<[email protected]> wrote:

>Hi Chris,
>
>comments inline
>
>On 27/01/16 05:28, Questions/problems related to using JGroups wrote:
>> Hi.
>>
>> I¹m using jgroups 3.6.4 within Infinispan 8.1.0 (embedded in my
>> application).  I¹ve got a 5 node cluster (running in AWS) and I¹m having
>> some issues getting it performing correctly.  I¹ve been reading a lot of
>> the documentation on jgroups, trying to understand the configuration
>> options possible.  While I think I¹m getting a grasp of it, trying to
>> get a good set of values for this configuration is not that easy,
>> especially when I¹m unsure just what values I should use and why.
>>
>> In the happy case, the cluster can see each other ok and entries go into
>> the cluster fine.   For discussion purposes, the number of entries in
>> the cluster is in the 100 to 200K, with the data varying in size of a
>> few K, to maybe 1M,
>
>As an aside and unrelated to this problem, adding a few very large
>values can lead to an uneven use of memory across the grid: based on the
>consistent hash of the key, a key/value pair is assigned a server (and a
>backup server) to host it. If we have a cluster {A,B,C,D,E} and 3 very
>large values x1,x2,x3, then with a bit of bad luck (and a bad CH impl),
>x1,x2,x3 could also end up on B and C. possibly leading to an OOME on
>B,C, while the other nodes are fine.
>
>I suggest chunking your data so all values have more or less the same
>size. By more or less, I mean stay within a couple of orders of
>magnitudes... :-)
>
>There's also a GridFilesystem, which provides an I/O abstraction to a
>grid and internally uses chunking.
>
>> updating at around 250+ entries per sec across the
>> cluster.  For production, the number of entries will be at least 10 to
>> 100 times this figure and I¹d really like to increase the update rate as
>> well.   I¹ve yet to really test concurrent read/writes from the system,
>> but read performance with no updates was not a problem.
>
>OK. I suggest run a few perftests such as IspnPerfTest (Infinispan
>related) or UPerf (JGroups) to get a feeling of what perf you're likely
>to get on your N-node cluster, with varying number of reader/writer
>threads and data sizes.
>
>> My main issue is when the cluster splits for whatever reason, e.g. if 1
>>node is killed.
>
>What do you mean here? A killed node is removed from the view and there
>should not be a split.
>
>>If this happens, all the (remaining) nodes fall out of
>> the cluster and become isolated islands, never to merge / rejoin each
>> other.   A new node spun up also seems to remain isolated.   This is not
>> ideal obviously!
>
>Yes, obviously!
>
>Killing a single member should NOT lead to a split cluster! That's
>problem #1 that we need to tackle
>
>To find out what's going on, I suggest enable tracing on
>- FD_SOCK/FD_ALL/FD: find out why nodes are suspected / excluded
>- MERGE3: this should merge split clusters back into a single cluster
>
>If you can debug this, set a breakpoint into MERGE3.InfoSender.run()
>[line 388] and see which responses you get.
>
>So here are my top suspects which may cause this:
>
>#1 AWS_PING
>- If tracing above shows that the modified AWS_PING doesn't return all
>members, the merge will never succeed. To confirm / reject this, replace
>AWS_PING with S3_PING
>- I assume if you run this system in a local network, not in AWS, a
>member leaving doesn't cause a split and if there was a split, it would
>heal again? This would also point to AWS/AWS_PING/TCP_NIO
>
>#2 TCP_NIO
>- This is very old and deprecated
>- I suggest switch to TCP (or TCP_NIO2 in a later JGroups version)
>
>#3 AWS security policy
>- Can you open all ports, just to see if this fixes things?
>
>#4 Your configuration
>- Thread pool sizing etc, I'll comment on this further down
>
>
>
><TCP_NIO bind_port="7800"
>
>Replace this with TCP or TCP_NIO2
>
>
>
>> sock_conn_timeout="10000"
>
>Reduce to 300
>(In general I suggest use the stock tcp.xml file and modify it slightly)
>
>
>> internal_thread_pool.enabled="true"
>>
>> internal_thread_pool.min_threads="2"
>>
>> internal_thread_pool.max_threads="500"
>>
>> internal_thread_pool.keep_alive_time="15000"
>>
>> internal_thread_pool.queue_enabled="true"
>>
>> internal_thread_pool.queue_max_size="50000"
>>
>> internal_thread_pool.rejection_policy="discard"
>
>
>Set queue_enabled to false
>
>
>
>> thread_pool.enabled="true"
>>
>> thread_pool.min_threads="2"
>>
>> thread_pool.max_threads="500"
>>
>> thread_pool.keep_alive_time="15000"
>>
>> thread_pool.queue_enabled="true"
>>
>> thread_pool.queue_max_size="50000"
>>
>> thread_pool.rejection_policy="discard"
>
>
>
>Set queue_enabled to false
>
>
>
>> oob_thread_pool.enabled="true"
>>
>> oob_thread_pool.min_threads="1"
>>
>> oob_thread_pool.max_threads="500"
>>
>> oob_thread_pool.keep_alive_time="30000"
>>
>> oob_thread_pool.queue_enabled="true"
>>
>> oob_thread_pool.queue_max_size="10000"
>>
>> oob_thread_pool.rejection_policy="discard"/>
>
>
>Set queue_enabled to false
>
>
>> <com.meltmedia.jgroups.aws.AWS_PING
>
>
>Try S3_PING or TCPGOSSIP. The latter requires a GossipRouter, but if
>this works, then it's cler that AWS_PING has an issue
>
>
>
><FD_SOCKbind_addr="GLOBAL"num_tries="3"get_cache_timeout="10000"sock_conn_
>timeout="10000"suspect_msg_interval="15000"/>
>
>Use the defaults, e.g. sock_conn_timeout="1000"
>
>
>
>> <pbcast.NAKACK2use_mcast_xmit="false"
>>
>> xmit_interval="500"
>
>1000 would be sufficient
>
>
>
><pbcast.GMSprint_local_addr="true"join_timeout="60000"leave_timeout="15000
>"merge_timeout="60000"resume_task_timeout="15000"
>
>
>join_timeout of 60s? This means that first member will take 60s to
>join!!! BAD! Set it to 3000.
>
>
>
>> In the logs I get various exceptions, including these after the
>>incident has happened:
>>
>> DEBUG [2016-01-27 04:18:47,750] unknown.jul.logger: Read operation on
>>socket failed
>> ! java.io.IOException: Peer closed socket
>> ! at 
>>org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTa
>>bleNIO.java:855)
>> ! at 
>>org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTabl
>>eNIO.java:808)
>> ! at 
>>org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.
>>java:741)
>> ! at java.lang.Thread.run(Thread.java:745)
>
>
>That may be normal, but - again - don't use TCP_NIO!
>
>
>
>> DEBUG [2016-01-27 04:18:43,591] unknown.jul.logger: Read operation on
>>socket failed
>> ! java.io.IOException: Connection reset by peer
>> ! at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>> ! at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>> ! at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>> ! at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>> ! at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
>> ! at 
>>org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTa
>>bleNIO.java:852)
>> ! at 
>>org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTabl
>>eNIO.java:808)
>> ! at 
>>org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.
>>java:741)
>> ! at java.lang.Thread.run(Thread.java:745)
>
>Same thing
>
>
>
>-- 
>Bela Ban, JGroups lead (http://www.jgroups.org)
>
>
>--------------------------------------------------------------------------
>----
>Site24x7 APM Insight: Get Deep Visibility into Application Performance
>APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>Monitor end-to-end web transactions and take corrective actions now
>Troubleshoot faster and improve end-user experience. Signup Now!
>http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
>_______________________________________________
>javagroups-users mailing list
>[email protected]
>https://lists.sourceforge.net/lists/listinfo/javagroups-users


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140