Re: [jgroups-users] Cluster configuration suggestions.

I can report that so far, just those changes have done wonders.  It’s
working as I expected.  If a node gets stopped/killed, the others remain
cohesive and start rebalancing.  When a new node joins, it does the state
transfer/rebalance as well.  I’ve got more tests to do, but I’m happier
that it’s looking a lot better now.

So it appears my original bad config values plus the old TCP_NIO were the
main culprits.

Thanks for your help.

Chris

On 29/01/2016 03:31, "Questions/problems related to using JGroups"
<[email protected]> wrote:

>
>
>On 27/01/16 22:22, Questions/problems related to using JGroups wrote:
>> Bela,
>>
>> Thanks for the reply.  It will take me some time to go through and test
>> everything with the changes.
>>
>> I have had a lot of logging on in past tests (TRACE level) so I¹ll go
>>back
>> through that to see if it shows anything.
>> I¹m 99% certain AWS_PING is returning the members correctly - the
>>original
>> members, the reduced member set when a node is killed, then the new
>>member
>> set when the new node is brought online.  I can connect to each node
>>with
>> jconsole and review the JMX data, so I could see the state of membership
>> at any point.
>
>Yes, I don't think the first step of this investigation should be to see
>why the merge isn't working, it should be to find out why a
>crashed/killed member leads to a split brain. This should definitely not
>be the case and I've never seen such behavior!
>
>
>> My first pass at this will be:
>>
>> * upgrade to latest jgroups (3.6.7 it seems?)
>
>EXCELLENT !!!! :-)
>
>> * change configuration as suggested
>> * re-run my current tests
>
>OK
>
>  > Depending on the outcome I¹ll proceed from there (probably to add the
>> tests you mention).
>>
>> I note your statements about the uneven memory sizes, but at least in my
>> case I have more than enough memory head room to avoid OOME.  Still this
>> is something I should probably revisit in the future to improve the
>>system.
>
>Yes, I was just concerned about the (extremely unlikely) scenario where
>a crappy consistent hash algorithm would allocate all large values on
>the _same_ server.
>
>
>> Thanks,
>>
>> Chris
>>
>>
>> On 27/01/2016 20:30, "Questions/problems related to using JGroups"
>> <[email protected]> wrote:
>>
>>> Hi Chris,
>>>
>>> comments inline
>>>
>>> On 27/01/16 05:28, Questions/problems related to using JGroups wrote:
>>>> Hi.
>>>>
>>>> I¹m using jgroups 3.6.4 within Infinispan 8.1.0 (embedded in my
>>>> application).  I¹ve got a 5 node cluster (running in AWS) and I¹m
>>>>having
>>>> some issues getting it performing correctly.  I¹ve been reading a lot
>>>>of
>>>> the documentation on jgroups, trying to understand the configuration
>>>> options possible.  While I think I¹m getting a grasp of it, trying to
>>>> get a good set of values for this configuration is not that easy,
>>>> especially when I¹m unsure just what values I should use and why.
>>>>
>>>> In the happy case, the cluster can see each other ok and entries go
>>>>into
>>>> the cluster fine.   For discussion purposes, the number of entries in
>>>> the cluster is in the 100 to 200K, with the data varying in size of a
>>>> few K, to maybe 1M,
>>>
>>> As an aside and unrelated to this problem, adding a few very large
>>> values can lead to an uneven use of memory across the grid: based on
>>>the
>>> consistent hash of the key, a key/value pair is assigned a server (and
>>>a
>>> backup server) to host it. If we have a cluster {A,B,C,D,E} and 3 very
>>> large values x1,x2,x3, then with a bit of bad luck (and a bad CH impl),
>>> x1,x2,x3 could also end up on B and C. possibly leading to an OOME on
>>> B,C, while the other nodes are fine.
>>>
>>> I suggest chunking your data so all values have more or less the same
>>> size. By more or less, I mean stay within a couple of orders of
>>> magnitudes... :-)
>>>
>>> There's also a GridFilesystem, which provides an I/O abstraction to a
>>> grid and internally uses chunking.
>>>
>>>> updating at around 250+ entries per sec across the
>>>> cluster.  For production, the number of entries will be at least 10 to
>>>> 100 times this figure and I¹d really like to increase the update rate
>>>>as
>>>> well.   I¹ve yet to really test concurrent read/writes from the
>>>>system,
>>>> but read performance with no updates was not a problem.
>>>
>>> OK. I suggest run a few perftests such as IspnPerfTest (Infinispan
>>> related) or UPerf (JGroups) to get a feeling of what perf you're likely
>>> to get on your N-node cluster, with varying number of reader/writer
>>> threads and data sizes.
>>>
>>>> My main issue is when the cluster splits for whatever reason, e.g. if
>>>>1
>>>> node is killed.
>>>
>>> What do you mean here? A killed node is removed from the view and there
>>> should not be a split.
>>>
>>>> If this happens, all the (remaining) nodes fall out of
>>>> the cluster and become isolated islands, never to merge / rejoin each
>>>> other.   A new node spun up also seems to remain isolated.   This is
>>>>not
>>>> ideal obviously!
>>>
>>> Yes, obviously!
>>>
>>> Killing a single member should NOT lead to a split cluster! That's
>>> problem #1 that we need to tackle
>>>
>>> To find out what's going on, I suggest enable tracing on
>>> - FD_SOCK/FD_ALL/FD: find out why nodes are suspected / excluded
>>> - MERGE3: this should merge split clusters back into a single cluster
>>>
>>> If you can debug this, set a breakpoint into MERGE3.InfoSender.run()
>>> [line 388] and see which responses you get.
>>>
>>> So here are my top suspects which may cause this:
>>>
>>> #1 AWS_PING
>>> - If tracing above shows that the modified AWS_PING doesn't return all
>>> members, the merge will never succeed. To confirm / reject this,
>>>replace
>>> AWS_PING with S3_PING
>>> - I assume if you run this system in a local network, not in AWS, a
>>> member leaving doesn't cause a split and if there was a split, it would
>>> heal again? This would also point to AWS/AWS_PING/TCP_NIO
>>>
>>> #2 TCP_NIO
>>> - This is very old and deprecated
>>> - I suggest switch to TCP (or TCP_NIO2 in a later JGroups version)
>>>
>>> #3 AWS security policy
>>> - Can you open all ports, just to see if this fixes things?
>>>
>>> #4 Your configuration
>>> - Thread pool sizing etc, I'll comment on this further down
>>>
>>>
>>>
>>> <TCP_NIO bind_port="7800"
>>>
>>> Replace this with TCP or TCP_NIO2
>>>
>>>
>>>
>>>> sock_conn_timeout="10000"
>>>
>>> Reduce to 300
>>> (In general I suggest use the stock tcp.xml file and modify it
>>>slightly)
>>>
>>>
>>>> internal_thread_pool.enabled="true"
>>>>
>>>> internal_thread_pool.min_threads="2"
>>>>
>>>> internal_thread_pool.max_threads="500"
>>>>
>>>> internal_thread_pool.keep_alive_time="15000"
>>>>
>>>> internal_thread_pool.queue_enabled="true"
>>>>
>>>> internal_thread_pool.queue_max_size="50000"
>>>>
>>>> internal_thread_pool.rejection_policy="discard"
>>>
>>>
>>> Set queue_enabled to false
>>>
>>>
>>>
>>>> thread_pool.enabled="true"
>>>>
>>>> thread_pool.min_threads="2"
>>>>
>>>> thread_pool.max_threads="500"
>>>>
>>>> thread_pool.keep_alive_time="15000"
>>>>
>>>> thread_pool.queue_enabled="true"
>>>>
>>>> thread_pool.queue_max_size="50000"
>>>>
>>>> thread_pool.rejection_policy="discard"
>>>
>>>
>>>
>>> Set queue_enabled to false
>>>
>>>
>>>
>>>> oob_thread_pool.enabled="true"
>>>>
>>>> oob_thread_pool.min_threads="1"
>>>>
>>>> oob_thread_pool.max_threads="500"
>>>>
>>>> oob_thread_pool.keep_alive_time="30000"
>>>>
>>>> oob_thread_pool.queue_enabled="true"
>>>>
>>>> oob_thread_pool.queue_max_size="10000"
>>>>
>>>> oob_thread_pool.rejection_policy="discard"/>
>>>
>>>
>>> Set queue_enabled to false
>>>
>>>
>>>> <com.meltmedia.jgroups.aws.AWS_PING
>>>
>>>
>>> Try S3_PING or TCPGOSSIP. The latter requires a GossipRouter, but if
>>> this works, then it's cler that AWS_PING has an issue
>>>
>>>
>>>
>>> 
>>><FD_SOCKbind_addr="GLOBAL"num_tries="3"get_cache_timeout="10000"sock_con
>>>n_
>>> timeout="10000"suspect_msg_interval="15000"/>
>>>
>>> Use the defaults, e.g. sock_conn_timeout="1000"
>>>
>>>
>>>
>>>> <pbcast.NAKACK2use_mcast_xmit="false"
>>>>
>>>> xmit_interval="500"
>>>
>>> 1000 would be sufficient
>>>
>>>
>>>
>>> 
>>><pbcast.GMSprint_local_addr="true"join_timeout="60000"leave_timeout="150
>>>00
>>> "merge_timeout="60000"resume_task_timeout="15000"
>>>
>>>
>>> join_timeout of 60s? This means that first member will take 60s to
>>> join!!! BAD! Set it to 3000.
>>>
>>>
>>>
>>>> In the logs I get various exceptions, including these after the
>>>> incident has happened:
>>>>
>>>> DEBUG [2016-01-27 04:18:47,750] unknown.jul.logger: Read operation on
>>>> socket failed
>>>> ! java.io.IOException: Peer closed socket
>>>> ! at
>>>> 
>>>>org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(Connection
>>>>Ta
>>>> bleNIO.java:855)
>>>> ! at
>>>> 
>>>>org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTa
>>>>bl
>>>> eNIO.java:808)
>>>> ! at
>>>> 
>>>>org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNI
>>>>O.
>>>> java:741)
>>>> ! at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>>> That may be normal, but - again - don't use TCP_NIO!
>>>
>>>
>>>
>>>> DEBUG [2016-01-27 04:18:43,591] unknown.jul.logger: Read operation on
>>>> socket failed
>>>> ! java.io.IOException: Connection reset by peer
>>>> ! at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>>>> ! at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>>>> ! at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>>>> ! at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>>>> ! at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
>>>> ! at
>>>> 
>>>>org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(Connection
>>>>Ta
>>>> bleNIO.java:852)
>>>> ! at
>>>> 
>>>>org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTa
>>>>bl
>>>> eNIO.java:808)
>>>> ! at
>>>> 
>>>>org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNI
>>>>O.
>>>> java:741)
>>>> ! at java.lang.Thread.run(Thread.java:745)
>>>
>>> Same thing
>>>
>>>
>>>
>>> --
>>> Bela Ban, JGroups lead (http://www.jgroups.org)
>>>
>>>
>>> 
>>>------------------------------------------------------------------------
>>>--
>>> ----
>>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>>> Monitor end-to-end web transactions and take corrective actions now
>>> Troubleshoot faster and improve end-user experience. Signup Now!
>>> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
>>> _______________________________________________
>>> javagroups-users mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>
>>
>> 
>>-------------------------------------------------------------------------
>>-----
>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>> Monitor end-to-end web transactions and take corrective actions now
>> Troubleshoot faster and improve end-user experience. Signup Now!
>> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
>> _______________________________________________
>> javagroups-users mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>
>
>-- 
>Bela Ban, JGroups lead (http://www.jgroups.org)
>
>
>--------------------------------------------------------------------------
>----
>Site24x7 APM Insight: Get Deep Visibility into Application Performance
>APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>Monitor end-to-end web transactions and take corrective actions now
>Troubleshoot faster and improve end-user experience. Signup Now!
>http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
>_______________________________________________
>javagroups-users mailing list
>[email protected]
>https://lists.sourceforge.net/lists/listinfo/javagroups-users

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users