Re: [jgroups-users] Cluster configuration suggestions.

At the time I felt that I was so far away from it working, I applied all
of them at once. :)  I think it was a combination effect of the timeout
changes (greatest effect), plus the TCP_NIO2 that did it.

On 29/01/2016 17:57, "Questions/problems related to using JGroups"
<[email protected]> wrote:

>Great to hear that!
>Can you point to the change that made the diff, or did you implement
>them all at once?
>
>On 28/01/16 21:46, Questions/problems related to using JGroups wrote:
>> I can report that so far, just those changes have done wonders.  It’s
>> working as I expected.  If a node gets stopped/killed, the others remain
>> cohesive and start rebalancing.  When a new node joins, it does the
>>state
>> transfer/rebalance as well.  I’ve got more tests to do, but I’m happier
>> that it’s looking a lot better now.
>> 
>> So it appears my original bad config values plus the old TCP_NIO were
>>the
>> main culprits.
>> 
>> Thanks for your help.
>> 
>> Chris
>> 
>> On 29/01/2016 03:31, "Questions/problems related to using JGroups"
>> <[email protected]> wrote:
>> 
>>>
>>>
>>> On 27/01/16 22:22, Questions/problems related to using JGroups wrote:
>>>> Bela,
>>>>
>>>> Thanks for the reply.  It will take me some time to go through and
>>>>test
>>>> everything with the changes.
>>>>
>>>> I have had a lot of logging on in past tests (TRACE level) so I¹ll go
>>>> back
>>>> through that to see if it shows anything.
>>>> I¹m 99% certain AWS_PING is returning the members correctly - the
>>>> original
>>>> members, the reduced member set when a node is killed, then the new
>>>> member
>>>> set when the new node is brought online.  I can connect to each node
>>>> with
>>>> jconsole and review the JMX data, so I could see the state of
>>>>membership
>>>> at any point.
>>>
>>> Yes, I don't think the first step of this investigation should be to
>>>see
>>> why the merge isn't working, it should be to find out why a
>>> crashed/killed member leads to a split brain. This should definitely
>>>not
>>> be the case and I've never seen such behavior!
>>>
>>>
>>>> My first pass at this will be:
>>>>
>>>> * upgrade to latest jgroups (3.6.7 it seems?)
>>>
>>> EXCELLENT !!!! :-)
>>>
>>>> * change configuration as suggested
>>>> * re-run my current tests
>>>
>>> OK
>>>
>>>   > Depending on the outcome I¹ll proceed from there (probably to add
>>>the
>>>> tests you mention).
>>>>
>>>> I note your statements about the uneven memory sizes, but at least in
>>>>my
>>>> case I have more than enough memory head room to avoid OOME.  Still
>>>>this
>>>> is something I should probably revisit in the future to improve the
>>>> system.
>>>
>>> Yes, I was just concerned about the (extremely unlikely) scenario where
>>> a crappy consistent hash algorithm would allocate all large values on
>>> the _same_ server.
>>>
>>>
>>>> Thanks,
>>>>
>>>> Chris
>>>>
>>>>
>>>> On 27/01/2016 20:30, "Questions/problems related to using JGroups"
>>>> <[email protected]> wrote:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>> comments inline
>>>>>
>>>>> On 27/01/16 05:28, Questions/problems related to using JGroups wrote:
>>>>>> Hi.
>>>>>>
>>>>>> I¹m using jgroups 3.6.4 within Infinispan 8.1.0 (embedded in my
>>>>>> application).  I¹ve got a 5 node cluster (running in AWS) and I¹m
>>>>>> having
>>>>>> some issues getting it performing correctly.  I¹ve been reading a
>>>>>>lot
>>>>>> of
>>>>>> the documentation on jgroups, trying to understand the configuration
>>>>>> options possible.  While I think I¹m getting a grasp of it, trying
>>>>>>to
>>>>>> get a good set of values for this configuration is not that easy,
>>>>>> especially when I¹m unsure just what values I should use and why.
>>>>>>
>>>>>> In the happy case, the cluster can see each other ok and entries go
>>>>>> into
>>>>>> the cluster fine.   For discussion purposes, the number of entries
>>>>>>in
>>>>>> the cluster is in the 100 to 200K, with the data varying in size of
>>>>>>a
>>>>>> few K, to maybe 1M,
>>>>>
>>>>> As an aside and unrelated to this problem, adding a few very large
>>>>> values can lead to an uneven use of memory across the grid: based on
>>>>> the
>>>>> consistent hash of the key, a key/value pair is assigned a server
>>>>>(and
>>>>> a
>>>>> backup server) to host it. If we have a cluster {A,B,C,D,E} and 3
>>>>>very
>>>>> large values x1,x2,x3, then with a bit of bad luck (and a bad CH
>>>>>impl),
>>>>> x1,x2,x3 could also end up on B and C. possibly leading to an OOME on
>>>>> B,C, while the other nodes are fine.
>>>>>
>>>>> I suggest chunking your data so all values have more or less the same
>>>>> size. By more or less, I mean stay within a couple of orders of
>>>>> magnitudes... :-)
>>>>>
>>>>> There's also a GridFilesystem, which provides an I/O abstraction to a
>>>>> grid and internally uses chunking.
>>>>>
>>>>>> updating at around 250+ entries per sec across the
>>>>>> cluster.  For production, the number of entries will be at least 10
>>>>>>to
>>>>>> 100 times this figure and I¹d really like to increase the update
>>>>>>rate
>>>>>> as
>>>>>> well.   I¹ve yet to really test concurrent read/writes from the
>>>>>> system,
>>>>>> but read performance with no updates was not a problem.
>>>>>
>>>>> OK. I suggest run a few perftests such as IspnPerfTest (Infinispan
>>>>> related) or UPerf (JGroups) to get a feeling of what perf you're
>>>>>likely
>>>>> to get on your N-node cluster, with varying number of reader/writer
>>>>> threads and data sizes.
>>>>>
>>>>>> My main issue is when the cluster splits for whatever reason, e.g.
>>>>>>if
>>>>>> 1
>>>>>> node is killed.
>>>>>
>>>>> What do you mean here? A killed node is removed from the view and
>>>>>there
>>>>> should not be a split.
>>>>>
>>>>>> If this happens, all the (remaining) nodes fall out of
>>>>>> the cluster and become isolated islands, never to merge / rejoin
>>>>>>each
>>>>>> other.   A new node spun up also seems to remain isolated.   This is
>>>>>> not
>>>>>> ideal obviously!
>>>>>
>>>>> Yes, obviously!
>>>>>
>>>>> Killing a single member should NOT lead to a split cluster! That's
>>>>> problem #1 that we need to tackle
>>>>>
>>>>> To find out what's going on, I suggest enable tracing on
>>>>> - FD_SOCK/FD_ALL/FD: find out why nodes are suspected / excluded
>>>>> - MERGE3: this should merge split clusters back into a single cluster
>>>>>
>>>>> If you can debug this, set a breakpoint into MERGE3.InfoSender.run()
>>>>> [line 388] and see which responses you get.
>>>>>
>>>>> So here are my top suspects which may cause this:
>>>>>
>>>>> #1 AWS_PING
>>>>> - If tracing above shows that the modified AWS_PING doesn't return
>>>>>all
>>>>> members, the merge will never succeed. To confirm / reject this,
>>>>> replace
>>>>> AWS_PING with S3_PING
>>>>> - I assume if you run this system in a local network, not in AWS, a
>>>>> member leaving doesn't cause a split and if there was a split, it
>>>>>would
>>>>> heal again? This would also point to AWS/AWS_PING/TCP_NIO
>>>>>
>>>>> #2 TCP_NIO
>>>>> - This is very old and deprecated
>>>>> - I suggest switch to TCP (or TCP_NIO2 in a later JGroups version)
>>>>>
>>>>> #3 AWS security policy
>>>>> - Can you open all ports, just to see if this fixes things?
>>>>>
>>>>> #4 Your configuration
>>>>> - Thread pool sizing etc, I'll comment on this further down
>>>>>
>>>>>
>>>>>
>>>>> <TCP_NIO bind_port="7800"
>>>>>
>>>>> Replace this with TCP or TCP_NIO2
>>>>>
>>>>>
>>>>>
>>>>>> sock_conn_timeout="10000"
>>>>>
>>>>> Reduce to 300
>>>>> (In general I suggest use the stock tcp.xml file and modify it
>>>>> slightly)
>>>>>
>>>>>
>>>>>> internal_thread_pool.enabled="true"
>>>>>>
>>>>>> internal_thread_pool.min_threads="2"
>>>>>>
>>>>>> internal_thread_pool.max_threads="500"
>>>>>>
>>>>>> internal_thread_pool.keep_alive_time="15000"
>>>>>>
>>>>>> internal_thread_pool.queue_enabled="true"
>>>>>>
>>>>>> internal_thread_pool.queue_max_size="50000"
>>>>>>
>>>>>> internal_thread_pool.rejection_policy="discard"
>>>>>
>>>>>
>>>>> Set queue_enabled to false
>>>>>
>>>>>
>>>>>
>>>>>> thread_pool.enabled="true"
>>>>>>
>>>>>> thread_pool.min_threads="2"
>>>>>>
>>>>>> thread_pool.max_threads="500"
>>>>>>
>>>>>> thread_pool.keep_alive_time="15000"
>>>>>>
>>>>>> thread_pool.queue_enabled="true"
>>>>>>
>>>>>> thread_pool.queue_max_size="50000"
>>>>>>
>>>>>> thread_pool.rejection_policy="discard"
>>>>>
>>>>>
>>>>>
>>>>> Set queue_enabled to false
>>>>>
>>>>>
>>>>>
>>>>>> oob_thread_pool.enabled="true"
>>>>>>
>>>>>> oob_thread_pool.min_threads="1"
>>>>>>
>>>>>> oob_thread_pool.max_threads="500"
>>>>>>
>>>>>> oob_thread_pool.keep_alive_time="30000"
>>>>>>
>>>>>> oob_thread_pool.queue_enabled="true"
>>>>>>
>>>>>> oob_thread_pool.queue_max_size="10000"
>>>>>>
>>>>>> oob_thread_pool.rejection_policy="discard"/>
>>>>>
>>>>>
>>>>> Set queue_enabled to false
>>>>>
>>>>>
>>>>>> <com.meltmedia.jgroups.aws.AWS_PING
>>>>>
>>>>>
>>>>> Try S3_PING or TCPGOSSIP. The latter requires a GossipRouter, but if
>>>>> this works, then it's cler that AWS_PING has an issue
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 
>>>>><FD_SOCKbind_addr="GLOBAL"num_tries="3"get_cache_timeout="10000"sock_c
>>>>>on
>>>>> n_
>>>>> timeout="10000"suspect_msg_interval="15000"/>
>>>>>
>>>>> Use the defaults, e.g. sock_conn_timeout="1000"
>>>>>
>>>>>
>>>>>
>>>>>> <pbcast.NAKACK2use_mcast_xmit="false"
>>>>>>
>>>>>> xmit_interval="500"
>>>>>
>>>>> 1000 would be sufficient
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 
>>>>><pbcast.GMSprint_local_addr="true"join_timeout="60000"leave_timeout="1
>>>>>50
>>>>> 00
>>>>> "merge_timeout="60000"resume_task_timeout="15000"
>>>>>
>>>>>
>>>>> join_timeout of 60s? This means that first member will take 60s to
>>>>> join!!! BAD! Set it to 3000.
>>>>>
>>>>>
>>>>>
>>>>>> In the logs I get various exceptions, including these after the
>>>>>> incident has happened:
>>>>>>
>>>>>> DEBUG [2016-01-27 04:18:47,750] unknown.jul.logger: Read operation
>>>>>>on
>>>>>> socket failed
>>>>>> ! java.io.IOException: Peer closed socket
>>>>>> ! at
>>>>>>
>>>>>> 
>>>>>>org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(Connecti
>>>>>>on
>>>>>> Ta
>>>>>> bleNIO.java:855)
>>>>>> ! at
>>>>>>
>>>>>> 
>>>>>>org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(Connection
>>>>>>Ta
>>>>>> bl
>>>>>> eNIO.java:808)
>>>>>> ! at
>>>>>>
>>>>>> 
>>>>>>org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTable
>>>>>>NI
>>>>>> O.
>>>>>> java:741)
>>>>>> ! at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>>
>>>>> That may be normal, but - again - don't use TCP_NIO!
>>>>>
>>>>>
>>>>>
>>>>>> DEBUG [2016-01-27 04:18:43,591] unknown.jul.logger: Read operation
>>>>>>on
>>>>>> socket failed
>>>>>> ! java.io.IOException: Connection reset by peer
>>>>>> ! at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>>>>>> ! at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>>>>>> ! at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>>>>>> ! at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>>>>>> ! at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
>>>>>> ! at
>>>>>>
>>>>>> 
>>>>>>org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(Connecti
>>>>>>on
>>>>>> Ta
>>>>>> bleNIO.java:852)
>>>>>> ! at
>>>>>>
>>>>>> 
>>>>>>org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(Connection
>>>>>>Ta
>>>>>> bl
>>>>>> eNIO.java:808)
>>>>>> ! at
>>>>>>
>>>>>> 
>>>>>>org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTable
>>>>>>NI
>>>>>> O.
>>>>>> java:741)
>>>>>> ! at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> Same thing
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Bela Ban, JGroups lead (http://www.jgroups.org)
>>>>>
>>>>>
>>>>>
>>>>> 
>>>>>----------------------------------------------------------------------
>>>>>--
>>>>> --
>>>>> ----
>>>>> Site24x7 APM Insight: Get Deep Visibility into Application
>>>>>Performance
>>>>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>>>>> Monitor end-to-end web transactions and take corrective actions now
>>>>> Troubleshoot faster and improve end-user experience. Signup Now!
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
>>>>> _______________________________________________
>>>>> javagroups-users mailing list
>>>>> [email protected]
>>>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>>>
>>>>
>>>>
>>>> 
>>>>-----------------------------------------------------------------------
>>>>--
>>>> -----
>>>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>>>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>>>> Monitor end-to-end web transactions and take corrective actions now
>>>> Troubleshoot faster and improve end-user experience. Signup Now!
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
>>>> _______________________________________________
>>>> javagroups-users mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>>>
>>>
>>> -- 
>>> Bela Ban, JGroups lead (http://www.jgroups.org)
>>>
>>>
>>> 
>>>------------------------------------------------------------------------
>>>--
>>> ----
>>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>>> Monitor end-to-end web transactions and take corrective actions now
>>> Troubleshoot faster and improve end-user experience. Signup Now!
>>> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
>>> _______________________________________________
>>> javagroups-users mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>> 
>> 
>>-------------------------------------------------------------------------
>>-----
>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>> Monitor end-to-end web transactions and take corrective actions now
>> Troubleshoot faster and improve end-user experience. Signup Now!
>> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
>> _______________________________________________
>> javagroups-users mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>> 
>
>-- 
>Bela Ban, JGroups lead (http://www.jgroups.org)
>
>
>--------------------------------------------------------------------------
>----
>Site24x7 APM Insight: Get Deep Visibility into Application Performance
>APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>Monitor end-to-end web transactions and take corrective actions now
>Troubleshoot faster and improve end-user experience. Signup Now!
>http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
>_______________________________________________
>javagroups-users mailing list
>[email protected]
>https://lists.sourceforge.net/lists/listinfo/javagroups-users

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users