Re: [jgroups-users] Cluster configuration suggestions.

Great to hear that!
Can you point to the change that made the diff, or did you implement
them all at once?

On 28/01/16 21:46, Questions/problems related to using JGroups wrote:
> I can report that so far, just those changes have done wonders.  It’s
> working as I expected.  If a node gets stopped/killed, the others remain
> cohesive and start rebalancing.  When a new node joins, it does the state
> transfer/rebalance as well.  I’ve got more tests to do, but I’m happier
> that it’s looking a lot better now.
> 
> So it appears my original bad config values plus the old TCP_NIO were the
> main culprits.
> 
> Thanks for your help.
> 
> Chris
> 
> On 29/01/2016 03:31, "Questions/problems related to using JGroups"
> <[email protected]> wrote:
> 
>>
>>
>> On 27/01/16 22:22, Questions/problems related to using JGroups wrote:
>>> Bela,
>>>
>>> Thanks for the reply.  It will take me some time to go through and test
>>> everything with the changes.
>>>
>>> I have had a lot of logging on in past tests (TRACE level) so I¹ll go
>>> back
>>> through that to see if it shows anything.
>>> I¹m 99% certain AWS_PING is returning the members correctly - the
>>> original
>>> members, the reduced member set when a node is killed, then the new
>>> member
>>> set when the new node is brought online.  I can connect to each node
>>> with
>>> jconsole and review the JMX data, so I could see the state of membership
>>> at any point.
>>
>> Yes, I don't think the first step of this investigation should be to see
>> why the merge isn't working, it should be to find out why a
>> crashed/killed member leads to a split brain. This should definitely not
>> be the case and I've never seen such behavior!
>>
>>
>>> My first pass at this will be:
>>>
>>> * upgrade to latest jgroups (3.6.7 it seems?)
>>
>> EXCELLENT !!!! :-)
>>
>>> * change configuration as suggested
>>> * re-run my current tests
>>
>> OK
>>
>>   > Depending on the outcome I¹ll proceed from there (probably to add the
>>> tests you mention).
>>>
>>> I note your statements about the uneven memory sizes, but at least in my
>>> case I have more than enough memory head room to avoid OOME.  Still this
>>> is something I should probably revisit in the future to improve the
>>> system.
>>
>> Yes, I was just concerned about the (extremely unlikely) scenario where
>> a crappy consistent hash algorithm would allocate all large values on
>> the _same_ server.
>>
>>
>>> Thanks,
>>>
>>> Chris
>>>
>>>
>>> On 27/01/2016 20:30, "Questions/problems related to using JGroups"
>>> <[email protected]> wrote:
>>>
>>>> Hi Chris,
>>>>
>>>> comments inline
>>>>
>>>> On 27/01/16 05:28, Questions/problems related to using JGroups wrote:
>>>>> Hi.
>>>>>
>>>>> I¹m using jgroups 3.6.4 within Infinispan 8.1.0 (embedded in my
>>>>> application).  I¹ve got a 5 node cluster (running in AWS) and I¹m
>>>>> having
>>>>> some issues getting it performing correctly.  I¹ve been reading a lot
>>>>> of
>>>>> the documentation on jgroups, trying to understand the configuration
>>>>> options possible.  While I think I¹m getting a grasp of it, trying to
>>>>> get a good set of values for this configuration is not that easy,
>>>>> especially when I¹m unsure just what values I should use and why.
>>>>>
>>>>> In the happy case, the cluster can see each other ok and entries go
>>>>> into
>>>>> the cluster fine.   For discussion purposes, the number of entries in
>>>>> the cluster is in the 100 to 200K, with the data varying in size of a
>>>>> few K, to maybe 1M,
>>>>
>>>> As an aside and unrelated to this problem, adding a few very large
>>>> values can lead to an uneven use of memory across the grid: based on
>>>> the
>>>> consistent hash of the key, a key/value pair is assigned a server (and
>>>> a
>>>> backup server) to host it. If we have a cluster {A,B,C,D,E} and 3 very
>>>> large values x1,x2,x3, then with a bit of bad luck (and a bad CH impl),
>>>> x1,x2,x3 could also end up on B and C. possibly leading to an OOME on
>>>> B,C, while the other nodes are fine.
>>>>
>>>> I suggest chunking your data so all values have more or less the same
>>>> size. By more or less, I mean stay within a couple of orders of
>>>> magnitudes... :-)
>>>>
>>>> There's also a GridFilesystem, which provides an I/O abstraction to a
>>>> grid and internally uses chunking.
>>>>
>>>>> updating at around 250+ entries per sec across the
>>>>> cluster.  For production, the number of entries will be at least 10 to
>>>>> 100 times this figure and I¹d really like to increase the update rate
>>>>> as
>>>>> well.   I¹ve yet to really test concurrent read/writes from the
>>>>> system,
>>>>> but read performance with no updates was not a problem.
>>>>
>>>> OK. I suggest run a few perftests such as IspnPerfTest (Infinispan
>>>> related) or UPerf (JGroups) to get a feeling of what perf you're likely
>>>> to get on your N-node cluster, with varying number of reader/writer
>>>> threads and data sizes.
>>>>
>>>>> My main issue is when the cluster splits for whatever reason, e.g. if
>>>>> 1
>>>>> node is killed.
>>>>
>>>> What do you mean here? A killed node is removed from the view and there
>>>> should not be a split.
>>>>
>>>>> If this happens, all the (remaining) nodes fall out of
>>>>> the cluster and become isolated islands, never to merge / rejoin each
>>>>> other.   A new node spun up also seems to remain isolated.   This is
>>>>> not
>>>>> ideal obviously!
>>>>
>>>> Yes, obviously!
>>>>
>>>> Killing a single member should NOT lead to a split cluster! That's
>>>> problem #1 that we need to tackle
>>>>
>>>> To find out what's going on, I suggest enable tracing on
>>>> - FD_SOCK/FD_ALL/FD: find out why nodes are suspected / excluded
>>>> - MERGE3: this should merge split clusters back into a single cluster
>>>>
>>>> If you can debug this, set a breakpoint into MERGE3.InfoSender.run()
>>>> [line 388] and see which responses you get.
>>>>
>>>> So here are my top suspects which may cause this:
>>>>
>>>> #1 AWS_PING
>>>> - If tracing above shows that the modified AWS_PING doesn't return all
>>>> members, the merge will never succeed. To confirm / reject this,
>>>> replace
>>>> AWS_PING with S3_PING
>>>> - I assume if you run this system in a local network, not in AWS, a
>>>> member leaving doesn't cause a split and if there was a split, it would
>>>> heal again? This would also point to AWS/AWS_PING/TCP_NIO
>>>>
>>>> #2 TCP_NIO
>>>> - This is very old and deprecated
>>>> - I suggest switch to TCP (or TCP_NIO2 in a later JGroups version)
>>>>
>>>> #3 AWS security policy
>>>> - Can you open all ports, just to see if this fixes things?
>>>>
>>>> #4 Your configuration
>>>> - Thread pool sizing etc, I'll comment on this further down
>>>>
>>>>
>>>>
>>>> <TCP_NIO bind_port="7800"
>>>>
>>>> Replace this with TCP or TCP_NIO2
>>>>
>>>>
>>>>
>>>>> sock_conn_timeout="10000"
>>>>
>>>> Reduce to 300
>>>> (In general I suggest use the stock tcp.xml file and modify it
>>>> slightly)
>>>>
>>>>
>>>>> internal_thread_pool.enabled="true"
>>>>>
>>>>> internal_thread_pool.min_threads="2"
>>>>>
>>>>> internal_thread_pool.max_threads="500"
>>>>>
>>>>> internal_thread_pool.keep_alive_time="15000"
>>>>>
>>>>> internal_thread_pool.queue_enabled="true"
>>>>>
>>>>> internal_thread_pool.queue_max_size="50000"
>>>>>
>>>>> internal_thread_pool.rejection_policy="discard"
>>>>
>>>>
>>>> Set queue_enabled to false
>>>>
>>>>
>>>>
>>>>> thread_pool.enabled="true"
>>>>>
>>>>> thread_pool.min_threads="2"
>>>>>
>>>>> thread_pool.max_threads="500"
>>>>>
>>>>> thread_pool.keep_alive_time="15000"
>>>>>
>>>>> thread_pool.queue_enabled="true"
>>>>>
>>>>> thread_pool.queue_max_size="50000"
>>>>>
>>>>> thread_pool.rejection_policy="discard"
>>>>
>>>>
>>>>
>>>> Set queue_enabled to false
>>>>
>>>>
>>>>
>>>>> oob_thread_pool.enabled="true"
>>>>>
>>>>> oob_thread_pool.min_threads="1"
>>>>>
>>>>> oob_thread_pool.max_threads="500"
>>>>>
>>>>> oob_thread_pool.keep_alive_time="30000"
>>>>>
>>>>> oob_thread_pool.queue_enabled="true"
>>>>>
>>>>> oob_thread_pool.queue_max_size="10000"
>>>>>
>>>>> oob_thread_pool.rejection_policy="discard"/>
>>>>
>>>>
>>>> Set queue_enabled to false
>>>>
>>>>
>>>>> <com.meltmedia.jgroups.aws.AWS_PING
>>>>
>>>>
>>>> Try S3_PING or TCPGOSSIP. The latter requires a GossipRouter, but if
>>>> this works, then it's cler that AWS_PING has an issue
>>>>
>>>>
>>>>
>>>>
>>>> <FD_SOCKbind_addr="GLOBAL"num_tries="3"get_cache_timeout="10000"sock_con
>>>> n_
>>>> timeout="10000"suspect_msg_interval="15000"/>
>>>>
>>>> Use the defaults, e.g. sock_conn_timeout="1000"
>>>>
>>>>
>>>>
>>>>> <pbcast.NAKACK2use_mcast_xmit="false"
>>>>>
>>>>> xmit_interval="500"
>>>>
>>>> 1000 would be sufficient
>>>>
>>>>
>>>>
>>>>
>>>> <pbcast.GMSprint_local_addr="true"join_timeout="60000"leave_timeout="150
>>>> 00
>>>> "merge_timeout="60000"resume_task_timeout="15000"
>>>>
>>>>
>>>> join_timeout of 60s? This means that first member will take 60s to
>>>> join!!! BAD! Set it to 3000.
>>>>
>>>>
>>>>
>>>>> In the logs I get various exceptions, including these after the
>>>>> incident has happened:
>>>>>
>>>>> DEBUG [2016-01-27 04:18:47,750] unknown.jul.logger: Read operation on
>>>>> socket failed
>>>>> ! java.io.IOException: Peer closed socket
>>>>> ! at
>>>>>
>>>>> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(Connection
>>>>> Ta
>>>>> bleNIO.java:855)
>>>>> ! at
>>>>>
>>>>> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTa
>>>>> bl
>>>>> eNIO.java:808)
>>>>> ! at
>>>>>
>>>>> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNI
>>>>> O.
>>>>> java:741)
>>>>> ! at java.lang.Thread.run(Thread.java:745)
>>>>
>>>>
>>>> That may be normal, but - again - don't use TCP_NIO!
>>>>
>>>>
>>>>
>>>>> DEBUG [2016-01-27 04:18:43,591] unknown.jul.logger: Read operation on
>>>>> socket failed
>>>>> ! java.io.IOException: Connection reset by peer
>>>>> ! at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>>>>> ! at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>>>>> ! at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>>>>> ! at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>>>>> ! at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
>>>>> ! at
>>>>>
>>>>> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(Connection
>>>>> Ta
>>>>> bleNIO.java:852)
>>>>> ! at
>>>>>
>>>>> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTa
>>>>> bl
>>>>> eNIO.java:808)
>>>>> ! at
>>>>>
>>>>> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNI
>>>>> O.
>>>>> java:741)
>>>>> ! at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> Same thing
>>>>
>>>>
>>>>
>>>> --
>>>> Bela Ban, JGroups lead (http://www.jgroups.org)
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>> --
>>>> ----
>>>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>>>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>>>> Monitor end-to-end web transactions and take corrective actions now
>>>> Troubleshoot faster and improve end-user experience. Signup Now!
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
>>>> _______________________________________________
>>>> javagroups-users mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>>
>>>
>>>
>>> -------------------------------------------------------------------------
>>> -----
>>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>>> Monitor end-to-end web transactions and take corrective actions now
>>> Troubleshoot faster and improve end-user experience. Signup Now!
>>> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
>>> _______________________________________________
>>> javagroups-users mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>>
>>
>> -- 
>> Bela Ban, JGroups lead (http://www.jgroups.org)
>>
>>
>> --------------------------------------------------------------------------
>> ----
>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>> Monitor end-to-end web transactions and take corrective actions now
>> Troubleshoot faster and improve end-user experience. Signup Now!
>> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
>> _______________________________________________
>> javagroups-users mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
> 
> ------------------------------------------------------------------------------
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
> 

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users