Re: [jgroups-users] Cluster configuration suggestions.

On 27/01/16 22:22, Questions/problems related to using JGroups wrote:
> Bela,
>
> Thanks for the reply.  It will take me some time to go through and test
> everything with the changes.
>
> I have had a lot of logging on in past tests (TRACE level) so I¹ll go back
> through that to see if it shows anything.
> I¹m 99% certain AWS_PING is returning the members correctly - the original
> members, the reduced member set when a node is killed, then the new member
> set when the new node is brought online.  I can connect to each node with
> jconsole and review the JMX data, so I could see the state of membership
> at any point.

Yes, I don't think the first step of this investigation should be to see 
why the merge isn't working, it should be to find out why a 
crashed/killed member leads to a split brain. This should definitely not 
be the case and I've never seen such behavior!


> My first pass at this will be:
>
> * upgrade to latest jgroups (3.6.7 it seems?)

EXCELLENT !!!! :-)

> * change configuration as suggested
> * re-run my current tests

OK

  > Depending on the outcome I¹ll proceed from there (probably to add the
> tests you mention).
>
> I note your statements about the uneven memory sizes, but at least in my
> case I have more than enough memory head room to avoid OOME.  Still this
> is something I should probably revisit in the future to improve the system.

Yes, I was just concerned about the (extremely unlikely) scenario where 
a crappy consistent hash algorithm would allocate all large values on 
the _same_ server.


> Thanks,
>
> Chris
>
>
> On 27/01/2016 20:30, "Questions/problems related to using JGroups"
> <[email protected]> wrote:
>
>> Hi Chris,
>>
>> comments inline
>>
>> On 27/01/16 05:28, Questions/problems related to using JGroups wrote:
>>> Hi.
>>>
>>> I¹m using jgroups 3.6.4 within Infinispan 8.1.0 (embedded in my
>>> application).  I¹ve got a 5 node cluster (running in AWS) and I¹m having
>>> some issues getting it performing correctly.  I¹ve been reading a lot of
>>> the documentation on jgroups, trying to understand the configuration
>>> options possible.  While I think I¹m getting a grasp of it, trying to
>>> get a good set of values for this configuration is not that easy,
>>> especially when I¹m unsure just what values I should use and why.
>>>
>>> In the happy case, the cluster can see each other ok and entries go into
>>> the cluster fine.   For discussion purposes, the number of entries in
>>> the cluster is in the 100 to 200K, with the data varying in size of a
>>> few K, to maybe 1M,
>>
>> As an aside and unrelated to this problem, adding a few very large
>> values can lead to an uneven use of memory across the grid: based on the
>> consistent hash of the key, a key/value pair is assigned a server (and a
>> backup server) to host it. If we have a cluster {A,B,C,D,E} and 3 very
>> large values x1,x2,x3, then with a bit of bad luck (and a bad CH impl),
>> x1,x2,x3 could also end up on B and C. possibly leading to an OOME on
>> B,C, while the other nodes are fine.
>>
>> I suggest chunking your data so all values have more or less the same
>> size. By more or less, I mean stay within a couple of orders of
>> magnitudes... :-)
>>
>> There's also a GridFilesystem, which provides an I/O abstraction to a
>> grid and internally uses chunking.
>>
>>> updating at around 250+ entries per sec across the
>>> cluster.  For production, the number of entries will be at least 10 to
>>> 100 times this figure and I¹d really like to increase the update rate as
>>> well.   I¹ve yet to really test concurrent read/writes from the system,
>>> but read performance with no updates was not a problem.
>>
>> OK. I suggest run a few perftests such as IspnPerfTest (Infinispan
>> related) or UPerf (JGroups) to get a feeling of what perf you're likely
>> to get on your N-node cluster, with varying number of reader/writer
>> threads and data sizes.
>>
>>> My main issue is when the cluster splits for whatever reason, e.g. if 1
>>> node is killed.
>>
>> What do you mean here? A killed node is removed from the view and there
>> should not be a split.
>>
>>> If this happens, all the (remaining) nodes fall out of
>>> the cluster and become isolated islands, never to merge / rejoin each
>>> other.   A new node spun up also seems to remain isolated.   This is not
>>> ideal obviously!
>>
>> Yes, obviously!
>>
>> Killing a single member should NOT lead to a split cluster! That's
>> problem #1 that we need to tackle
>>
>> To find out what's going on, I suggest enable tracing on
>> - FD_SOCK/FD_ALL/FD: find out why nodes are suspected / excluded
>> - MERGE3: this should merge split clusters back into a single cluster
>>
>> If you can debug this, set a breakpoint into MERGE3.InfoSender.run()
>> [line 388] and see which responses you get.
>>
>> So here are my top suspects which may cause this:
>>
>> #1 AWS_PING
>> - If tracing above shows that the modified AWS_PING doesn't return all
>> members, the merge will never succeed. To confirm / reject this, replace
>> AWS_PING with S3_PING
>> - I assume if you run this system in a local network, not in AWS, a
>> member leaving doesn't cause a split and if there was a split, it would
>> heal again? This would also point to AWS/AWS_PING/TCP_NIO
>>
>> #2 TCP_NIO
>> - This is very old and deprecated
>> - I suggest switch to TCP (or TCP_NIO2 in a later JGroups version)
>>
>> #3 AWS security policy
>> - Can you open all ports, just to see if this fixes things?
>>
>> #4 Your configuration
>> - Thread pool sizing etc, I'll comment on this further down
>>
>>
>>
>> <TCP_NIO bind_port="7800"
>>
>> Replace this with TCP or TCP_NIO2
>>
>>
>>
>>> sock_conn_timeout="10000"
>>
>> Reduce to 300
>> (In general I suggest use the stock tcp.xml file and modify it slightly)
>>
>>
>>> internal_thread_pool.enabled="true"
>>>
>>> internal_thread_pool.min_threads="2"
>>>
>>> internal_thread_pool.max_threads="500"
>>>
>>> internal_thread_pool.keep_alive_time="15000"
>>>
>>> internal_thread_pool.queue_enabled="true"
>>>
>>> internal_thread_pool.queue_max_size="50000"
>>>
>>> internal_thread_pool.rejection_policy="discard"
>>
>>
>> Set queue_enabled to false
>>
>>
>>
>>> thread_pool.enabled="true"
>>>
>>> thread_pool.min_threads="2"
>>>
>>> thread_pool.max_threads="500"
>>>
>>> thread_pool.keep_alive_time="15000"
>>>
>>> thread_pool.queue_enabled="true"
>>>
>>> thread_pool.queue_max_size="50000"
>>>
>>> thread_pool.rejection_policy="discard"
>>
>>
>>
>> Set queue_enabled to false
>>
>>
>>
>>> oob_thread_pool.enabled="true"
>>>
>>> oob_thread_pool.min_threads="1"
>>>
>>> oob_thread_pool.max_threads="500"
>>>
>>> oob_thread_pool.keep_alive_time="30000"
>>>
>>> oob_thread_pool.queue_enabled="true"
>>>
>>> oob_thread_pool.queue_max_size="10000"
>>>
>>> oob_thread_pool.rejection_policy="discard"/>
>>
>>
>> Set queue_enabled to false
>>
>>
>>> <com.meltmedia.jgroups.aws.AWS_PING
>>
>>
>> Try S3_PING or TCPGOSSIP. The latter requires a GossipRouter, but if
>> this works, then it's cler that AWS_PING has an issue
>>
>>
>>
>> <FD_SOCKbind_addr="GLOBAL"num_tries="3"get_cache_timeout="10000"sock_conn_
>> timeout="10000"suspect_msg_interval="15000"/>
>>
>> Use the defaults, e.g. sock_conn_timeout="1000"
>>
>>
>>
>>> <pbcast.NAKACK2use_mcast_xmit="false"
>>>
>>> xmit_interval="500"
>>
>> 1000 would be sufficient
>>
>>
>>
>> <pbcast.GMSprint_local_addr="true"join_timeout="60000"leave_timeout="15000
>> "merge_timeout="60000"resume_task_timeout="15000"
>>
>>
>> join_timeout of 60s? This means that first member will take 60s to
>> join!!! BAD! Set it to 3000.
>>
>>
>>
>>> In the logs I get various exceptions, including these after the
>>> incident has happened:
>>>
>>> DEBUG [2016-01-27 04:18:47,750] unknown.jul.logger: Read operation on
>>> socket failed
>>> ! java.io.IOException: Peer closed socket
>>> ! at
>>> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTa
>>> bleNIO.java:855)
>>> ! at
>>> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTabl
>>> eNIO.java:808)
>>> ! at
>>> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.
>>> java:741)
>>> ! at java.lang.Thread.run(Thread.java:745)
>>
>>
>> That may be normal, but - again - don't use TCP_NIO!
>>
>>
>>
>>> DEBUG [2016-01-27 04:18:43,591] unknown.jul.logger: Read operation on
>>> socket failed
>>> ! java.io.IOException: Connection reset by peer
>>> ! at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>>> ! at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>>> ! at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>>> ! at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>>> ! at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
>>> ! at
>>> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTa
>>> bleNIO.java:852)
>>> ! at
>>> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTabl
>>> eNIO.java:808)
>>> ! at
>>> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.
>>> java:741)
>>> ! at java.lang.Thread.run(Thread.java:745)
>>
>> Same thing
>>
>>
>>
>> --
>> Bela Ban, JGroups lead (http://www.jgroups.org)
>>
>>
>> --------------------------------------------------------------------------
>> ----
>> Site24x7 APM Insight: Get Deep Visibility into Application Performance
>> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
>> Monitor end-to-end web transactions and take corrective actions now
>> Troubleshoot faster and improve end-user experience. Signup Now!
>> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
>> _______________________________________________
>> javagroups-users mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>
>
> ------------------------------------------------------------------------------
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140