Re: [jgroups-users] [E] Re: Can probe be used to recover network splits?

I need to confirm once again.  We are planning to upgrade jboss but it is a long term plan. Can I use latest jgroups with jboss-4.2.0.GA to fix the issues that we are seeing currently?.   Thanks for all your suggestions.

Thanks,
Vamshi J.

-----Original Message-----
From: Questions/problems related to using JGroups [mailto:[email protected]] 
Sent: Saturday, March 19, 2016 5:02 AM
To: [email protected]
Subject: [E] Re: [jgroups-users] Can probe be used to recover network splits?



On 18/03/16 18:42, Questions/problems related to using JGroups wrote:


>     >     I'm sure there's something else going on, preventing a merge. Have you
>     >     checked your thread pools? Do they have queues enabled perhaps? Do you
>     >     configure the internal and/or timer pools as well?
>     >
>     > Thread pool queue is enabled  for Regular thread pools but disabled for
>     > OOB.
>     > I have internal and timer threads configured too.
>
>
>     [Bela] OK. An idea is to check with probe.sh (next time this occurs)
>     what the size of the pools is (plus the active sizes)
>
>     /[PoojaK] : the internal_queue_size and time_queue_size is all 0

[Bela] I meant pool size not queue size.


>     everywhere but the OOB_queue_size: most of the nodes reported queue
>     size to be around 1000 and oob.thread_pool.queue_max_size is 1000.

[Bela] The OOB queue size is ignored as the queue is disabled, but the regular queue size should be always more or less 0!

>     Does views() go through OOB channel?

[Bela] No, a new view is sent as regular message. So, if your regular queue size is > 0, then that could mean views are delayed.

> I think I should disable the  queue on OOB thread pool for sure. so that messages reach right away.

[Bela] I thought you're doing that already. But I'd agree that the regular pool's queue could be disabled, too.


>     >     What's you config?
>     >
>     >
>     > Below is my config:
>     >
>      >         <TCP
>     >         thread_pool.enabled="true"
>     >         thread_pool.min_threads="50"
>     >         thread_pool.max_threads="100"
>     >         thread_pool.keep_alive_time="5000"
>     >         thread_pool.queue_enabled="true"
>     >         thread_pool.queue_max_size="50000"
>     >         thread_pool.rejection_policy="Discard"
>
>     [Bela]
>     So if you have more than 50 threads active at the same time, new
>     messages will be queued up to 50000. Is 50 the max number of nodes that
>     are sending?
>     An idea might be to disable the queue and up max_size to a large number.
>     OTOH, you won't need more than (max_nodes + B) threads where B is a
>     small buffer (say 3) as messages from the same sender S will simply get
>     added to S's table is a thread is already processing a message (or
>     batch) from S.
>
> /[PoojaK]: 50 was kept as higher limit since this system was going to
> increase to 50 but currently its 32 members. But when I was checking
> this system again I noticed min_threads are only 10 in current system
> where issue happen whereas we have 32 nodes. This might be the main
> culprit :(. That must be manual error (and I will get this corrected
> right away to 35+). /


[Bela] Yes, unless you have 50'000 messages in the queue, you'll never 
have more than 10 threads processing messages from 32 nodes. So, in the 
worst case, messages (and views) get delayed a lot.


> /Also, most nodes are running with rejection_policy as Run for OOB and
> regular thread pools. This might be another culprit. I need to get this
> changed to "Discard" or "Run"/


[Bela] Yes, +1,  definitely don't use "run"!


> /For the queues, do you suggest disabling OOB , regular , timer and
> internal thread pool queues? or just regular and OOB?/

[Bela] Just regular and OOB. The other thread pools have tasks that are 
never supposed to block.



>     >         initial_hosts=""
>
>     [Bela] I assume you're setting this value at startup time?
>     /[PoojaK]: Yes and we have been very accurate with this to not see
>     "No Physical Address found" :-)/
>
>     >         <MERGE3
>     >         max_interval="200000"
>     >         max_participants_in_merge="200"
>     >         min_interval="20000" />
>
>     [Bela] So on average it will take 200000 * 1.6 (check_interval,
>     computed) to start a merge
>
> /[PoojaK]: I checked the check_interval is 320 seconds for us right
> now.Do you think this is adding to delay in merging? /

[Bela] Not necessarily, but if you only check every ~5 minutes, then you 
can have 12 merges per hour _max_. Since merge detection is done with 
unreliable messages (not retransmitted since they're below NAKACK2 / 
UNICAST3), a merge might not happen at all, or happen partially (not 
merging the entire cluster).

I don't think this is the main culprit, but it doesn't help having to 
wait for 5+ minutes for merge detection.


>     >         <FD_ALL
>     >         timeout="600000"
>     >         interval="60000" />
>
>
>     [Bela] 10 minutes to detect a hung or crashed member?
>
> /[PoojaK]: that was done keeping in mind the nature of Application. This
> application is data intensive and needs to store and re-transmit data if
> the outage is within 10 minutes. I do remember from your workshop that
> this shouldnt be very high. I will keep this point in mind for a change. /


[Bela] Yes, but I also suggested to pair FD_ALL with FD_SOCK. The latter 
would catch 90% of all crashes.


>     >         <pbcast.GMS
>     >
>     >         print_local_addr="true"
>     >
>     >         join_timeout="6000"
>     >
>     >         view_bundling="true"
>     >
>     >         merge_timeout="60000"
>     >
>     >         view_ack_collection_timeout="10000" />
>
>     [Bela] Not a good idea; it might take up to 10s to install a view unless
>     VIEW-ACKs from all members are received quickly
>
> /[PoojaK]: If I recollect this variable is not stopping anything. Its
> asynchronous right? /


[Bela] Nope. The view installer (coordinator) waits for 10s until it has 
acks from all members of the new view. So this will delay view 
installation (also merge views).


>     >         <FRAG2
>     >         frag_size="60000" />
>
>     [Bela] Oh the horror, why aren't you using FRAG2?
>
> /[PoojaK]: I think I am using FRAG2 only. did you mean something else? /


[Bela] Sorry, my bad, I thought I read FRAG... :-)


>      >         <pbcast.STATE_TRANSFER/>
>      >
>      >         </config>
>      >
>      >     > This is a cluster of 30 Nodes running with TCP as transport.
>      >     > The nodes are geographically co-located across the globe.
>      >     >
>      >     >
>      >     > On Thu, Mar 17, 2016 at 4:27 AM, JGroups - General mailing
>     list [via
>      >      > JGroups] <[hidden email]
>      >     </user/SendEmail.jtp?type=node&node=10995&i=0>>
>      >      > wrote:
>      >      >
>      >      >     Hi Pooja,
>      >      >
>      >      >     On 16/03/16 21:25, Questions/problems related to using
>      >     JGroups wrote:
>      >      >     > Hi ,
>      >      >     >
>      >      >     > Say if there is a split in the cluster and it is not
>     recovering
>      >      >     > automatically inspite of no network issues, has any one
>      >     tried any probe
>      >      >     > commands related to merge3 to heal the cluster
>     automatically?
>      >      >
>      >      >     There's a @ManagedOperation MERGE3.sendInfo() that can be
>      >     triggered, so
>      >      >     everyone in a cluster sends their information,
>     allowing the
>      >     coordinator
>      >      >     to start a merge.
>      >      >
>      >      >     However, this doesn't automatically merge the cluster,
>     e.g.
>      >     if everyone
>      >      >     has the same view, then nothing will happen.
>      >      >
>      >      >     If there is no network problem, and MERGE3 does *not*
>     recover the
>      >      >     cluster, then that would be a bug. In such a case,
>     what would be
>      >      >     required for diagnosis is:
>      >      >     - TRACE logs for all members of GMS and MERGE3
>      >      >     - Views of all members. This could be used for a
>     reproducer
>      >      >
>      >      >     I haven't yet come across a scenario with MERGE3 (*not*
>      >     MERGE2!) that
>      >      >     doesn't heal a network partition when the network is
>      >     functioning ok
>      >      >     again.
>      >      >
>      >      >      > Thanks
>      >      >      > Pooja
>      >      >      >
>      >      >      >
>      >      >
>      >
>
>     --
>     Bela Ban, JGroups lead (http://www.jgroups.org)
>
>
>
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)


------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users

------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140