Re: [jgroups-users] Can probe be used to recover network splits?

> Looks like another discussion is also going on in the same thread.

Requesting to keep the thread different so each post can be helpful to
other readers too.

I got lost in my own thread for a while ;-).


>>
>>     I'm sure there's something else going on, preventing a merge. Have you
>>> >     >     checked your thread pools? Do they have queues enabled
>>> perhaps? Do you
>>> >     >     configure the internal and/or timer pools as well?
>>> >     >
>>> >     > Thread pool queue is enabled  for Regular thread pools but
>>> disabled for
>>> >     > OOB.
>>> >     > I have internal and timer threads configured too.
>>> >
>>> >
>>> [Bela] OK. An idea is to check with probe.sh (next time this occurs)
>>>      what the size of the pools is (plus the active sizes)
>>> [PoojaK] : the internal_queue_size and time_queue_size is all 0
>>>
>>> [Bela] I meant pool size not queue size.
>>>
>>> *[PoojaK]: Oh OK. Sorry for the confusion,but the configuration for
>> OOB_threadPool=true that *
>>
> *I gave you was overwritten to false on the system when the issue
>> happened. *
>>
> *So in this case, OOB thread pool and Regular both were enabled. *
>>
> *oob_thread_pool.queue_max_size is configured to be 1000.*
>>
> *oob_queue_size became 1000 for some nodes that were stuck and *
>>
> *their respective oob_pool_size were 100.*
>>
> *regular_pool_size was somewhere between 10 and 18.*
>>
> *So I think this is playing a major role here. *
>>
>
>>
>>>
>>> >     everywhere but the OOB_queue_size: most of the nodes reported queue
>>> >     size to be around 1000 and oob.thread_pool.queue_max_size is 1000.
>>>
>>> [Bela] The OOB queue size is ignored as the queue is disabled, but the
>>> regular queue size should be always more or less 0!
>>>
>>> >     Does views() go through OOB channel?
>>>
>>> [Bela] No, a new view is sent as regular message. So, if your regular
>>> queue size is > 0, then that could mean views are delayed.
>>>
>>> > I think I should disable the  queue on OOB thread pool for sure. so
>>> that messages reach right away.
>>>
>>> [Bela] I thought you're doing that already. But I'd agree that the
>>> regular pool's queue could be disabled, too.
>>>
>>>
>>> >     >     What's you config?
>>> >     >
>>> >     >
>>> >     > Below is my config:
>>> >     >
>>> >      >         <TCP
>>> >     >         thread_pool.enabled="true"
>>> >     >         thread_pool.min_threads="50"
>>> >     >         thread_pool.max_threads="100"
>>> >     >         thread_pool.keep_alive_time="5000"
>>> >     >         thread_pool.queue_enabled="true"
>>> >     >         thread_pool.queue_max_size="50000"
>>> >     >         thread_pool.rejection_policy="Discard"
>>> >
>>> >     [Bela]
>>> >     So if you have more than 50 threads active at the same time, new
>>> >     messages will be queued up to 50000. Is 50 the max number of nodes
>>> that
>>> >     are sending?
>>> >     An idea might be to disable the queue and up max_size to a large
>>> number.
>>> >     OTOH, you won't need more than (max_nodes + B) threads where B is a
>>> >     small buffer (say 3) as messages from the same sender S will
>>> simply get
>>> >     added to S's table is a thread is already processing a message (or
>>> >     batch) from S.
>>> >
>>> > /[PoojaK]: 50 was kept as higher limit since this system was going to
>>> > increase to 50 but currently its 32 members. But when I was checking
>>> > this system again I noticed min_threads are only 10 in current system
>>> > where issue happen whereas we have 32 nodes. This might be the main
>>> > culprit :(. That must be manual error (and I will get this corrected
>>> > right away to 35+). /
>>>
>>>
>>> [Bela] Yes, unless you have 50'000 messages in the queue, you'll never
>>> have more than 10 threads processing messages from 32 nodes. So, in the
>>> worst case, messages (and views) get delayed a lot.
>>>
>>>
>>> > /Also, most nodes are running with rejection_policy as Run for OOB and
>>> > regular thread pools. This might be another culprit. I need to get this
>>> > changed to "Discard" or "Run"/
>>>
>>>
>>> [Bela] Yes, +1,  definitely don't use "run"!
>>>
>>>
>>> > /For the queues, do you suggest disabling OOB , regular , timer and
>>> > internal thread pool queues? or just regular and OOB?/
>>>
>>> [Bela] Just regular and OOB. The other thread pools have tasks that are
>>> never supposed to block.
>>>
>>>
>>>
>>> >     >         initial_hosts=""
>>> >
>>> >     [Bela] I assume you're setting this value at startup time?
>>> >     /[PoojaK]: Yes and we have been very accurate with this to not see
>>> >     "No Physical Address found" :-)/
>>> >
>>> >     >         <MERGE3
>>> >     >         max_interval="200000"
>>> >     >         max_participants_in_merge="200"
>>> >     >         min_interval="20000" />
>>> >
>>> >     [Bela] So on average it will take 200000 * 1.6 (check_interval,
>>> >     computed) to start a merge
>>> >
>>> > /[PoojaK]: I checked the check_interval is 320 seconds for us right
>>> > now.Do you think this is adding to delay in merging? /
>>>
>>> [Bela] Not necessarily, but if you only check every ~5 minutes, then you
>>> can have 12 merges per hour _max_. Since merge detection is done with
>>> unreliable messages (not retransmitted since they're below NAKACK2 /
>>> UNICAST3), a merge might not happen at all, or happen partially (not
>>> merging the entire cluster).
>>>
>>> I don't think this is the main culprit, but it doesn't help having to
>>> wait for 5+ minutes for merge detection.
>>>
>>>
>>> >     >         <FD_ALL
>>> >     >         timeout="600000"
>>> >     >         interval="60000" />
>>> >
>>> >
>>> >     [Bela] 10 minutes to detect a hung or crashed member?
>>> >
>>> > /[PoojaK]: that was done keeping in mind the nature of Application.
>>> This
>>> > application is data intensive and needs to store and re-transmit data
>>> if
>>> > the outage is within 10 minutes. I do remember from your workshop that
>>> > this shouldnt be very high. I will keep this point in mind for a
>>> change. /
>>>
>>>
>>> [Bela] Yes, but I also suggested to pair FD_ALL with FD_SOCK. The latter
>>> would catch 90% of all crashes.
>>>
>>>
>>> >     >         <pbcast.GMS
>>> >     >
>>> >     >         print_local_addr="true"
>>> >     >
>>> >     >         join_timeout="6000"
>>> >     >
>>> >     >         view_bundling="true"
>>> >     >
>>> >     >         merge_timeout="60000"
>>> >     >
>>> >     >         view_ack_collection_timeout="10000" />
>>> >
>>> >     [Bela] Not a good idea; it might take up to 10s to install a view
>>> unless
>>> >     VIEW-ACKs from all members are received quickly
>>> >
>>> > /[PoojaK]: If I recollect this variable is not stopping anything. Its
>>> > asynchronous right? /
>>>
>>>
>>> [Bela] Nope. The view installer (coordinator) waits for 10s until it has
>>> acks from all members of the new view. So this will delay view
>>> installation (also merge views).
>>>
>>>
>>> >     >         <FRAG2
>>> >     >         frag_size="60000" />
>>> >
>>> >     [Bela] Oh the horror, why aren't you using FRAG2?
>>> >
>>> > /[PoojaK]: I think I am using FRAG2 only. did you mean something else?
>>> /
>>>
>>>
>>> [Bela] Sorry, my bad, I thought I read FRAG... :-)
>>>
>>>
>>> >      >         <pbcast.STATE_TRANSFER/>
>>> >      >
>>> >      >         </config>
>>> >      >
>>> >      >     > This is a cluster of 30 Nodes running with TCP as
>>> transport.
>>> >      >     > The nodes are geographically co-located across the globe.
>>> >      >     >
>>> >      >     >
>>> >      >     > On Thu, Mar 17, 2016 at 4:27 AM, JGroups - General mailing
>>> >     list [via
>>> >      >      > JGroups] <[hidden email]
>>> >      >     </user/SendEmail.jtp?type=node&node=10995&i=0>>
>>> >      >      > wrote:
>>> >      >      >
>>> >      >      >     Hi Pooja,
>>> >      >      >
>>> >      >      >     On 16/03/16 21:25, Questions/problems related to
>>> using
>>> >      >     JGroups wrote:
>>> >      >      >     > Hi ,
>>> >      >      >     >
>>> >      >      >     > Say if there is a split in the cluster and it is
>>> not
>>> >     recovering
>>> >      >      >     > automatically inspite of no network issues, has
>>> any one
>>> >      >     tried any probe
>>> >      >      >     > commands related to merge3 to heal the cluster
>>> >     automatically?
>>> >      >      >
>>> >      >      >     There's a @ManagedOperation MERGE3.sendInfo() that
>>> can be
>>> >      >     triggered, so
>>> >      >      >     everyone in a cluster sends their information,
>>> >     allowing the
>>> >      >     coordinator
>>> >      >      >     to start a merge.
>>> >      >      >
>>> >      >      >     However, this doesn't automatically merge the
>>> cluster,
>>> >     e.g.
>>> >      >     if everyone
>>> >      >      >     has the same view, then nothing will happen.
>>> >      >      >
>>> >      >      >     If there is no network problem, and MERGE3 does *not*
>>> >     recover the
>>> >      >      >     cluster, then that would be a bug. In such a case,
>>> >     what would be
>>> >      >      >     required for diagnosis is:
>>> >      >      >     - TRACE logs for all members of GMS and MERGE3
>>> >      >      >     - Views of all members. This could be used for a
>>> >     reproducer
>>> >      >      >
>>> >      >      >     I haven't yet come across a scenario with MERGE3
>>> (*not*
>>> >      >     MERGE2!) that
>>> >      >      >     doesn't heal a network partition when the network is
>>> >      >     functioning ok
>>> >      >      >     again.
>>> >      >      >
>>> >      >      >      > Thanks
>>> >      >      >      > Pooja
>>> >      >      >      >
>>> >      >      >      >
>>> >      >      >
>>> >      >
>>> >
>>> >     --
>>> >     Bela Ban, JGroups lead (http://www.jgroups.org)
>>> >
>>> >
>>> >
>>> > javagroups-users mailing list
>>> > [email protected]
>>> > https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>> >
>>>
>>> --
>>> Bela Ban, JGroups lead (http://www.jgroups.org)
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Transform Data into Opportunity.
>>> Accelerate data analysis in your applications with
>>> Intel Data Analytics Acceleration Library.
>>> Click to learn more.
>>> http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
>>> _______________________________________________
>>> javagroups-users mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>>
>>
>>
>

------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users