Re: [jgroups-users] [E] Re: Can probe be used to recover network splits?

On 21/03/16 15:25, Questions/problems related to using JGroups wrote:
> I need to confirm once again.  We are planning to upgrade jboss but
> it is a long term plan. Can I use latest jgroups with jboss-4.2.0.GA

No

> to fix the issues that we are seeing currently?.   Thanks for all
> your suggestions.
>
> Thanks, Vamshi J.
>
> -----Original Message----- From: Questions/problems related to using
> JGroups [mailto:[email protected]] Sent:
> Saturday, March 19, 2016 5:02 AM To:
> [email protected] Subject: [E] Re:
> [jgroups-users] Can probe be used to recover network splits?
>
>
>
> On 18/03/16 18:42, Questions/problems related to using JGroups
> wrote:
>
>
>>> I'm sure there's something else going on, preventing a merge.
>>> Have you checked your thread pools? Do they have queues enabled
>>> perhaps? Do you configure the internal and/or timer pools as
>>> well?
>>>
>>> Thread pool queue is enabled  for Regular thread pools but
>>> disabled for OOB. I have internal and timer threads configured
>>> too.
>>
>>
>> [Bela] OK. An idea is to check with probe.sh (next time this
>> occurs) what the size of the pools is (plus the active sizes)
>>
>> /[PoojaK] : the internal_queue_size and time_queue_size is all 0
>
> [Bela] I meant pool size not queue size.
>
>
>> everywhere but the OOB_queue_size: most of the nodes reported
>> queue size to be around 1000 and oob.thread_pool.queue_max_size is
>> 1000.
>
> [Bela] The OOB queue size is ignored as the queue is disabled, but
> the regular queue size should be always more or less 0!
>
>> Does views() go through OOB channel?
>
> [Bela] No, a new view is sent as regular message. So, if your regular
> queue size is > 0, then that could mean views are delayed.
>
>> I think I should disable the  queue on OOB thread pool for sure. so
>> that messages reach right away.
>
> [Bela] I thought you're doing that already. But I'd agree that the
> regular pool's queue could be disabled, too.
>
>
>>> What's you config?
>>>
>>>
>>> Below is my config:
>>>
>>> <TCP thread_pool.enabled="true" thread_pool.min_threads="50"
>>> thread_pool.max_threads="100" thread_pool.keep_alive_time="5000"
>>> thread_pool.queue_enabled="true"
>>> thread_pool.queue_max_size="50000"
>>> thread_pool.rejection_policy="Discard"
>>
>> [Bela] So if you have more than 50 threads active at the same time,
>> new messages will be queued up to 50000. Is 50 the max number of
>> nodes that are sending? An idea might be to disable the queue and
>> up max_size to a large number. OTOH, you won't need more than
>> (max_nodes + B) threads where B is a small buffer (say 3) as
>> messages from the same sender S will simply get added to S's table
>> is a thread is already processing a message (or batch) from S.
>>
>> /[PoojaK]: 50 was kept as higher limit since this system was going
>> to increase to 50 but currently its 32 members. But when I was
>> checking this system again I noticed min_threads are only 10 in
>> current system where issue happen whereas we have 32 nodes. This
>> might be the main culprit :(. That must be manual error (and I will
>> get this corrected right away to 35+). /
>
>
> [Bela] Yes, unless you have 50'000 messages in the queue, you'll
> never have more than 10 threads processing messages from 32 nodes.
> So, in the worst case, messages (and views) get delayed a lot.
>
>
>> /Also, most nodes are running with rejection_policy as Run for OOB
>> and regular thread pools. This might be another culprit. I need to
>> get this changed to "Discard" or "Run"/
>
>
> [Bela] Yes, +1,  definitely don't use "run"!
>
>
>> /For the queues, do you suggest disabling OOB , regular , timer
>> and internal thread pool queues? or just regular and OOB?/
>
> [Bela] Just regular and OOB. The other thread pools have tasks that
> are never supposed to block.
>
>
>
>>> initial_hosts=""
>>
>> [Bela] I assume you're setting this value at startup time?
>> /[PoojaK]: Yes and we have been very accurate with this to not see
>> "No Physical Address found" :-)/
>>
>>> <MERGE3 max_interval="200000" max_participants_in_merge="200"
>>> min_interval="20000" />
>>
>> [Bela] So on average it will take 200000 * 1.6 (check_interval,
>> computed) to start a merge
>>
>> /[PoojaK]: I checked the check_interval is 320 seconds for us
>> right now.Do you think this is adding to delay in merging? /
>
> [Bela] Not necessarily, but if you only check every ~5 minutes, then
> you can have 12 merges per hour _max_. Since merge detection is done
> with unreliable messages (not retransmitted since they're below
> NAKACK2 / UNICAST3), a merge might not happen at all, or happen
> partially (not merging the entire cluster).
>
> I don't think this is the main culprit, but it doesn't help having
> to wait for 5+ minutes for merge detection.
>
>
>>> <FD_ALL timeout="600000" interval="60000" />
>>
>>
>> [Bela] 10 minutes to detect a hung or crashed member?
>>
>> /[PoojaK]: that was done keeping in mind the nature of Application.
>> This application is data intensive and needs to store and
>> re-transmit data if the outage is within 10 minutes. I do remember
>> from your workshop that this shouldnt be very high. I will keep
>> this point in mind for a change. /
>
>
> [Bela] Yes, but I also suggested to pair FD_ALL with FD_SOCK. The
> latter would catch 90% of all crashes.
>
>
>>> <pbcast.GMS
>>>
>>> print_local_addr="true"
>>>
>>> join_timeout="6000"
>>>
>>> view_bundling="true"
>>>
>>> merge_timeout="60000"
>>>
>>> view_ack_collection_timeout="10000" />
>>
>> [Bela] Not a good idea; it might take up to 10s to install a view
>> unless VIEW-ACKs from all members are received quickly
>>
>> /[PoojaK]: If I recollect this variable is not stopping anything.
>> Its asynchronous right? /
>
>
> [Bela] Nope. The view installer (coordinator) waits for 10s until it
> has acks from all members of the new view. So this will delay view
> installation (also merge views).
>
>
>>> <FRAG2 frag_size="60000" />
>>
>> [Bela] Oh the horror, why aren't you using FRAG2?
>>
>> /[PoojaK]: I think I am using FRAG2 only. did you mean something
>> else? /
>
>
> [Bela] Sorry, my bad, I thought I read FRAG... :-)
>
>
>>> <pbcast.STATE_TRANSFER/>
>>>
>>> </config>
>>>
>>>> This is a cluster of 30 Nodes running with TCP as transport.
>>>> The nodes are geographically co-located across the globe.
>>>>
>>>>
>>>> On Thu, Mar 17, 2016 at 4:27 AM, JGroups - General mailing
>> list [via
>>>> JGroups] <[hidden email]
>>> </user/SendEmail.jtp?type=node&node=10995&i=0>>
>>>> wrote:
>>>>
>>>> Hi Pooja,
>>>>
>>>> On 16/03/16 21:25, Questions/problems related to using
>>> JGroups wrote:
>>>>> Hi ,
>>>>>
>>>>> Say if there is a split in the cluster and it is not
>> recovering
>>>>> automatically inspite of no network issues, has any one
>>> tried any probe
>>>>> commands related to merge3 to heal the cluster
>> automatically?
>>>>
>>>> There's a @ManagedOperation MERGE3.sendInfo() that can be
>>> triggered, so
>>>> everyone in a cluster sends their information,
>> allowing the
>>> coordinator
>>>> to start a merge.
>>>>
>>>> However, this doesn't automatically merge the cluster,
>> e.g.
>>> if everyone
>>>> has the same view, then nothing will happen.
>>>>
>>>> If there is no network problem, and MERGE3 does *not*
>> recover the
>>>> cluster, then that would be a bug. In such a case,
>> what would be
>>>> required for diagnosis is: - TRACE logs for all members of GMS
>>>> and MERGE3 - Views of all members. This could be used for a
>> reproducer
>>>>
>>>> I haven't yet come across a scenario with MERGE3 (*not*
>>> MERGE2!) that
>>>> doesn't heal a network partition when the network is
>>> functioning ok
>>>> again.
>>>>
>>>>> Thanks Pooja
>>>>>
>>>>>
>>>>
>>>
>>
>> -- Bela Ban, JGroups lead (http://www.jgroups.org)
>>
>>
>>
>> javagroups-users mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>
>

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)


------------------------------------------------------------------------------
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140