Re: [jgroups-dev] Problem with FD_SOCK failure detection

On 8/21/13 5:43 PM, mayank.agarwal wrote:
>
>
> From: "Bela Ban [via JGroups]" <[hidden email]
> </user/SendEmail.jtp?type=node&node=9853&i=0>>
> Date: Wednesday 21 August 2013 8:43 PM
> To: Mayank Agarwal <[hidden email]
> </user/SendEmail.jtp?type=node&node=9853&i=1>>
> Subject: Re: Problem with FD_SOCK failure detection
>
> Works for me: I used udp.xml, commented FD_ALL and increased the timeout
> in VERIFY_SUSPECT to 5000.
>
> The I started members A and B. Then I killed A (will get removed ca. 5 s
> later) and immediately started C. C looped around the failed JOIN until
> A was removed, then B joined and the new view was {B,C}.
>
> Mayank: The exact case is first start C and than kill A immediately.
> This happens intermittentlty (If you repeat the process many times)


C should join immediately. Usually, you have no chance of killing A 
before C has joined, as this should take only a few milliseconds.

Note that a port_range of 5 is very high for TCPPING, I suggest setting 
a low sock_conn_timeout in TCP, e.g. 300 (ms). You have 4 nodes listed 
in TCPPING.initial_hosts, which means you're doing 4 * 6 = 24 TCP 
connections, out of which 20 (when all 4 nodes are up) will fail...


> Comments inine.
>
> On 8/21/13 12:29 PM, mayank.agarwal wrote:
>> yes, this is reproducible.
>> FD_SOCK start() method is not called as JChannel.connect call wasn't
>> succeeded because it was trying to JOIN killed member(co-ordinator)
>
> This should not be the case as the killed coordinator will be removed
> from the view, so a discovery should not return it anymore.
> Do you use return_entire_cache="true" by any chance ?
>
> Mayank: return_entire_cache is by default false and we are not modifying
> this in xml
>
>> there are three nodes in cluster. (member-89,member-101,member-88,member-87)
>> we are getting lot of warnings mentioned below. it is trying to contact
>> member-89(which is killed)
>> [pool-3-thread-1] WARN  o.jgroups.protocols.pbcast.GMS -
>> JOIN(member-87-22015) sent to member-89-55050 timed out (after 7000 ms), on
>> try 2852
>
> It attempted to join 2852 times ?
> Note that you can set GMS.max_join_attempts to put a ceiling to the
> number of tries.
>
> Mayank: By default GMS.max_join_attempts is 0 which means 'never give
> up'. Can you suggest optimal value for this?

It's really up to you, when you want to form a singleton cluster, only 
to later be merged with the other sub-cluster(s).

However, I'd like to take a look at the root cause for you issue before 
going down this path.


>> On discovery request from member-87: member-88 is returning view
>> [member-89-55050|10].
>
> This shouldn't be the case, as member-89 should have been removed from
> the view !
> What's the config you use ?
> Mayank: config is attached
>
>> So it is always sending JOIN request to member-89(killed).
>>
>> member-88,89,101 has view {member-89,member-101,member-88,member-87}
>> member-87 does't have any view.
>> Scenario is: let say 2  A,B,C is in cluster new node D try to join and at
>> the same time A goes down. than C has already closed the socket with A
>> gracefully, assuming D will create the socket with A . D's FD_SOCK pinger
>> thread is not started. ( i am not seeing the msg "pinger_thread started" in
>> logs)
>
>
> Yes, that would be the case if D always gets discovery response(s) which
> still indicate A is the coordinator. This is the core issue, and we need
> to find out why anyone's still returning A in its discovery response.
>
>
>> We are not using combo of FD_SOCK with FD/FD_ALL. we are using only in
>> FD_SOCK.
>
> Why ? The recommended config includes both FD_SOCK *and* either FD (TCP)
> or FD_ALL (UDP).
>
> Mayank: we were using FD some months ago. But there are many false
> suspicions because of GC so we removed it.


I recommend to leave it in, but increase the timeout.


> If you can come up with exact steps to reproduce this, or a small
> program, than that would be good. Then I could create a JIRA.
>
> Mayank: I haven't created a program to reproduce this. We are repeating
> this cases by deploying our app to system and killing member in this manner.
> I will try to reproduce this through a small program


That would be good, and probably the only way I can take a stab at this 
(if I can run this locally).


-- 
Bela Ban, JGroups lead (http://www.jgroups.org)

------------------------------------------------------------------------------
Introducing Performance Central, a new site from SourceForge and 
AppDynamics. Performance Central is your source for news, insights, 
analysis and resources for efficient Application Performance Management. 
Visit us today!
http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk
_______________________________________________
Javagroups-development mailing list