Re: [jgroups-users] callRemoteMethodsWithFuture blocks on coordinator crash
Questions/problems related to using JGroups <[email protected]> Wed, 10 Jan 2018 16:27:30 +0100
| Newsgroups | gmane.comp.java.javagroups.general |
|---|---|
| Message-ID | <mailman.21624.1515598067.13211.javagroups-users@lists.sourceforge.net> |
Hi Kishore,
I guess the biggest API change is that Message will become an interface
with a bunch of implementations, and users can register their own
message impls.
This is described in [1] (read the comments starting Nov 3).
I'll look into other features that don't change the API, such as
multiple transports, in subsequent releases, ie. 5.1, 5.2 etc.
I have no release date for 5.0. Currently all work is done on branch
JGRP-2218.msg, if you want to check it out. When I'm happy with the
changes, I'll merge the branch onto master and create a 4.x branch.
Master will then be 5.0.
5.0 should get some community baking time, and when people are happy
with the changes, I'll release it. I also want to work with the
Infinispan folks to see if ObjectMessage fits their bill.
[1] https://issues.jboss.org/browse/JGRP-2218
On 10/01/18 14:54, kishore kumar wrote:
> Hi Bela,
> Just curious to know what are the big changes to go in 5.0 and when the
> major release is expected.
>
> Regards,
> Kishore
>
> On Friday, December 1, 2017 at 1:47:20 PM UTC+5:30, Bela Ban wrote:
>
>
>
> On 29/11/17 17:04, kishore kumar wrote:
> > Hi Bela,
> > Glad to hear from you after a break. You must have been busy with
> the
> > JGroups workshop. I will follow the link provided for a better
> > understanding of the issues in the current SEQUENCER implementation,
> > also that can help make me appreciate SEQUENCER3 more :)
>
> Note that the SEQUENCER3 JIRA is just a collection of ideas, because
> I'm
> not overly happy with either SEQUENCER or SEQUENCER2, especially the
> logic for coordinator crash/failover.
>
> And it is not exactly high priority; I'll deal with it after the big
> changes that'll go into in 5.0.
>
>
> > Thanks & Regards,
> > Kishore
> >
> > On Wednesday, November 29, 2017 at 3:56:04 PM UTC+5:30, Bela Ban
> wrote:
> >
> > Hi Kishore,
> >
> > sorry for the long delay! Comments inline
> >
> > On 17/11/17 13:18, kishore kumar wrote:
> > > Okay I am trying to understand the behavior by running a few
> > scenarios.
> > > When I set the threshold to any value above 0, ack_mode is ON
> > only until
> > > the specified threshold is elapsed. Setting this to 1
> never blocks
> > > because it immediately sets the ack_mode to false
> (shouldn't this
> > wait
> > > for an ack from the coordinator before toggling the
> ack_mode?)
> >
> > OK, so threshold=0 means ack_mode is always on.
> >
> > A threshold of 1 means that we're sending the message and _do
> block_
> > but
> > when the ack has been received, we unblock and ack_mode is
> disabled
> > (until the next view).
> >
> > In forwardToCoord(), we forward the message to the
> coordinator, then
> > block. When the message is broadcast by the coordinator, we
> receive it,
> > too and deliver() is called, which unblocks the sender and
> disables
> > ack_mode.
> >
> > So a threshold of 1 blocks for only 1 ack. This is correct, IMO.
> >
> >
> > > if(ack_mode && !flushing && threshold > 0 &&
> ++num_acks >=
> > > threshold) {
> > > ack_mode=false;
> > > num_acks=0;
> > > }
> > > |
> > >
> > > The >= sign will become true without even receiving an ACK
> from the
> > > coordinator. Is this intentional? Or should that be
> num_acks++
> > instead
> > > of ++num_acks? That is how I understood from your
> description - "so
> > > ack_mode is set to false after a given number
> > > of acks, and that's governed by 'threshold'".
> >
> > Correct. This blocks for just 1 ack.
> >
> > > Irrespective of what threshold value is set, the blocking
> > (forwarding to
> > > coordinator) happens only when the ack_mode is true. When
> this
> > becomes
> > > false I don't experience 40 second delay (default FD_ALL
> timeout)
> > during
> > > network cable unplugging.
> >
> > Right, because we're not waiting for acks when ack_mode is
> false.
> >
> > > However upon a view change, the ack_mode is
> > > again set to true. This remains until the specified threshold
> > number of
> > > ACKs are successfully received. The reason I never
> experience 40
> > second
> > > delay with network cable unplugging is because of the
> threshold
> > value of
> > > "1" which is equivalent to ack_mode=false, always. Setting a
> > threshold
> > > of 1 can avoid this delay without modifying the FD_ALL
> > parameters. Is my
> > > understanding correct?
> > >
> > > If ack_mode is on, then a message P2 sent by P will block
> > until P1 sent
> > > by P has been acked by the sequencer (coordinator) when
> > forwarding. In
> > > other words, forwarding all messages of a given member
> P is
> > sequential.
> > >
> > > Does this mean there is no sequential guarantee when the
> ack_mode is
> > > false? If total ordering is always required should the
> threshold
> > be 0
> > > (ack_mode is always ON).
> >
> > Yes, the change of a coord can lead to violations in the
> sequential
> > guarantees. That's what threshold is trying to fix: ack_mode
> always on
> > slows things down, as every message involves a round trip to
> the coord.
> > Disabling it is fast, but can lead to ordering violations
> during coord
> > change. Setting it to a non-0 value reduces the changes of
> misorderings
> > to occur during view changes.
> >
> > To be honest, I've never been completely happy with
> SEQUENCER, that's
> > why SEQUENCER2 came into play. However, even that protocol
> has issues,
> > so I've compiled thoughts about a better impl (SEQUENCER3) in
> [1].
> >
> > [1] https://issues.jboss.org/browse/JGRP-1830
> <https://issues.jboss.org/browse/JGRP-1830>
> > <https://issues.jboss.org/browse/JGRP-1830
> <https://issues.jboss.org/browse/JGRP-1830>>
> >
> > > - Kishore
> > >
> > >
> > > On Friday, November 17, 2017 at 1:26:37 PM UTC+5:30, Bela
> Ban wrote:
> > >
> > > If ack_mode is on, then a message P2 sent by P will block
> > until P1 sent
> > > by P has been acked by the sequencer (coordinator) when
> > forwarding. In
> > > other words, forwarding all messages of a given member
> P is
> > sequential.
> > >
> > > This is of course slow, so ack_mode is set to false
> after a
> > given
> > > number
> > > of acks, and that's governed by 'threshold'. Also, when
> > there's a view
> > > change, ack_mode is enabled.
> > >
> > > When ack_mode is on, a message M forwarded to the
> coordinator
> > blocks
> > > until an ack has been received by the coordinator. So the
> > sender will
> > > block until this is the case.
> > >
> > > If the coordinator crashed, the sender will block
> until old
> > coordinator
> > > has been removed from the view and a new coordinator has
> > taken over.
> > > How
> > > long this takes depends on your configuration, ie.
> timeouts
> > in FD_ALL.
> > > If you have FD_SOCK in your stack and the coordinator
> > crashed, then
> > > this
> > > should be a few secs max, depending on timeouts in
> > VERIFY_SUSPECT.
> > >
> > > On 15/11/17 12:29, kishore kumar wrote:
> > > >
> > > >
> > > > That's right: the sender needs to block until
> it knows
> > who
> > > the new
> > > > coordinator is.
> > > > If a coordinator leaves gracefully, the switch
> to the new
> > > coord should
> > > > be immediate. If it crashes, then FD_SOCK
> should also
> > detect
> > > this
> > > > quickly and switch to the new coord. Only if
> FD_ALL
> > kicks in
> > > (kernel
> > > > panic, network cable pulled etc) would you run
> into
> > the full
> > > timeout.
> > > >
> > > > What's you config? I'm mainly interested in
> FD_SOCK,
> > FD_ALL and
> > > > VERIFY_SUSPECT settings, but do post the full
> config.
> > > >
> > > >
> > > > I have noticed a strange behavior to this case.
> Setting the
> > > threshold to
> > > > 1, i.e. <SEQUENCER threshold="1" />, does not
> block. Any
> > value other
> > > > than 1 (default is 0 from the documentation and
> hence I
> > have not
> > > tried
> > > > to manually set the value to 0) say 2, 3, 10 etc.
> blocks
> > until a
> > > view
> > > > change. What causes this? Is this the expected
> behavior or is
> > > something
> > > > wrong with setting threshold to 1? Can you provide
> more
> > details
> > > on what
> > > > 'threshold' means with respect to SEQUENCER?
> > > >
> > > > Regards,
> > > > Kishore
> > > >
> > > > --
> > > > You received this message because you are
> subscribed to
> > the Google
> > > > Groups "jgroups-dev" group.
> > > > To unsubscribe from this group and stop receiving
> emails
> > from it,
> > > send
> > > > an email to [email protected]
> <javascript:>
> > > > <mailto:[email protected]
> <javascript:>
> > <javascript:> <javascript:>>.
> > > > To post to this group, send email to
> > [email protected]
> > > <javascript:>
> > > > <mailto:[email protected] <javascript:>>.
> > > > To view this discussion on the web visit
> > > >
> > >
> >
> https://groups.google.com/d/msgid/jgroups-dev/5a935320-96eb-4f8e-a0ac-8ef1144de33b%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/5a935320-96eb-4f8e-a0ac-8ef1144de33b%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/jgroups-dev/5a935320-96eb-4f8e-a0ac-8ef1144de33b%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/5a935320-96eb-4f8e-a0ac-8ef1144de33b%40googlegroups.com>>
>
> >
> > >
> >
> <https://groups.google.com/d/msgid/jgroups-dev/5a935320-96eb-4f8e-a0ac-8ef1144de33b%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/5a935320-96eb-4f8e-a0ac-8ef1144de33b%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/jgroups-dev/5a935320-96eb-4f8e-a0ac-8ef1144de33b%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/5a935320-96eb-4f8e-a0ac-8ef1144de33b%40googlegroups.com>>>
>
> >
> > >
> > > >
> > >
> >
> <https://groups.google.com/d/msgid/jgroups-dev/5a935320-96eb-4f8e-a0ac-8ef1144de33b%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/jgroups-dev/5a935320-96eb-4f8e-a0ac-8ef1144de33b%40googlegroups.com?utm_medium=email&utm_source=footer>
>
> >
> <https://groups.google.com/d/msgid/jgroups-dev/5a935320-96eb-4f8e-a0ac-8ef1144de33b%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/jgroups-dev/5a935320-96eb-4f8e-a0ac-8ef1144de33b%40googlegroups.com?utm_medium=email&utm_source=footer>>
>
> >
> > >
> >
> <https://groups.google.com/d/msgid/jgroups-dev/5a935320-96eb-4f8e-a0ac-8ef1144de33b%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/jgroups-dev/5a935320-96eb-4f8e-a0ac-8ef1144de33b%40googlegroups.com?utm_medium=email&utm_source=footer>
>
> >
> <https://groups.google.com/d/msgid/jgroups-dev/5a935320-96eb-4f8e-a0ac-8ef1144de33b%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/jgroups-dev/5a935320-96eb-4f8e-a0ac-8ef1144de33b%40googlegroups.com?utm_medium=email&utm_source=footer>>>>.
>
> >
> > >
> > > > For more options, visit
> https://groups.google.com/d/optout <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>
> > > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>>.
> > >
> > > --
> > > Bela Ban | http://www.jgroups.org
> > >
> > > --
> > > You received this message because you are subscribed to
> the Google
> > > Groups "jgroups-dev" group.
> > > To unsubscribe from this group and stop receiving emails
> from it,
> > send
> > > an email to [email protected] <javascript:>
> > > <mailto:[email protected]
> <javascript:> <javascript:>>.
> > > To post to this group, send email to
> [email protected]
> > <javascript:>
> > > <mailto:[email protected] <javascript:>>.
> > > To view this discussion on the web visit
> > >
> >
> https://groups.google.com/d/msgid/jgroups-dev/7d7a07ca-544e-4ea4-b623-4012f6581a3d%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/7d7a07ca-544e-4ea4-b623-4012f6581a3d%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/jgroups-dev/7d7a07ca-544e-4ea4-b623-4012f6581a3d%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/7d7a07ca-544e-4ea4-b623-4012f6581a3d%40googlegroups.com>>
>
> >
> > >
> >
> <https://groups.google.com/d/msgid/jgroups-dev/7d7a07ca-544e-4ea4-b623-4012f6581a3d%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/jgroups-dev/7d7a07ca-544e-4ea4-b623-4012f6581a3d%40googlegroups.com?utm_medium=email&utm_source=footer>
>
> >
> <https://groups.google.com/d/msgid/jgroups-dev/7d7a07ca-544e-4ea4-b623-4012f6581a3d%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/jgroups-dev/7d7a07ca-544e-4ea4-b623-4012f6581a3d%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
>
> >
> > > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>
> > <https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>>.
> >
> > --
> > Bela Ban | http://www.jgroups.org
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "jgroups-dev" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to [email protected] <javascript:>
> > <mailto:[email protected] <javascript:>>.
> > To post to this group, send email to [email protected]
> <javascript:>
> > <mailto:[email protected] <javascript:>>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/jgroups-dev/0506e7ac-2d8a-474a-b38a-716b5160d523%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/0506e7ac-2d8a-474a-b38a-716b5160d523%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/jgroups-dev/0506e7ac-2d8a-474a-b38a-716b5160d523%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/jgroups-dev/0506e7ac-2d8a-474a-b38a-716b5160d523%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> Bela Ban | http://www.jgroups.org
>
> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [email protected]
> <mailto:[email protected]>.
> To post to this group, send email to [email protected]
> <mailto:[email protected]>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jgroups-dev/9ad69507-aa5c-4ffc-bbb6-48ea5c94639b%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/9ad69507-aa5c-4ffc-bbb6-48ea5c94639b%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.
--
Bela Ban | http://www.jgroups.org
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users