Re: [jgroups-users] unexplained port usage

Comments inline, marked with [bela]

On 20/06/16 18:18, Questions/problems related to using JGroups wrote:
> Ok, am finally realizing how little I knew about the way the stack is
> put together. Hopefully simple questions below.
>
> On Tue, Jun 14, 2016 at 10:29 AM, Questions/problems related to using
> JGroups <[email protected]
> <mailto:[email protected]>> wrote:
>
>     >     the only other port opened should be by FD_SOCK. This can be set by
>     >     start_port.
>     >
>     >
>     > Thanks, I can watch FD_SOCK#run in the debugger and see where it's
>     > picking up the random port. For whatever reason, I never realized that
>     > FD_SOCK was using a different port than the bind port used in TCP -- I
>     > guess I thought the TCP socket was used for everything.
>
>     Nope - you can use FD_SOCK also with UDP as transport. And even if TCP
>     is used, FD_SOCK cannot rely on TCP closing a connection as idle
>     connections may get reaped and closed.
>
>
> I copied this originally from tcp.xml, and ended up with:
>
>          stack.addProtocol(new TCP()
>                  [values ommitted]
>              .addProtocol(new TCPPING()
>                  [values ommitted]
>              .addProtocol(new MERGE3()
>                  [values ommitted]
>              .addProtocol(new FD_SOCK())  [all default values used]
>              .addProtocol(new FD
>                  .setValue(<max tries and timeout here>)
>              .addProtocol(new VERIFY_SUSPECT()
>                  .setValue("timeout", <timeout>))
>              .addProtocol(new BARRIER())
>              [etc]
>
> I guess I thought that FD needed FD_SOCK below it, but now see that FD*
> and VERIFY_SUSPECT are all unrelated and just need TCP below them.

[bela]
Yes, they're completely unrelated. They don't require a specific 
transport, either, you could also use UDP here.

> Question 1: I let users specify values for FD, but not FD_SOCK. Does
> that mean that really long values (set by user) for FD might not matter
> if FD_SOCK discovers a failure first?

[bela]
Exactly. See [1] and [2] for details. [2] is somewhat old but explains 
why we have FD_SOCK and FD_ALL in the default config.

> Question 2: If users only open very few ports in their firewall, for
> instance bind_port in TCP and database ports, failover still happens
> anyway though it sounds like FD_SOCK can't work. Is it working because
> FD can still talk to other members using TCP's bind_port?

[bela]
This should actually fail as FD_SOCK won't be able to connect to its 
peer, so you should see a lot of suspicions. Perhaps they're cancelled 
by VERIFY_SUSPECT, but this still leads to unneeded traffic. I suggest 
enable TRACE on FD_SOCK to confirm this.

If you want to use FD_SOCK, you need to open a port in the firewall for it.

> Question 3: Moving forward, based on other unrelated discussions with
> you, I've replaced FD with FD_ALL.

[bela] Excellent, good choice!

> Can I just pull FD_SOCK out of there and things should work ok?

[bela]
Yes, but you won't have immediate view changes when a member crashes; 
now a new view is installed after FD_ALL's timeout kicked which may take 
time. This will block (1) blocking RPCs, (2) STABLE and (3) flow control.

> The current customer asking about the random
> ports would like to not have anything random, and it seems like I could
> just pull that. Another product we write here just uses this in the
> stack, and it's been working fine:
>
>      [TCP and TCP_GOSSIP below this]
>      <MERGE2 min_interval="10000" merge_fast="true" max_interval="30000"/>
>      <FD timeout="30000"/>
>      <FD_ALL timeout="30000"/>
>      <BARRIER/>
>      [other stuff]

[bela]
I suggest pull FD here; both FD protocols are redundant.

> ...though I don't know what the advantages are of multiple FD* protocols
> in there.

[bela]
There aren't any, only additional traffic.

> So, to get it all in one place, I'm proposing this stack
> (values removed, but it's the same as pasted in the initial mail in this
> thread with FD -> FD_ALL). Do you see any issues?
>
>          stack.addProtocol(new TCP()
>              .addProtocol(new TCPPING()
>              .addProtocol(new MERGE3()
>              .addProtocol(new FD_SOCK())
>              .addProtocol(new FD_ALL2()
>              .addProtocol(new VERIFY_SUSPECT()
>              .addProtocol(new BARRIER())
>              .addProtocol(new NAKACK2()
>              .addProtocol(new UNICAST3())
>              .addProtocol(new STABLE()
>              .addProtocol(createAuthProtocol(efmNode))
>              .addProtocol(new GMS()
>              .addProtocol(new MFC()
>              .addProtocol(new FRAG2())
>              .addProtocol(new STATE_TRANSFER());

[bela]
Looks ok. Note that FD_ALL2 has never seen a lot of testing versus 
FD_ALL. It's not a complex protocol though so you should be fine.

> Thanks again,
> Bobby
>
> p.s. I think you mentioned once we might not need STATE_TRANSFER, but I
> do use it so that new members get a copy of a state object whenever they
> join, and the coordinator can tell everyone to update their copy when
> needed.

OK


[1] http://www.jgroups.org/manual/index.html#FailureDetection
[2] https://developer.jboss.org/wiki/FDVersusFDSOCK

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)


------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape