Re: [jgroups-users] SEQUENCER on Docker+k8s

On 31/01/17 16:55, Questions/problems related to using JGroups wrote:
> Hi everyone,
>
> I'm currently trying to run a JGroups SEQUENCER using Docker and
> Kubernetes on AWS EC2. (1 master on a m3.medium and 5 minions on
> c4.2xlarge instances)
>
> My goal is to see how much can I scale JGroups SEQUENCER before it
> starts to fail. (Either not every messages are delivered by everyone or
> out of order or simply cluster failure)
>
> I'm using a custom config[1] making use of the TCPGOSSIP server as we
> want to use UNICAST.
>
> My code is really simple[2], every peer connect to the same cluster and
> at a given time they start sending messages every second for a given
> duration.
>
> When trying with that many peers (200) I get a lot of exceptions and
> JGroups starts to fail with these messages[3] (Not exhaustive but errors
> are always related to the native send method and a "Invalid Argument").

This may point to an incorrect configuration. Are you using IPv4 or 
IPv6? If you want to force IPv4, use -Djava.net.preferIPv4Stack=true to 
start all nodes and the GossipRouter.

Which version of JGroups is this?

> Is JGroups expected to fail with this many peers or are these errors
> strange and shows a problem somewhere in my configuration or somewhere else?
>
> P.S.: I tried messing with the sysctl parameters from [4] with no luck:

You should not need these as the cluster (AIUI) has 6 members. This 
means 25 TCP connections, which is nothing.

How do the clients connect? Does each client connect to a random cluster 
member? That would be another 200 connections, not an issue, either.

Note that sequencer based total order will always render the sequencer a 
bottleneck, as all messages go through the sequencer.

Re your config:

- Remove FLUSH: I don't assume you want virtual synchrony *and* total 
order? Note that FLUSH is quite old and hasn't really been maintained 
the last ~8 years. Also, the SEQUENCER/FLUSH combo has never been tested 
to my knowledge...

- You should define UDP.ucast_recv_buf_size / ucast_send_buf_size; the 
defaults are quite small; I suggest copy the UDP section from fast.xml 
shipped with JGroups. This also contains thread pool min and max sizes

- FD_ALL: the default timeouts are quite low which might result in false 
suspicions; comment this protocol for your perf runs or increase the 
timeouts

- BARRIER and STATE_TRANSFER can be removed unless you do state transfer

- UDP.max_bundle_size and FRAG2.frag_size may need tuning


> - sysctl net.ipv4.ip_local_port_range="15000 61000"
> - sysctl net.ipv4.tcp_fin_timeout=30
> - sysctl net.core.somaxconn=1024
> - ifconfig eth0 txqueuelen 5000
> - sysctl net.core.netdev_max_backlog=2000
> - sysctl net.ipv4.tcp_max_syn_backlog=2048
>
>
> Cheers
> JT
>
>
> [1]
> https://github.com/jocelynthode/EpTOTester/blob/master/projects/JGroupsTester/src/main/resources/sequencer-tcpgossip.xml
>
> [2]
> https://github.com/jocelynthode/EpTOTester/blob/master/projects/JGroupsTester/src/main/kotlin/EventTester.kt
>
> [3] https://gist.github.com/jocelynthode/b4f7d04e248c10672c10fe79210bda27
>
> [4] http://stackoverflow.com/a/3923785/2826574
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot