Re: [jgroups-users] SEQUENCER on Docker+k8s

Le 01. 02. 17 à 16:05, Questions/problems related to using JGroups a écrit :
> 
> 
> On 31/01/17 16:55, Questions/problems related to using JGroups wrote:
>> Hi everyone,
>>
>> I'm currently trying to run a JGroups SEQUENCER using Docker and
>> Kubernetes on AWS EC2. (1 master on a m3.medium and 5 minions on
>> c4.2xlarge instances)
>>
>> My goal is to see how much can I scale JGroups SEQUENCER before it
>> starts to fail. (Either not every messages are delivered by everyone or
>> out of order or simply cluster failure)
>>
>> I'm using a custom config[1] making use of the TCPGOSSIP server as we
>> want to use UNICAST.
>>
>> My code is really simple[2], every peer connect to the same cluster and
>> at a given time they start sending messages every second for a given
>> duration.
>>
>> When trying with that many peers (200) I get a lot of exceptions and
>> JGroups starts to fail with these messages[3] (Not exhaustive but errors
>> are always related to the native send method and a "Invalid Argument").
> 
> This may point to an incorrect configuration. Are you using IPv4 or 
> IPv6? If you want to force IPv4, use -Djava.net.preferIPv4Stack=true to 
> start all nodes and the GossipRouter.
> 
> Which version of JGroups is this?

Yes I am specifying -Djava.net.preferIPv4Stack=true. I'm using JGroups
3.6.11. Sorry I didn't specify this earlier: with less peers, say 100,
It is working fine.

> 
>> Is JGroups expected to fail with this many peers or are these errors
>> strange and shows a problem somewhere in my configuration or somewhere else?
>>
>> P.S.: I tried messing with the sysctl parameters from [4] with no luck:
> 
> You should not need these as the cluster (AIUI) has 6 members. This 
> means 25 TCP connections, which is nothing.
> 
> How do the clients connect? Does each client connect to a random cluster 
> member? That would be another 200 connections, not an issue, either.

They join the cluster with the channel.connect("EventCluster") function.

> 
> Note that sequencer based total order will always render the sequencer a 
> bottleneck, as all messages go through the sequencer.

Yes we specifically want to test for a centralized deterministic total
ordering algorithm.

> 
> Re your config:
> 
> - Remove FLUSH: I don't assume you want virtual synchrony *and* total 
> order? Note that FLUSH is quite old and hasn't really been maintained 
> the last ~8 years. Also, the SEQUENCER/FLUSH combo has never been tested 
> to my knowledge...

We do want the FLUSH/SEQUENCER combo. (I tested it with less than 200
peers and it was working fine)

> 
> - You should define UDP.ucast_recv_buf_size / ucast_send_buf_size; the 
> defaults are quite small; I suggest copy the UDP section from fast.xml 
> shipped with JGroups. This also contains thread pool min and max sizes
> 
> - FD_ALL: the default timeouts are quite low which might result in false 
> suspicions; comment this protocol for your perf runs or increase the 
> timeouts
> 
> - BARRIER and STATE_TRANSFER can be removed unless you do state transfer
> 
> - UDP.max_bundle_size and FRAG2.frag_size may need tuning
> 
- I will look into the unicast buffer sizes.
- We used the FD_ALL settings available in the "udp-largecluster.xml"
- We do want the state transfer.
- I will look into it

> 
>> - sysctl net.ipv4.ip_local_port_range="15000 61000"
>> - sysctl net.ipv4.tcp_fin_timeout=30
>> - sysctl net.core.somaxconn=1024
>> - ifconfig eth0 txqueuelen 5000
>> - sysctl net.core.netdev_max_backlog=2000
>> - sysctl net.ipv4.tcp_max_syn_backlog=2048
>>
>>
>> Cheers
>> JT
>>
>>
>> [1]
>> https://github.com/jocelynthode/EpTOTester/blob/master/projects/JGroupsTester/src/main/resources/sequencer-tcpgossip.xml
>>
>> [2]
>> https://github.com/jocelynthode/EpTOTester/blob/master/projects/JGroupsTester/src/main/kotlin/EventTester.kt
>>
>> [3] https://gist.github.com/jocelynthode/b4f7d04e248c10672c10fe79210bda27
>>
>> [4] http://stackoverflow.com/a/3923785/2826574
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> javagroups-users mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>
> 

Cheers
JT

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot