Re: [jgroups-users] SEQUENCER on Docker+k8s

On 01/02/17 16:23, Questions/problems related to using JGroups wrote:
>
>
> Le 01. 02. 17 à 16:05, Questions/problems related to using JGroups a écrit :
>>
>>
>> On 31/01/17 16:55, Questions/problems related to using JGroups wrote:
>>> Hi everyone,
>>>
>>> I'm currently trying to run a JGroups SEQUENCER using Docker and
>>> Kubernetes on AWS EC2. (1 master on a m3.medium and 5 minions on
>>> c4.2xlarge instances)
>>>
>>> My goal is to see how much can I scale JGroups SEQUENCER before it
>>> starts to fail. (Either not every messages are delivered by everyone or
>>> out of order or simply cluster failure)
>>>
>>> I'm using a custom config[1] making use of the TCPGOSSIP server as we
>>> want to use UNICAST.
>>>
>>> My code is really simple[2], every peer connect to the same cluster and
>>> at a given time they start sending messages every second for a given
>>> duration.
>>>
>>> When trying with that many peers (200) I get a lot of exceptions and
>>> JGroups starts to fail with these messages[3] (Not exhaustive but errors
>>> are always related to the native send method and a "Invalid Argument").
>>
>> This may point to an incorrect configuration. Are you using IPv4 or
>> IPv6? If you want to force IPv4, use -Djava.net.preferIPv4Stack=true to
>> start all nodes and the GossipRouter.
>>
>> Which version of JGroups is this?
>
> Yes I am specifying -Djava.net.preferIPv4Stack=true. I'm using JGroups
> 3.6.11. Sorry I didn't specify this earlier: with less peers, say 100,
> It is working fine.
>
>>
>>> Is JGroups expected to fail with this many peers or are these errors
>>> strange and shows a problem somewhere in my configuration or somewhere else?
>>>
>>> P.S.: I tried messing with the sysctl parameters from [4] with no luck:
>>
>> You should not need these as the cluster (AIUI) has 6 members. This
>> means 25 TCP connections, which is nothing.
>>
>> How do the clients connect? Does each client connect to a random cluster
>> member? That would be another 200 connections, not an issue, either.
>
> They join the cluster with the channel.connect("EventCluster") function.


So your cluster has 200 members? I don't think that's a good idea as the 
clients will all have to send their multicast messages through the 
sequencer to obtain total order.

A core cluster of 5-20 members and clients connecting to one of the core 
servers (e.g. via STOMP) would be better.


>> Note that sequencer based total order will always render the sequencer a
>> bottleneck, as all messages go through the sequencer.
>
> Yes we specifically want to test for a centralized deterministic total
> ordering algorithm.
>
>>
>> Re your config:
>>
>> - Remove FLUSH: I don't assume you want virtual synchrony *and* total
>> order? Note that FLUSH is quite old and hasn't really been maintained
>> the last ~8 years. Also, the SEQUENCER/FLUSH combo has never been tested
>> to my knowledge...
>
> We do want the FLUSH/SEQUENCER combo. (I tested it with less than 200
> peers and it was working fine)

OK, as you please sir! But you're on your own here, you've been 
warned... :-)


>> - You should define UDP.ucast_recv_buf_size / ucast_send_buf_size; the
>> defaults are quite small; I suggest copy the UDP section from fast.xml
>> shipped with JGroups. This also contains thread pool min and max sizes
>>
>> - FD_ALL: the default timeouts are quite low which might result in false
>> suspicions; comment this protocol for your perf runs or increase the
>> timeouts
>>
>> - BARRIER and STATE_TRANSFER can be removed unless you do state transfer
>>
>> - UDP.max_bundle_size and FRAG2.frag_size may need tuning
>>
> - I will look into the unicast buffer sizes.
> - We used the FD_ALL settings available in the "udp-largecluster.xml"
> - We do want the state transfer.
> - I will look into it
>
>>
>>> - sysctl net.ipv4.ip_local_port_range="15000 61000"
>>> - sysctl net.ipv4.tcp_fin_timeout=30
>>> - sysctl net.core.somaxconn=1024
>>> - ifconfig eth0 txqueuelen 5000
>>> - sysctl net.core.netdev_max_backlog=2000
>>> - sysctl net.ipv4.tcp_max_syn_backlog=2048
>>>
>>>
>>> Cheers
>>> JT
>>>
>>>
>>> [1]
>>> https://github.com/jocelynthode/EpTOTester/blob/master/projects/JGroupsTester/src/main/resources/sequencer-tcpgossip.xml
>>>
>>> [2]
>>> https://github.com/jocelynthode/EpTOTester/blob/master/projects/JGroupsTester/src/main/kotlin/EventTester.kt
>>>
>>> [3] https://gist.github.com/jocelynthode/b4f7d04e248c10672c10fe79210bda27
>>>
>>> [4] http://stackoverflow.com/a/3923785/2826574
>>>
>>> ------------------------------------------------------------------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> javagroups-users mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>>
>>
>
> Cheers
> JT
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot