Re: [jgroups-users] SEQUENCER on Docker+k8s

Le 01. 02. 17 à 16:52, Questions/problems related to using JGroups a écrit :
> 
> 
> On 01/02/17 16:23, Questions/problems related to using JGroups wrote:
>>
>>
>> Le 01. 02. 17 à 16:05, Questions/problems related to using JGroups a écrit :
>>>
>>>
>>> On 31/01/17 16:55, Questions/problems related to using JGroups wrote:
>>>> Hi everyone,
>>>>
>>>> I'm currently trying to run a JGroups SEQUENCER using Docker and
>>>> Kubernetes on AWS EC2. (1 master on a m3.medium and 5 minions on
>>>> c4.2xlarge instances)
>>>>
>>>> My goal is to see how much can I scale JGroups SEQUENCER before it
>>>> starts to fail. (Either not every messages are delivered by everyone or
>>>> out of order or simply cluster failure)
>>>>
>>>> I'm using a custom config[1] making use of the TCPGOSSIP server as we
>>>> want to use UNICAST.
>>>>
>>>> My code is really simple[2], every peer connect to the same cluster and
>>>> at a given time they start sending messages every second for a given
>>>> duration.
>>>>
>>>> When trying with that many peers (200) I get a lot of exceptions and
>>>> JGroups starts to fail with these messages[3] (Not exhaustive but errors
>>>> are always related to the native send method and a "Invalid Argument").
>>>
>>> This may point to an incorrect configuration. Are you using IPv4 or
>>> IPv6? If you want to force IPv4, use -Djava.net.preferIPv4Stack=true to
>>> start all nodes and the GossipRouter.
>>>
>>> Which version of JGroups is this?
>>
>> Yes I am specifying -Djava.net.preferIPv4Stack=true. I'm using JGroups
>> 3.6.11. Sorry I didn't specify this earlier: with less peers, say 100,
>> It is working fine.
>>
>>>
>>>> Is JGroups expected to fail with this many peers or are these errors
>>>> strange and shows a problem somewhere in my configuration or somewhere else?
>>>>
>>>> P.S.: I tried messing with the sysctl parameters from [4] with no luck:
>>>
>>> You should not need these as the cluster (AIUI) has 6 members. This
>>> means 25 TCP connections, which is nothing.
>>>
>>> How do the clients connect? Does each client connect to a random cluster
>>> member? That would be another 200 connections, not an issue, either.
>>
>> They join the cluster with the channel.connect("EventCluster") function.
> 
> 
> So your cluster has 200 members? I don't think that's a good idea as the 
> clients will all have to send their multicast messages through the 
> sequencer to obtain total order.
> 
> A core cluster of 5-20 members and clients connecting to one of the core 
> servers (e.g. via STOMP) would be better.

We do specifically want to test having a large number of members all
exchanging through a choke point compared to a decentralized epidemic
protocol. The goal is to find out just how much JGroups in this
configuration can hold and see if the epidemic total order algorithm
does better at this size.

Considering this, how many members do you think the SEQUENCER could hold?

I agree that in a scenario where you would want to use the SEQUENCER in
production, decentralizing the task of distributing the order to the
peers is a good idea.

> 
> 
>>> Note that sequencer based total order will always render the sequencer a
>>> bottleneck, as all messages go through the sequencer.
>>
>> Yes we specifically want to test for a centralized deterministic total
>> ordering algorithm.
>>
>>>
>>> Re your config:
>>>
>>> - Remove FLUSH: I don't assume you want virtual synchrony *and* total
>>> order? Note that FLUSH is quite old and hasn't really been maintained
>>> the last ~8 years. Also, the SEQUENCER/FLUSH combo has never been tested
>>> to my knowledge...
>>
>> We do want the FLUSH/SEQUENCER combo. (I tested it with less than 200
>> peers and it was working fine)
> 
> OK, as you please sir! But you're on your own here, you've been 
> warned... :-)
> 
> 
>>> - You should define UDP.ucast_recv_buf_size / ucast_send_buf_size; the
>>> defaults are quite small; I suggest copy the UDP section from fast.xml
>>> shipped with JGroups. This also contains thread pool min and max sizes
>>>
>>> - FD_ALL: the default timeouts are quite low which might result in false
>>> suspicions; comment this protocol for your perf runs or increase the
>>> timeouts
>>>
>>> - BARRIER and STATE_TRANSFER can be removed unless you do state transfer
>>>
>>> - UDP.max_bundle_size and FRAG2.frag_size may need tuning
>>>
>> - I will look into the unicast buffer sizes.
>> - We used the FD_ALL settings available in the "udp-largecluster.xml"
>> - We do want the state transfer.
>> - I will look into it
>>
>>>
>>>> - sysctl net.ipv4.ip_local_port_range="15000 61000"
>>>> - sysctl net.ipv4.tcp_fin_timeout=30
>>>> - sysctl net.core.somaxconn=1024
>>>> - ifconfig eth0 txqueuelen 5000
>>>> - sysctl net.core.netdev_max_backlog=2000
>>>> - sysctl net.ipv4.tcp_max_syn_backlog=2048
>>>>
>>>>
>>>> Cheers
>>>> JT
>>>>
>>>>
>>>> [1]
>>>> https://github.com/jocelynthode/EpTOTester/blob/master/projects/JGroupsTester/src/main/resources/sequencer-tcpgossip.xml
>>>>
>>>> [2]
>>>> https://github.com/jocelynthode/EpTOTester/blob/master/projects/JGroupsTester/src/main/kotlin/EventTester.kt
>>>>
>>>> [3] https://gist.github.com/jocelynthode/b4f7d04e248c10672c10fe79210bda27
>>>>
>>>> [4] http://stackoverflow.com/a/3923785/2826574
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Check out the vibrant tech community on one of the world's most
>>>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>>> _______________________________________________
>>>> javagroups-users mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>>>
>>>
>>
>> Cheers
>> JT
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> javagroups-users mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>
> 

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot