Re: [jgroups-users] SEQUENCER on Docker+k8s

Thanks for your time and all the answers!

Cheers
JT

Le 03. 02. 17 à 16:41, Questions/problems related to using JGroups a écrit :
> 
> 
> On 01/02/17 17:04, Questions/problems related to using JGroups wrote:
>>
>>
>> Le 01. 02. 17 à 16:52, Questions/problems related to using JGroups a écrit :
>>>
>>>
>>> On 01/02/17 16:23, Questions/problems related to using JGroups wrote:
>>>>
>>>>
>>>> Le 01. 02. 17 à 16:05, Questions/problems related to using JGroups a écrit :
>>>>>
>>>>>
>>>>> On 31/01/17 16:55, Questions/problems related to using JGroups wrote:
>>>>>> Hi everyone,
>>>>>>
>>>>>> I'm currently trying to run a JGroups SEQUENCER using Docker and
>>>>>> Kubernetes on AWS EC2. (1 master on a m3.medium and 5 minions on
>>>>>> c4.2xlarge instances)
>>>>>>
>>>>>> My goal is to see how much can I scale JGroups SEQUENCER before it
>>>>>> starts to fail. (Either not every messages are delivered by everyone or
>>>>>> out of order or simply cluster failure)
>>>>>>
>>>>>> I'm using a custom config[1] making use of the TCPGOSSIP server as we
>>>>>> want to use UNICAST.
>>>>>>
>>>>>> My code is really simple[2], every peer connect to the same cluster and
>>>>>> at a given time they start sending messages every second for a given
>>>>>> duration.
>>>>>>
>>>>>> When trying with that many peers (200) I get a lot of exceptions and
>>>>>> JGroups starts to fail with these messages[3] (Not exhaustive but errors
>>>>>> are always related to the native send method and a "Invalid Argument").
>>>>>
>>>>> This may point to an incorrect configuration. Are you using IPv4 or
>>>>> IPv6? If you want to force IPv4, use -Djava.net.preferIPv4Stack=true to
>>>>> start all nodes and the GossipRouter.
>>>>>
>>>>> Which version of JGroups is this?
>>>>
>>>> Yes I am specifying -Djava.net.preferIPv4Stack=true. I'm using JGroups
>>>> 3.6.11. Sorry I didn't specify this earlier: with less peers, say 100,
>>>> It is working fine.
>>>>
>>>>>
>>>>>> Is JGroups expected to fail with this many peers or are these errors
>>>>>> strange and shows a problem somewhere in my configuration or somewhere else?
>>>>>>
>>>>>> P.S.: I tried messing with the sysctl parameters from [4] with no luck:
>>>>>
>>>>> You should not need these as the cluster (AIUI) has 6 members. This
>>>>> means 25 TCP connections, which is nothing.
>>>>>
>>>>> How do the clients connect? Does each client connect to a random cluster
>>>>> member? That would be another 200 connections, not an issue, either.
>>>>
>>>> They join the cluster with the channel.connect("EventCluster") function.
>>>
>>>
>>> So your cluster has 200 members? I don't think that's a good idea as the
>>> clients will all have to send their multicast messages through the
>>> sequencer to obtain total order.
>>>
>>> A core cluster of 5-20 members and clients connecting to one of the core
>>> servers (e.g. via STOMP) would be better.
>>
>> We do specifically want to test having a large number of members all
>> exchanging through a choke point compared to a decentralized epidemic
>> protocol. The goal is to find out just how much JGroups in this
>> configuration can hold and see if the epidemic total order algorithm
>> does better at this size.
> 
> OK
> 
>> Considering this, how many members do you think the SEQUENCER could hold?
> 
> I don't know. It depends on how many messages every node TO-sends per 
> sec, and how big those messages are. In any case, the max rate at which 
> a sequencer can order and send is the bandwidth, e.g. 1000Mbits/sec.
> 
>> I agree that in a scenario where you would want to use the SEQUENCER in
>> production, decentralizing the task of distributing the order to the
>> peers is a good idea.
>>
>>>
>>>
>>>>> Note that sequencer based total order will always render the sequencer a
>>>>> bottleneck, as all messages go through the sequencer.
>>>>
>>>> Yes we specifically want to test for a centralized deterministic total
>>>> ordering algorithm.
>>>>
>>>>>
>>>>> Re your config:
>>>>>
>>>>> - Remove FLUSH: I don't assume you want virtual synchrony *and* total
>>>>> order? Note that FLUSH is quite old and hasn't really been maintained
>>>>> the last ~8 years. Also, the SEQUENCER/FLUSH combo has never been tested
>>>>> to my knowledge...
>>>>
>>>> We do want the FLUSH/SEQUENCER combo. (I tested it with less than 200
>>>> peers and it was working fine)
>>>
>>> OK, as you please sir! But you're on your own here, you've been
>>> warned... :-)
>>>
>>>
>>>>> - You should define UDP.ucast_recv_buf_size / ucast_send_buf_size; the
>>>>> defaults are quite small; I suggest copy the UDP section from fast.xml
>>>>> shipped with JGroups. This also contains thread pool min and max sizes
>>>>>
>>>>> - FD_ALL: the default timeouts are quite low which might result in false
>>>>> suspicions; comment this protocol for your perf runs or increase the
>>>>> timeouts
>>>>>
>>>>> - BARRIER and STATE_TRANSFER can be removed unless you do state transfer
>>>>>
>>>>> - UDP.max_bundle_size and FRAG2.frag_size may need tuning
>>>>>
>>>> - I will look into the unicast buffer sizes.
>>>> - We used the FD_ALL settings available in the "udp-largecluster.xml"
>>>> - We do want the state transfer.
>>>> - I will look into it
>>>>
>>>>>
>>>>>> - sysctl net.ipv4.ip_local_port_range="15000 61000"
>>>>>> - sysctl net.ipv4.tcp_fin_timeout=30
>>>>>> - sysctl net.core.somaxconn=1024
>>>>>> - ifconfig eth0 txqueuelen 5000
>>>>>> - sysctl net.core.netdev_max_backlog=2000
>>>>>> - sysctl net.ipv4.tcp_max_syn_backlog=2048
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>> JT
>>>>>>
>>>>>>
>>>>>> [1]
>>>>>> https://github.com/jocelynthode/EpTOTester/blob/master/projects/JGroupsTester/src/main/resources/sequencer-tcpgossip.xml
>>>>>>
>>>>>> [2]
>>>>>> https://github.com/jocelynthode/EpTOTester/blob/master/projects/JGroupsTester/src/main/kotlin/EventTester.kt
>>>>>>
>>>>>> [3] https://gist.github.com/jocelynthode/b4f7d04e248c10672c10fe79210bda27
>>>>>>
>>>>>> [4] http://stackoverflow.com/a/3923785/2826574
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>> Check out the vibrant tech community on one of the world's most
>>>>>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>>>>> _______________________________________________
>>>>>> javagroups-users mailing list
>>>>>> [email protected]
>>>>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>>>>>
>>>>>
>>>>
>>>> Cheers
>>>> JT
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Check out the vibrant tech community on one of the world's most
>>>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>>> _______________________________________________
>>>> javagroups-users mailing list
>>>> [email protected]
>>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>>>
>>>
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>> _______________________________________________
>> javagroups-users mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>
> 

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot