Re: [jgroups-dev] JGroups 3.0.8 TCP and large numbers of unreachable hosts

On 5/8/12 4:11 PM, Wellard, Matthew wrote:
> Hi,
>
> We are currently running JGroups 3.0.8.Final to allow data distribution across 80 nodes using TCP/TCPPING (as UDP is not allowed on our network). Current configuration is appended at the end.


Make sure you list *all* 80 nodes in TCPPING.initial_hosts. Also, if 
every node sends messages, you're going to have a mesh of ~6400 TCP 
connections. 1 node P doing a 'multicast' (message to all) M needs to 
send M 79 times. So if M is 1K, P actually sends 79K worth of data to 
the switch.

Therefore I don't recommend TCP for larger clusters.

OK, having said that, I recommend you enable connection reaping, so that 
idle connections are closed after some time of inactivity: 
TCP.conn_expire_time="60000" TCP.reaper_interval="30000".

I see that you're using DAISYCHAIN, interesting ! This should mitigate 
the connection problem above to some degree, however, DAISYCHAIN is 
relatively untested... I recommend move it up the stack, just under NAKACK.

Speaking of which, moving to 3.1.x, UNICAST2 has been ported to use 
Table, and there is a new NAKACK2 protocol, both of which should reduce 
memory / CPU used, so I suggest consider moving to 3.1 once it's 
released. There were quite a few things implemented in 3.1, which makes 
running JGroups in large clusters better. Note that there is a 
udp-largecluster.xml which could be taken as a starting point, replace 
UDP with TCP and PING with TCPPING to arrive at a TCP based config, plus 
some minor modifications.


> During testing there have been issues with the cluster not forming correctly. This appears to happen when the number of unavailable machines multiplied by the port range, exceeds the number of threads. These are no obvious errors from the system.

I suggest set the port range to 0. If your bind_port is 7800, then we'll 
only probe 7800, so if a node is running at 7801, it won't be found, but 
I believe that's ok unless you're running multiple instances on a 
physical box.

Note that if you have a shared filesystem, you could replace TCPPING 
with FILE_PING, which makes things more dynamic, and doesn't require a 
static list of nodes.


> I've narrowed now our test to have 32 entries in the initial hosts, 2 physical machines and another 30 entries that don't exist i.e. are unreachable. Running up the two physical instances, with a port_range of 0, result in a cluster of 2 forming correctly. If the port range is increased from 0, both report being coordinator and never appear to merge. Increasing the number of unreachable machines also results in a cluster not forming. Increasing the max threads to cover the number of unreachable machines (and ports) allows the cluster to form.  I've tried playing the queue settings with no difference; it seems to be directly related to the number of missing machines, the port rage and the number of available threads. Our system needs to be resilient to most of the cluster being missing/un
 reachable, so this is a major risk as we can't guarantee the cluster forming.
> Are there any recommended settings to use in a TCP configuration of 80 nodes (port_range="0")?


The discovery requests to the 80 nodes (assuming you have 80 listed) are 
sent in parallel, using the timer, so the timer should be configured to 
have at least 80 max_threads, e.g.
timer.min_threads="20" timer.max_threads="80" timer.keep_alive_time="5000"

Since the requests and responses are received using the OOB pool, we 
have to look at the OOB thread pool config as well. I see that you 
changed the default to have a queue enabled. This is not recommended, I 
suggest use the default, e.g.:
oob_thread_pool.min_threads="10
oob_thread_pool.max_threads="80"
oob_thread_pool.keep_alive_time="10000"
oob_thread_pool.queue_enabled="false"
oob_thread_pool.rejection_policy="discard"



> Is there something else I'm missing that is preventing the cluster forming under these circumstances?
>
> I've tried the setup under UDP with no issues but we are unable to use this within our target system.
>
> Many thanks for any suggestions,
> Matt
>
> Current Configuration (Initial hosts shortened for clarity):
> <config xmlns="urn:org:jgroups" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/JGroups-3.0.xsd">
>    <TCP
>            bind_addr="192.168.10.1"
> bind_port="7800"
> loopback="true"
> recv_buf_size="200000000"
> send_buf_size="640000"
> discard_incompatible_packets="true"
> max_bundle_size="64000"
> max_bundle_timeout="30"
> enable_bundling="true"
> use_send_queues="true"
> sock_conn_timeout="300"
> enable_diagnostics="false"
>
> thread_pool.enabled="true"
> thread_pool.min_threads="2"
> thread_pool.max_threads="30"
> thread_pool.keep_alive_time="60000"
> thread_pool.queue_enabled="true"
> thread_pool.queue_max_size="100"
> thread_pool.rejection_policy="discard"
>
> oob_thread_pool.enabled="true"
> oob_thread_pool.min_threads="2"
> oob_thread_pool.max_threads="30"
> oob_thread_pool.keep_alive_time="60000"
> oob_thread_pool.queue_enabled="true"
> oob_thread_pool.queue_max_size="100"
> oob_thread_pool.rejection_policy="discard" />
>    <DAISYCHAIN />
>    <TCPPING
> timeout="3000"
> initial_hosts="192.168.10.1,192.168.10.2,192.168.99.1 ,<...>, 192.168.99.30"
> port_range="0"
> num_initial_members="3" />
>    <MERGE2 min_interval="10000" max_interval="30000" />
>    <FD_SOCK />
>    <FD_ALL />
>    <VERIFY_SUSPECT timeout="1500" />
>    <pbcast.NAKACK
> use_mcast_xmit="false"
> exponential_backoff="300,600,1200,2400,4800"
> discard_delivered_msgs="false" />
>    <UNICAST2 timeout="300,600,1200" />
>    <pbcast.STABLE
> stability_delay="1000"
> desired_avg_gossip="50000"
> max_bytes="400000" />
>    <pbcast.GMS
> print_local_addr="true"
> join_timeout="7000"
> view_bundling="true" />
>    <UFC max_credits="2000000" min_threshold="0.10" />
>    <MFC max_credits="2000000" min_threshold="0.10" />
>    <FRAG2 frag_size="60000" />
> </config>


[1] http://belaban.blogspot.com/2012/02/jgroups-310alpha2-released.html

-- 
Bela Ban, JGroups lead (http://www.jgroups.org)

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Javagroups-development mailing list