[jgroups-dev] JGroups 3.0.8 TCP and large numbers of unreachable hosts

Hi,

We are currently running JGroups 3.0.8.Final to allow data distribution across 80 nodes using TCP/TCPPING (as UDP is not allowed on our network) Current configuration is appended at the end.

During testing there have been issues with the cluster not forming correctly. This appears to happen when the number of unavailable machines multiplied by the port range, exceeds the number of threads. These are no obvious errors from the system.

I've narrowed now our test to have 32 entries in the initial hosts, 2 physical machines and another 30 entries that don't exist i.e. are unreachable. Running up the two physical instances, with a port_range of 0, result in a cluster of 2 forming correctly. If the port range is increased from 0, both report being coordinator and never appear to merge. Increasing the number of unreachable machines also results in a cluster not forming. Increasing the max threads to cover the number of unreachable machines (and ports) allows the cluster to form.  I've tried playing the queue settings with no difference; it seems to be directly related to the number of missing machines, the port rage and the number of available threads. Our system needs to be resilient to most of the cluster being missing/unreachable, so this is a major risk as we can't guarantee the cluster forming.

Are there any recommended settings to use in a TCP configuration of 80 nodes (port_range="0")?

Is there something else I'm missing that is preventing the cluster forming under these circumstances?

I've tried the setup under UDP with no issues but we are unable to use this within our target system.

Many thanks for any suggestions,
Matt

Current Configuration (Initial hosts shortened for clarity):
<config xmlns="urn:org:jgroups" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/JGroups-3.0.xsd">
  <TCP
          bind_addr="192.168.10.1"
bind_port="7800"
loopback="true"
recv_buf_size="200000000"
send_buf_size="640000"
discard_incompatible_packets="true"
max_bundle_size="64000"
max_bundle_timeout="30"
enable_bundling="true"
use_send_queues="true"
sock_conn_timeout="300"
enable_diagnostics="false"

thread_pool.enabled="true"
thread_pool.min_threads="2"
thread_pool.max_threads="30"
thread_pool.keep_alive_time="60000"
thread_pool.queue_enabled="true"
thread_pool.queue_max_size="100"
thread_pool.rejection_policy="discard"

oob_thread_pool.enabled="true"
oob_thread_pool.min_threads="2"
oob_thread_pool.max_threads="30"
oob_thread_pool.keep_alive_time="60000"
oob_thread_pool.queue_enabled="true"
oob_thread_pool.queue_max_size="100"
oob_thread_pool.rejection_policy="discard" />
  <DAISYCHAIN />
  <TCPPING
timeout="3000"
initial_hosts="192.168.10.1,192.168.10.2,192.168.99.1 ,<...>, 192.168.99.30"
port_range="0"
num_initial_members="3" />
  <MERGE2 min_interval="10000" max_interval="30000" />
  <FD_SOCK />
  <FD_ALL />
  <VERIFY_SUSPECT timeout="1500" />
  <pbcast.NAKACK
use_mcast_xmit="false"
exponential_backoff="300,600,1200,2400,4800"
discard_delivered_msgs="false" />
  <UNICAST2 timeout="300,600,1200" />
  <pbcast.STABLE
stability_delay="1000"
desired_avg_gossip="50000"
max_bytes="400000" />
  <pbcast.GMS
print_local_addr="true"
join_timeout="7000"
view_bundling="true" />
  <UFC max_credits="2000000" min_threshold="0.10" />
  <MFC max_credits="2000000" min_threshold="0.10" />
  <FRAG2 frag_size="60000" />
</config>

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/

_______________________________________________
Javagroups-development mailing list