Re: [jgroups-dev] Issue with members joining clusters
Bela Ban <[email protected]> Sun, 02 Dec 2012 17:01:00 +0100
| Newsgroups | gmane.comp.java.javagroups.devel |
|---|---|
| Message-ID | <[email protected]> |
The config you're using is not the one that's shipped with 2.12 (tcp.xml). For instance, TCP has a rejection policy of "discard", not "run". Also, FC has been replaced with UFC and MFC. I suggest try this out with a more recent release, e.g. 3.2. If you can't then use at least the latest version from the 2.12.x branch, and copy (and modify) the tcp.xml config that ships with it. If the problem persists on 3.2, then create a reproduceable use case (code or exact steps to reproduce) that I can run on my system. On 12/1/12 7:54 PM, Sundararajan, Mukund wrote: > Bela, > > Here is my config file. We inject the bind address for TCP and the > heartbeat interval and retry count for FD programmatically We also > inject the initial_hosts list to TCPPINGH programmatically by reading it > from a database. All other settings come from this file > > <?xml version="1.0" encoding="UTF-8"?> > > <!-- JGroups protocol stack properties. --> > > <config xmlns="urn:org:jgroups" > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xsi:schemaLocation="urn:org:jgroups > http://www.jgroups.org/schema/JGroups-3.0.xsd"> > > <TCP bind_port="9802" loopback="true" recv_buf_size="20000000" > > send_buf_size="640000" > discard_incompatible_packets="true" > > max_bundle_size="64000" > max_bundle_timeout="30" > > enable_bundling="true" > > use_send_queues="true" > sock_conn_timeout="300" > > thread_pool.enabled="true" > thread_pool.min_threads="1" > > thread_pool.max_threads="25" > thread_pool.keep_alive_time="5000" > > thread_pool.queue_enabled="false" > thread_pool.queue_max_size="100" > > thread_pool.rejection_policy="run" > oob_thread_pool.enabled="true" > > oob_thread_pool.min_threads="1" > oob_thread_pool.max_threads="8" > > oob_thread_pool.keep_alive_time="5000" > > oob_thread_pool.queue_enabled="false" > > oob_thread_pool.queue_max_size="100" > > oob_thread_pool.rejection_policy="run" /> > > <TCPPING timeout="3000" initial_hosts="" > port_range="0" num_initial_members="1" num_ping_requests="3" /> > > <MERGE2 max_interval="100000" > min_interval="20000" /> > > <FD_SOCK /> > > <FD timeout="10000" max_tries="5" /> > > <VERIFY_SUSPECT timeout="1500" /> > > <BARRIER /> > > <pbcast.NAKACK use_mcast_xmit="false" > > > retransmit_timeout="300,600,1200,2400,4800" discard_delivered_msgs="true" /> > > <UNICAST timeout="300,600,1200" /> > > <pbcast.STABLE stability_delay="1000" > > desired_avg_gossip="50000" > max_bytes="400000" /> > > <pbcast.GMS print_local_addr="true" > join_timeout="3000" > > view_bundling="true" /> > > <FC max_credits="2000000" min_threshold="0.10" /> > > <FRAG2 frag_size="60000" /> > > <AUTH auth_class="org.jgroups.auth.X509Token" > auth_value="IPC_SYSTEMS" > > > keystore_path="/opt/ipc/security/keymgmt/certs/jgroupsks" > > keystore_password="ef675510092260c3" > > cert_alias="1" > > cipher_type="RSA"></AUTH> > > <ENCRYPT encrypt_entire_message="false" > sym_init="128" sym_algorithm="AES/ECB/PKCS5Padding" asym_init="512" > asym_algorithm="RSA" /> > > <!--<pbcast.STREAMING_STATE_TRANSFER /> --> > > <pbcast.STATE_TRANSFER/> > > </config> > > The log snippets I gave you only contain two GET_MBRS_REQ requests but > the log contains all of them. I just included the two to show that these > requests were going out. We are using ENCRYPT with a static shared key > that is pre-installed. I cannot change Discovery to use a protocol based > on multicast. Given this restriction is TCPPING the best option in your > opinion? I do have a list of all hosts in the cluster but all of them > may not be connected to the cluster at all times. > > Another issue we are seeing is that our application on all nodes cannot > send or receive any messages from jgroups. We took thread dumps and see > that they are all blocked on FC. Do our FC settings look ok to you? Here > is the stack for the thread that is stuck à > > RMI TCP Connection(546)-25.221.4.196" id=842175 idx=0x110 tid=10585 > prio=5 alive, parked, native_blocked, daemon > > at jrockit/vm/Locks.park0(J)V(Native Method) > > at jrockit/vm/Locks.park(Locks.java:2230)[optimized] > > at jrockit/proxy/sun/misc/Unsafe.park(Unsafe.java:616)[inlined] > > at > java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:196)[inlined] > > at > java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2116)[optimized] > > at org/jgroups/protocols/FC.handleDownMessage(FC.java:554) > > at org/jgroups/protocols/FC.down(FC.java:427) > > at org/jgroups/protocols/FRAG2.down(FRAG2.java:155) > > at org/jgroups/protocols/AUTH.down(AUTH.java:174) > > at org/jgroups/protocols/ENCRYPT.passItDown(ENCRYPT.java:889) > > at org/jgroups/protocols/ENCRYPT.sendDown(ENCRYPT.java:947) > > at org/jgroups/protocols/ENCRYPT.down(ENCRYPT.java:846) > > at > org/jgroups/protocols/pbcast/STATE_TRANSFER.down(STATE_TRANSFER.java:214) > > at org/jgroups/stack/ProtocolStack.down(ProtocolStack.java:1081) > > at org/jgroups/JChannel.down(JChannel.java:1654) > > at org/jgroups/JChannel.send(JChannel.java:734) > > at > com/ipc/ds/interzone/channel/IZChannel.sendMessage(IZChannel.java:138) > > at > com/ipc/ds/interzone/replicator/channel/PersistentIZChannel.sendMessage(PersistentIZChannel.java:160) > > Thanks > > Mukund > > -----Original Message----- > From: Bela Ban [mailto:[email protected]] > Sent: Saturday, December 01, 2012 5:00 AM > To: Sundararajan, Mukund > Cc: [email protected] > Subject: Re: Issue with members joining clusters > > On 12/1/12 6:45 AM, Sundararajan, Mukund wrote: > > > Hi Bela, > > > > > > We are using jgroups 2.12.Final. Our cluster consists of 25 + nodes. > > > Each node is a RHEL box.We use TCP with a static host list. > > You seem to only list 2 hosts in the static list: 25.221.12.39:9802 and > 25.221.12.38:9802. OTOH, further down the GET_MBRS_REQs are sent to > > 25.221.5.229:9802 and 25.221.6.100:9802, which doesn't see right. > > With TCPPING, you need to list *all* cluster nodes, or else this won't > work ! If you cannot list all nodes, the use a dynamic discovery > protocol such as MPING, TCPGOSSIP, FILE_PING, S3_PING etc... > > You could use PDC [1] in conjunction with TCPPING and then only list > some bootstrap members, but that's available only in later versions. > > BTW, note that I don't support 2.12.x, see [2] for details. > > > We are > > > seeing at some of our customer sites that a node will start and > > > establishes TCP connections successfully to other nodes but doesn’t join > > > the cluster. After a few minutes a Merge View happens and then the node > > > joins the cluster. The delay is several minutes in some cases. Do you > > > have any idea what might be the issue? Netstat shows established > > > connections between the node and its peers. I see the following > > > messages (in blue below) in the jgroups log which suggests the node is > > > sending GET_MBRS_REQ to other nodes and is timing out. The network is > > > good. No issues there. Our protocol stack contains ENCRYPT. My > > > understanding is that ENCRYPT only encrypts application payload. Do GMS > > > packets also get encrypted? > > It depends on where in the stack ENCRYPT is (what's your config ?), e.g. > > if it's above GMS, then GMS headers will also get encrypted, see > > EncryptNoKeyStore.xml as an example. > > > Could there be an ENCRYPT issue? > > I don't think so, but you can try and run the system without ENCRYPT, to > > see if it is the culprit. > > > But that > > > doesn’t explain the merge that happens minutes later. Is this likely to > > > happen if some members in the static host list are down? > > Yes, this is exactly the downside of TCPPING: if all members listed in > > TCPPING.initial_hosts are down, then a new member will form a singleton > > cluster and not ever merge with the other cluster members. > > > The cluster is fine and communicating at the time that this issue > happens. It is fairly > > > easily reproducible in our customer networks although I am unable to > > > reproduce in a lab environment. > > [1] > > http://belaban.blogspot.ch/2012/11/persisting-discovery-responses-with.html > > [2] https://community.jboss.org/wiki/Support > > -- > > Bela Ban, JGroups lead (http://www.jgroups.org) > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > DISCLAIMER: This e-mail may contain information that is confidential, > privileged or otherwise protected from disclosure. If you are not an > intended recipient of this e-mail, do not duplicate or redistribute it > by any means. Please delete it and any attachments and notify the sender > that you have received it in error. Unintended recipients are prohibited > from taking action on the basis of information in this e-mail.E-mail > messages may contain computer viruses or other defects, may not be > accurately replicated on other systems, or may be intercepted, deleted > or interfered with without the knowledge of the sender or the intended > recipient. If you are not comfortable with the risks associated with > e-mail messages, you may decide not to use e-mail to communicate with > IPC. IPC reserves the right, to the extent and under circumstances > permitted by applicable law, to retain, monitor and intercept e-mail > messages to and from its systems. > > -- Bela Ban, JGroups lead (http://www.jgroups.org) ------------------------------------------------------------------------------ Keep yourself connected to Go Parallel: DESIGN Expert tips on starting your parallel project right. http://goparallel.sourceforge.net/ _______________________________________________ Javagroups-development mailing list