Re: [jgroups-users] Migration Question/Issue

Question: How many times did you attempt to crash and rejoin the cluster?  The timing seems to be key in that regards.  We would crash the system about 4.5 seconds after the last batch of messages are sent.  Again this is a timing issue as to when you crash to recreate the issue.  At times, it would work as expected and at other times would delays the batch of messages or continue to send messages to nodes that are crashed.

We are going to implement the changes we have currently using TCP_NIO2.  If we run into any issues with it, would it be possible to have a WebEx meeting to review these issues as they are hard to convey over email or sample programs.  We can also pay a fee for your time to help us rectify any issues.  Please contact me at [email protected] if this is possible.



CJ Binder
Lead Software Engineer 


-----Original Message-----
From: Questions/problems related to using JGroups <[email protected]> 
Sent: Thursday, December 27, 2018 4:29 AM
To: [email protected]
Subject: Re: [jgroups-users] Migration Question/Issue

This works for me without problems.

Your code is a bit weird though. Without understanding what you want to
achieve: you're multicastcasting a message with an AnycastAddress destination address, but it has always only *one* element! I suggest either use MessageDispatcher.sendMessage(target) where target is the non-self member, or MessageDispatcher.castMessage() with a null destination, which means the message will be sent to everyone.

You can exclude the sender from receiving the message by setting the transient flag DONT_LOOPBACK in RequestOptions.

Also, you set a timeout of 300'000 ms in FD_ALL, which is 5 minutes. 
This means blocking RPCs will time out after 30s (the timeout you set in
RequestOptions) *if* a member is killed without the OS closing the open process sockets.


On 19.12.18 23:20, Questions/problems related to using JGroups via javagroups-users wrote:
> Good Day,
> Using the sample program, I believe we have found a pattern difference between when the nodes successfully send and receive messages back and forth after a crash versus timing out when trying to send the messages.  Curious if this behavior is expected from the pattern below.
>   
> 
> Scenario to recreate pattern:
> 1. Bring up Node 1(Coordinator)
> 2. Bring up Node 2(Participant)
> 3. Force crash Node 2(doing so by hitting RED stop button in eclipse) 
> 4. Startup Node 2
> 
> The failure scenario seems to be based upon timing of the crash or how long you wait in between the crash and bringing back up Node 2.
> 
> I added logging of the socket used on the Coordinator when sending messages.  Below you will see that the Coordinator in Stable state uses the socket established by Node 2 to Node 1.  When it is in scenarios when the messaging fails to reach Node2, it appears that it is using a socket established by Node1 to Node2.  This could be expected behavior however it seems to be different when messaging is working as expected versus timing out and messages are not being received by the other nodes.
> 
> All output examples are from Node 1 sending data to Node 2
> 
> Node 1 Bind Address = 10.1.3.9:25001
> Node 2 Bind Address = 10.1.3.30:25001
> 
> Output from when Node 2 first comes online:
> 
> Local Address(/10.1.3.9:25001) Remote Address(/10.1.3.30:52391) - Appears to be sending messages upon socket that Node 2 established with Node 1.
> 
> Output from when Node 2 crashes and we wait long enough for Node 1 to verify suspect and knows not to send anymore messages to Node 2:
> 
> Local Address(/10.1.3.9:25001) Remote Address(/10.1.3.30:53245) - Appears to be sending messages upon socket that Node 2 established with Node 1.
> 
> Output from when Node 2 crashes and we startup Node 2 right away or can sometimes occur based upon when crashed(this is scenario where Node 1 and Node 2 are timing out trying to send messages to each other):
> 
> Local Address(/10.1.3.9:52136) Remote Address(/10.1.3.30:25001) - Appears to be sending messages upon socket that Node 1 established with Node 2.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> CJ Binder
> Lead Software Engineer
> e. [email protected]
> p. 1.800.949.4696 x716
> w. GoAnywhere.com | HelpSystems.com
> 
> 
> -----Original Message-----
> From: Questions/problems related to using JGroups via javagroups-users 
> <[email protected]>
> Sent: Tuesday, December 18, 2018 10:08 AM
> To: [email protected]
> Cc: Questions/problems related to using JGroups 
> <[email protected]>
> Subject: Re: [jgroups-users] Migration Question/Issue
> 
> Were you able to get the sample program?  I am including a Public Link where you can download the sample application.  V2 is the latest version.
> 
> Sample Application Link = 
> https://portal.linoma.com/link/40a870e1-751c-42b4-91c3-7aaf5902b8bb
> 
> We took your note and applied it to the application.  So every node that comes up uses the same initialHosts including itself.
> 
> We have removed ASYM_ENCRYPT at the moment to avoid running into the bugs that have already been reported.
> 
> Our main testing being done is related to crashing the node then starting the node back up and the behavior in between.
> 
> Here is a list of issues we have encountered when terminating a node and bringing it back up.  Most often, we need to repeat the process over and over to get the cluster in a bad state.
> 
> * Certain nodes within the cluster will continue to fail sending messages to the node that crashed and was brought back up.  The sending node will continue to fail indefinitely when sending to the re-established node(killed and brought back up).
> * Certain nodes within the cluster will take 15 seconds on average to send the batch of messages(a single message to each node in the view) when prior to the node crashing would take less than 1 second to send batch.  This seems to rectify itself if you wait long enough.
> 
> 
> Some items from previous testing:
> * If the cluster contains different sub net ip's, we ran into a situation where the cluster already had a Coordinator and once the node with the different subnet crashed and was restarted, that node came up as Coordinator.
> * Inconsistent view changes where some nodes are not notified of the view change and still attempt to send to crashed nodes.
> 
> CJ Binder
> Lead Software Engineer
> 
> 
> -----Original Message-----
> From: Questions/problems related to using JGroups 
> <[email protected]>
> Sent: Wednesday, December 12, 2018 7:19 AM
> To: [email protected]
> Subject: Re: [jgroups-users] Migration Question/Issue
> 
> Note that TCPPING.initial_hosts should always contain *all* (4) IP addresses.
> 
> On 11.12.18 16:28, Questions/problems related to using JGroups via javagroups-users wrote:
>> I have not received any responses on this issue since I built the sample project and sent.
>>
>> Has any of the JGroups developers had a chance to try and run the sample project?
>>
>>
>> CJ Binder
>> Lead Software Engineer
>>
>>
>> -----Original Message-----
>> From: CJ Binder
>> Sent: Thursday, October 11, 2018 4:23 PM
>> To: '[email protected]'
>> <[email protected]>
>> Subject: RE: [jgroups-users] Migration Question/Issue
>>
>> Good Day,
>> Attached is an Eclipse Project I built that mimics our application in a very simplistic manner.
>>
>> The class to run is called ClusterTest.  It takes up to 4 arguments.
>> 1st = Node Name
>> 2nd = Bind Address
>> 3rd = Bind Port
>> 4th = Initial Members(comma separated list members.  Each has ip and port separated by ":".
>>
>> To run the tests here are the arguments in order of startup.  Each subsequent instance needs to supply the initial members that are known.
>>
>> Here are some examples
>>
>> First server(ip address = 10.1.3.1)
>> 1st Arg = Node 1
>> 2nd Arg = 10.1.3.1
>> 3rd Arg = 25000
>> 4th Arg =
>>
>> Second Server(ip address = 10.1.4.1)
>> 1st Arg = Node 2
>> 2nd Arg = 10.1.4.1
>> 3rd Arg = 25000
>> 4th Arg =10.1.3.1:25000
>>
>> Third Server(ip address = 10.1.5.1)
>> 1st Arg = Node 3
>> 2nd Arg = 10.1.5.1
>> 3rd Arg = 25000
>> 4th Arg =10.1.3.1:25000,10.1.4.1:25000
>>
>> Scenarios:
>> Participant crashing
>> 1. Startup First Server
>> 2. Startup Second Server
>> 3. Startup Third Server.
>> 4. Kill Third Server.
>>
>> Does both First and Second server stop sending messages to 3rd server?  The behavior is not consistent so you may need to kill the participant and bring them backup multiple times.   We noticed that the alive nodes still think crashed node is alive and continues to try and send messages to it.
>>
>> Coordinator crashing
>> 1. Startup First Server
>> 2. Startup Second Server
>> 3. Startup Third Server.
>> 4. Kill First Server.
>>
>> Does Third server get the viewAccepted message?  The behavior is not consistent so you may need to kill the coordinator a few times.   We noticed that the Third Server does not get notified that the view changed and continues to try and send messages to First Server.  We do see the Second Server becomes coordinator however the particpant does not.
>>
>> Note: When bringing back up the server, the initial members known at the time may need to be adjusted depending on which servers your forcing to crash.
>>
>> We have tried multiple protocol stacks with varying behavior.  The one in the test was the last we tried.  It did seem a bit more stable when you run all three servers on the same machine in contrast to having them run on different machines in the same network.
>>
>> Side Note: After all servers are started up, the messaging seems to return within the same second.  However after a node crashes, the nodes that are still alive and being communicated with, tend to responed slower to each message sent and received.  We were seeing about a 15 second delay in the response.
>>
>> Thank You for your time and feeback on this manner!
>>
>>
>>
>>
>> CJ Binder
>>
>>
>> -----Original Message-----
>> From: Questions/problems related to using JGroups 
>> <[email protected]>
>> Sent: Tuesday, October 9, 2018 7:41 AM
>> To: [email protected]
>> Subject: Re: [jgroups-users] Migration Question/Issue
>>
>> I see that you don't have MERGE3 on the stack: note that [1] is a fix (not yet resolved) which will make coord leaving behave correctly.
>>
>> Does this work correctly if you remove ASYM_ENCRYPT?
>>
>> Also note that having both a LockService *and* a MessageDispatcher on top of the same channel will likely not work (hasn't been tested): if you want this, use ForkChannels.
>>
>> It would be easier if you came up with a *compilable* small test program (and instructions) that showed the issue...
>>
>> [1] https://issues.jboss.org/browse/JGRP-2293
>>
>> On 05/10/18 22:33, Questions/problems related to using JGroups via javagroups-users wrote:
>>> Good Day,
>>>
>>> We are currently attempting to migrate from 3.3.3 to 4.0.15.  I 
>>> couldn’t find any migration guides so the following is our previous 
>>> setup and current setup.  What we are experiencing is that when a 
>>> participant leaves the cluster when it’s a hard crash(kill jvm), 
>>> some of the participants are not notified(Coordinator is) and still 
>>> think other members exist and attempt to send messages to them 
>>> locking up the application until the timeout of 2 minutes occurs.  
>>> When this happens on the Coordinator, the oldest participant is 
>>> notified and takes over Coordinator role, however the other participants do not get notified.
>>> In our setup, we use the Message Dispatcher to send sync or asyn 
>>> messages to and from 1 or 1 to many members.  Within our 
>>> ClusterMembershipListener, we only have application code within the 
>>> viewAccepted method of the interface.   Any help would be greatly 
>>> appreciated.
>>>
>>> _Testing Scenario_
>>>
>>>     * Scenarios 1
>>>         o Start APP A (Coordinator)
>>>         o Start APP B (Participant)
>>>         o Start APP C (Participant)
>>>         o Kill APP C(Hard kill of process)
>>>         o Verify A can talk to B
>>>         o Verify B can talk to A
>>>     * Scenarios 2
>>>         o Start APP A (Coordinator)
>>>         o Start APP B (Participant)
>>>         o Start APP C (Participant)
>>>         o Kill APP A(Hard kill of process)
>>>         o Verify B takes over as Coordinator
>>>         o Verify B can talk to C
>>>         o Verify C can talk to B
>>>
>>> _Issues_
>>>
>>>     * Scenario 1
>>>         o Works
>>>             + Applications on the same machine
>>>         o Fails
>>>             + Applications are on other machines in same network
>>>                 # APP C still thinks that APP A is coordinator,
>>>                   viewAccepted not invoked
>>>         o Possible Fix
>>>             + Tried setting on GMS – use_delta_views = FALSE
>>>             + This seemed to fix the issue but then ran Scenario 2 still
>>>               an issue
>>>     * Scenario 2
>>>         o Works
>>>             + Applications on the same machine
>>>         o Fails
>>>             + Applications are on other machines in same network
>>>                 # APP B does take over as Coordinator but then is
>>>                   constantly sending Suspect messages to APP A
>>>                 # APP C still thinks App A is Coordinator and continues to
>>>                   send messages to it
>>>
>>> _Migration Changes Made_
>>>
>>>     * ENCRYPT changed to ASYM_ENCRYPT
>>>         o The previous location of encrypt was not working due to key
>>>           exchange.  It was moved up before nakack2 and seems to work in
>>>           our testing now.
>>>     * UNICAST2 changed to UNICAST3
>>>     * Address is no longer serializable, so we now store the mostSig and
>>>       leastSig values in the objects.  We have many messages that include
>>>       objects that store the Address of where the object comes from.  This
>>>       is so we know who to send specific message to.  After this change
>>>       messages are sent and received successfully
>>>     * Removed UFC Protocol as logging stated it is not necessary and is
>>>       safe to remove.
>>>     * Changed how the protocol is supplied to channel as old way is no
>>>       longer supported.
>>>
>>> _3.3.3 (Custom Protocol Stack Initialization)_
>>>
>>> channel= *new*JChannel(*false*);
>>>
>>> channel.setName(SystemInfo./getName/());
>>>
>>> channel.enableStats(ClusterConstants.*/STATS_ENABLED/*);
>>>
>>>                  ProtocolStack stack= *new*ProtocolStack();
>>>
>>> stack.enableStats(ClusterConstants.*/STATS_ENABLED/*);
>>>
>>> channel.setProtocolStack(stack);
>>>
>>> stack.addProtocol(getTCPProtocol());
>>>
>>> *if*(initialMembers!= *null*) {
>>>
>>> stack.addProtocol(getTCPPingProtocol());
>>>
>>>                  }
>>>
>>> stack.addProtocol(getFDSockProtocol());
>>>
>>> stack.addProtocol(getFDAllProtocol());
>>>
>>> stack.addProtocol(getVerifySuspectProtocol());
>>>
>>> stack.addProtocol(getNakack2Protocol());
>>>
>>> stack.addProtocol(getUnicast2Protocol());
>>>
>>> stack.addProtocol(getCentralLockProtocol());
>>>
>>> stack.addProtocol(getStableProtocol());
>>>
>>> stack.addProtocol(getAuthProtocol());
>>>
>>> stack.addProtocol(getGMSProtocol());
>>>
>>> stack.addProtocol(getUfcProtocol());
>>>
>>> stack.addProtocol(getMfcProtocol());
>>>
>>> stack.addProtocol(getEncryptProtocol());
>>>
>>> stack.addProtocol(getFrag2Protocol());
>>>
>>> stack.init();
>>>
>>> channel.setDiscardOwnMessages(*true*);
>>>
>>> dispatcher= *new*MessageDispatcher(channel, *null*,
>>>
>>> *new*ClusterMembershipListener(), *new*ClusterMessageReceiver());
>>>
>>> lockService= *new*LockService(channel);
>>>
>>> channel.connect(ClusterConstants.*/CLUSTER_NAME/*);
>>>
>>> _4.0.15 (Custom Protocol Stack Initialization)_
>>>
>>> List<Protocol> protocols= *new*ArrayList<Protocol>(14);
>>>
>>> protocols.add(getTCPProtocol());
>>>
>>> *if*(initialMembers!= *null*) {
>>>
>>> protocols.add(getTCPPingProtocol());
>>>
>>>                  }
>>>
>>> protocols.add(getFDSockProtocol());
>>>
>>> protocols.add(getFDAllProtocol());
>>>
>>> protocols.add(getVerifySuspectProtocol());
>>>
>>> protocols.add(getAsymEncryptProtocol());
>>>
>>> protocols.add(getNakack2Protocol());
>>>
>>> protocols.add(getUnicast3Protocol());
>>>
>>> protocols.add(getCentralLockProtocol());
>>>
>>> protocols.add(getStableProtocol());
>>>
>>> protocols.add(getAuthProtocol());
>>>
>>> protocols.add(getGMSProtocol());
>>>
>>> protocols.add(getMfcProtocol());
>>>
>>> protocols.add(getFrag2Protocol());
>>>
>>> channel= *new*JChannel(protocols);
>>>
>>> channel.setName(SystemInfo./getName/());
>>>
>>> channel.setStats(ClusterConstants.*/STATS_ENABLED/*);
>>>
>>> channel.getProtocolStack().enableStats(ClusterConstants.*/STATS_ENAB
>>> L
>>> E
>>> D/*);
>>>
>>> channel.setDiscardOwnMessages(*true*);
>>>
>>> dispatcher= *new*MessageDispatcher(channel,
>>>
>>> *new*ClusterMessageReceiver());
>>>
>>> dispatcher.setMembershipListener(*new*ClusterMembershipListener());
>>>
>>> lockService= *new*LockService(channel);
>>>
>>> channel.connect(ClusterConstants.*/CLUSTER_NAME/*);
>>>
>>> CJ Binder
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> javagroups-users mailing list
>>> [email protected]
>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>>
>>
>> --
>> Bela Ban | http://www.jgroups.org
>>
>>
>>
>> _______________________________________________
>> javagroups-users mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>
>> _______________________________________________
>> javagroups-users mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
>>
> 
> --
> Bela Ban | http://www.jgroups.org
> 
> 
> 
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
> 
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
> 
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
> 

--
Bela Ban | http://www.jgroups.org



_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users

_______________________________________________
javagroups-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/javagroups-users