Re: [jgroups-users] removing unknown address from cluster? JGRP000032
Questions/problems related to using JGroups <[email protected]> Mon, 3 May 2021 11:02:57 -0400
| Newsgroups | gmane.comp.java.javagroups.general |
|---|---|
| Message-ID | <mailman.50095.1620054232.1299.javagroups-users@lists.sourceforge.net> |
--===============8397434913736308890== Content-Type: multipart/alternative; boundary="000000000000e79e2905c16e43e8" --000000000000e79e2905c16e43e8 Content-Type: text/plain; charset="UTF-8" Hi again, Thanks for this. I have more information from the customer now, and see that the problem they're having isn't due to incorrect host information at startup like I thought. The setup to reproduce is pretty simple, and I understand their point that it doesn't look like user error. 1. Set up cluster A/B/C (A is coordinator). 2. At some point they don't need C in the cluster anymore and shut down the application there. It's a regular shutdown, not going suspect first. We use JChannel#close and then exit. 3. Later they use a node with the same address to join a different cluster with the same name. When C starts it only has D's address, and forms cluster D/C. After the above, the A/B cluster is getting a merge view change every ~minute, always including only A/B in the view. The log on A is also filling with: JGRP000032: <A>: no physical address for <D>, dropping message Because it's a merge view, we do extra processing to handle potential rejoin cases, which causes a couple other warnings every minute. I also see every ~minute that A tries to authorize itself with C. C's log has messages from our custom AuthToken class. If I use a different cluster for C/D that avoids a lot of the issues. There are no longer view changes and warnings in the first cluster, but the new one D/C has this in C's log constantly: JGRP000012: discarded message from different cluster <old> (our cluster is <new>). Sender was <A> That will help them some, but it's a large organization and they have a lot of clusters, since we thought it would be ok to reuse the name as long as the addresses weren't shared. Is there anything we can do to make a cluster forget a member that has left gracefully? Thanks, Bobby On Tue, Apr 6, 2021 at 7:46 AM Questions/problems related to using JGroups via javagroups-users <[email protected]> wrote: > You can always change the list of initial hosts in TCPPING > programmatically, via getInitialHosts() / setInitialHosts(). > > Detecting that an address is wrong is outside the scope of JGroups, and > should be done (IMO) by your application, e.g. at > config/installation/startup time. > > This can of course be arbitrarily difficult, e.g. > * See if a symbolic name resolves correctly > * Check if a host is pingable > > You could also disallow a user from entering hostnames/IP addresses > him/herself directly and instead generate them yourself, e.g. by > recording all hosts on which an installation was performed and using > this as initial_hosts. > > You could also think of adding a protocol which checks (in init() or > start()) that the hostnames/addresses in TCPPING.initial_hosts resolve, > and possibly ping all entries before starting the stack. > > On a related note, take a look at [1] (added in 4.2.12): it skips > unresolved/unresolvable entries until an entry finally does resolve. > > Hope this helps, > > [1] https://issues.redhat.com/browse/JGRP-2535 > > On 05.04.21 22:50, Questions/problems related to using JGroups wrote: > > Hi, > > > > Our product uses the TCP stack with jgroups 4.1.8. It gets set up by end > > users through a configuration file that contains (among other things), a > > list of IP addresses for a node to connect to when joining a cluster. We > > set this for TCPPING.initial_hosts. > > > > If they have a wrong address at startup they end up getting JGRP000032 > > warnings filling the logs. For instance, the following leads to logs > > filling on two nodes, one of which was set up correctly: > > > > 1. Start cluster A/B. A is the coordinator. > > 2. Start a one-node cluster C. > > 3. On node D, include addresses for D and B in the initial hosts list > > and attempt to join. > > 4. D will join C for a cluster C/D and, obviously, not join A/B since it > > didn't attempt to connect to the coordinator. > > > > After this, the logs for D will fill with: > > WARN: JGRP000032: <D>: no physical address for <A>, dropping message > > > > ...and B logs will fill with: > > WARN: JGRP000032: <B>: no physical address for <C>, dropping message > > > > I know this is a setup error on the user's side, but was wondering if > > there's anything we could add programmatically to stop it. For instance, > > when they see the logs on X filling up with messages about Y in another > > cluster, is there something we could do to tell X to forget Y exists? > > It's not enough just to stop/fix/start that cluster, as (in the case of > > A/B above) the cluster that was started correctly could be showing this > > problem. For some customers, getting a maintenance window to shut down > > all related clusters and restart them is a problem. > > > > For that matter, is there anything programmatically we could do to > > detect that this is happening? Besides parsing the jgroups logging > > output I mean. > > > > Thank you, > > Bobby > > > > > > > > _______________________________________________ > > javagroups-users mailing list > > [email protected] > > https://lists.sourceforge.net/lists/listinfo/javagroups-users > > > > -- > Bela Ban | http://www.jgroups.org > > > > _______________________________________________ > javagroups-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/javagroups-users > --000000000000e79e2905c16e43e8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr">Hi again,<div><br></div><div>Thanks for t= his. I have more information from the customer now, and see that the proble= m they're having isn't due to incorrect host information at startup= like I thought. The setup to reproduce is pretty simple, and I understand = their point that it doesn't look like user error.</div><div><br></div><= div>1. Set up cluster A/B/C (A is coordinator).</div><div>2. At some point = they don't need C in the cluster anymore and shut down the application = there. It's a regular shutdown, not going suspect first. We use JChanne= l#close and then exit.</div><div>3. Later they use a node with the same add= ress to join a different cluster with the same name. When C starts it only = has D's address, and forms cluster D/C.</div><div><br></div><div>After = the above, the A/B cluster is getting a merge view change every ~minute, al= ways including only A/B in the view. The log on A is also filling with:</di= v><div>JGRP000032: <A>: no physical address for <D>, dropping m= essage<br></div><div>Because it's a merge view, we do extra processing = to handle potential rejoin cases, which causes a couple other warnings ever= y minute.</div><div><br></div><div>I also see every ~minute that A tries to= authorize itself with C. C's log has messages from our custom=C2=A0Aut= hToken class.</div><div><br></div><div><br></div><div>If I use a different = cluster for C/D that avoids a lot of the issues. There are no longer view c= hanges and warnings in the first cluster, but the new one D/C has this in C= 's log constantly:</div><div>JGRP000012: discarded message from differe= nt cluster <old> (our cluster is <new>). Sender was <A><b= r></div><div><br></div><div>That will help them some, but it's a large = organization and they have a lot of clusters, since we thought it would be = ok to reuse the name as long as the addresses weren't shared. Is there = anything we can do to make a cluster forget a member that has left graceful= ly?</div><div><br></div><div>Thanks,</div><div>Bobby</div><div><br></div><d= iv><br></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D= "gmail_attr">On Tue, Apr 6, 2021 at 7:46 AM Questions/problems related to u= sing JGroups via javagroups-users <<a href=3D"mailto:javagroups-users@li= sts.sourceforge.net">[email protected]</a>> wrote:<= br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8e= x;border-left:1px solid rgb(204,204,204);padding-left:1ex">You can always c= hange the list of initial hosts in TCPPING <br> programmatically, via getInitialHosts() / setInitialHosts().<br> <br> Detecting that an address is wrong is outside the scope of JGroups, and <br= > should be done (IMO) by your application, e.g. at <br> config/installation/startup time.<br> <br> This can of course be arbitrarily difficult, e.g.<br> * See if a symbolic name resolves correctly<br> * Check if a host is pingable<br> <br> You could also disallow a user from entering hostnames/IP addresses <br> him/herself directly and instead generate them yourself, e.g. by <br> recording all hosts on which an installation was performed and using <br> this as initial_hosts.<br> <br> You could also think of adding a protocol which checks (in init() or <br> start()) that the hostnames/addresses in TCPPING.initial_hosts resolve, <br= > and possibly ping all entries before starting the stack.<br> <br> On a related note, take a look at [1] (added in 4.2.12): it skips <br> unresolved/unresolvable entries until an entry finally does resolve.<br> <br> Hope this helps,<br> <br> [1] <a href=3D"https://issues.redhat.com/browse/JGRP-2535" rel=3D"noreferre= r" target=3D"_blank">https://issues.redhat.com/browse/JGRP-2535</a><br> <br> On 05.04.21 22:50, Questions/problems related to using JGroups wrote:<br> > Hi,<br> > <br> > Our product uses the TCP stack with jgroups 4.1.8. It gets set up by e= nd <br> > users through a configuration file that contains (among other things),= a <br> > list of IP addresses for a node to connect to when joining a cluster. = We <br> > set this for=C2=A0TCPPING.initial_hosts.<br> > <br> > If they have a wrong address at startup they end up getting=C2=A0JGRP0= 00032 <br> > warnings filling the logs. For instance, the following leads to logs <= br> > filling on two nodes, one of which was set up correctly:<br> > <br> > 1. Start cluster A/B. A is the coordinator.<br> > 2. Start a one-node cluster C.<br> > 3. On node D, include addresses for D and B in the initial hosts list = <br> > and attempt to join.<br> > 4. D will join C for a cluster C/D and, obviously, not join A/B since = it <br> > didn't attempt to connect to the coordinator.<br> > <br> > After this, the logs for D will fill with:<br> > WARN: JGRP000032: <D>: no physical address for <A>, droppi= ng message<br> > <br> > ...and B logs will fill with:<br> > WARN: JGRP000032: <B>: no physical address for <C>, droppi= ng message<br> > <br> > I know this is a setup error on the user's side, but was wondering= if <br> > there's anything we could add programmatically to stop it. For ins= tance, <br> > when they see the logs on X filling up with messages about Y in anothe= r <br> > cluster, is there something we could do to tell X to forget Y exists? = <br> > It's not enough just to stop/fix/start that cluster, as (in the ca= se of <br> > A/B above) the cluster that was started correctly could be showing thi= s <br> > problem. For some customers, getting a maintenance window to shut down= <br> > all related clusters and restart them is a problem.<br> > <br> > For that matter, is there anything programmatically=C2=A0we could do t= o <br> > detect that this is happening? Besides parsing the jgroups logging <br= > > output I mean.<br> > <br> > Thank you,<br> > Bobby<br> > <br> > <br> > <br> > _______________________________________________<br> > javagroups-users mailing list<br> > <a href=3D"mailto:[email protected]" target=3D"_b= lank">[email protected]</a><br> > <a href=3D"https://lists.sourceforge.net/lists/listinfo/javagroups-use= rs" rel=3D"noreferrer" target=3D"_blank">https://lists.sourceforge.net/list= s/listinfo/javagroups-users</a><br> > <br> <br> -- <br> Bela Ban | <a href=3D"http://www.jgroups.org" rel=3D"noreferrer" target=3D"= _blank">http://www.jgroups.org</a><br> <br> <br> <br> _______________________________________________<br> javagroups-users mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank"= >[email protected]</a><br> <a href=3D"https://lists.sourceforge.net/lists/listinfo/javagroups-users" r= el=3D"noreferrer" target=3D"_blank">https://lists.sourceforge.net/lists/lis= tinfo/javagroups-users</a><br> </blockquote></div></div> --000000000000e79e2905c16e43e8-- --===============8397434913736308890== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============8397434913736308890== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ javagroups-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/javagroups-users --===============8397434913736308890==--