Re: [jgroups-users] JDBC - Cloning DB
Questions/problems related to using JGroups <[email protected]> Thu, 21 Jun 2018 12:50:38 +0000
| Newsgroups | gmane.comp.java.javagroups.general |
|---|---|
| Message-ID | <mailman.14050.1529585448.978.javagroups-users@lists.sourceforge.net> |
> -----Original Message----- > From: Questions/problems related to using JGroups [mailto:javagroups- > [email protected]] > Sent: 21 June 2018 13:38 > To: [email protected] > Subject: Re: [jgroups-users] JDBC - Cloning DB > > > > On 21/06/18 14:23, Questions/problems related to using JGroups wrote: > > Hi Bela, > > > >> -----Original Message----- > >> From: Questions/problems related to using JGroups [mailto:javagroups- > >> [email protected]] > >> Sent: 21 June 2018 12:52 > >> To: [email protected] > >> Subject: Re: [jgroups-users] JDBC - Cloning DB > >> > >> With all due respect, I don't think MERGE2 or MERGE3 is the root cause, > >> it is only used to fix the root cause... > > > > Given my poor knowledge on the matter I can only agree. > > > > Since we have updated, for other reasons, to MERGE3 I can't see the log > lines being printed out, in > > fact the problem "does not manifest itself", which does not mean I have > understood what the problem was. > > If I were to understand exactly what happened where do you reckon I > should be starting from ? > > I think the problem may have disappeared just by chance. I would start > looing into whether that export/import procedure creates exactly the > same DB table on both sides, and what happens when the table is updated. > > I still don't understand why you have 2 DBs instead of 1. And if you > need this for HA, then I suggest to use the DB's replication mechanism > to replicate your data, and not a self-made and brittle export/import > mechanism. > My bad, I probably did not make this clear. We don't have two DBs, but once a mistake occurred and I wanted to check what may have caused those messages being printed out - this may help me in clarifying how the whole thing works. Is there a quick win to log all actions performed on the table by a certain node ? Without me having to insert manual prints or place breakpoints. Well, I bothered you enough with this thing I reckon :-) > > > P2 writes: > >>>>> 17:58:00.913 WARN OOB-2,LT1038-21441 org.jgroups.protocols.TCP > >> xxxxxx: > >>>> no physical address for 0eece642-4813-bb4a-934f-46d886cad55b, > dropping > >>>> message > > > > What could be the source of that message, I can only speculate :-( > > > > Maybe when p1 and p2 have started p2 establishes a TCP connection to p1 - > as the JGroupsPing table is copied across, by mistake - and as a result of that > > includes it among its cluster member and sending messages to it ? > > > > > > Thanks, > > P. > > > > > >> > >> On 21/06/18 12:55, Questions/problems related to using JGroups wrote: > >>> Hi Bela, > >>> > >>> And indeed the "problem" manifest itself only if MERGE2 is used, while it > >> disappears when upgrading to MERGE3. > >>> > >>> Thanks (again) > >>> P. > >>> > >>> > >>>> -----Original Message----- > >>>> From: Questions/problems related to using JGroups [mailto:javagroups- > >>>> [email protected]] > >>>> Sent: 20 June 2018 15:28 > >>>> To: [email protected] > >>>> Subject: Re: [jgroups-users] JDBC - Cloning DB > >>>> > >>>> The config you show below suggest you're using an old version of > JGroups > >>>> (e.g. UNICAST2, MuxRequestCorrelator, TCP's shared transport etc), can > >>>> you try this with the latest 4.x or 3.6.x release? > >>>> > >>>> Comments below > >>>> > >>>> On 19/06/18 19:21, Questions/problems related to using JGroups wrote: > >>>>> Hello, > >>>>> > >>>>> The application I am working with makes use of JDBC_PING for > discovery > >>>> and I've recently stumbled upon a scenario which involves the following > >> stack > >>>> configuration : > >>>>> > >>>>> <config xmlns="urn:org:jgroups" > >>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > >>>>> xsi:schemaLocation="urn:org:jgroups file:schema/JGroups- > 3.2.xsd"> > >>>>> <TCP [..] > >>>>> /> > >>>>> > >>>>> <JDBC_PING connection_url="jdbc:mysql://localhost:3306/jgroups" > >>>>> connection_username="test" > >>>>> connection_password="test" > >>>>> connection_driver="com.mysql.jdbc.Driver" > >>>>> /> > >>>>> <MERGE2 max_interval="30000" min_interval="10000"/> > >>>>> <FD_SOCK/> > >>>>> <FD timeout="3000" max_tries="3"/> > >>>>> <VERIFY_SUSPECT timeout="1500"/> > >>>>> <pbcast.NAKACK2 use_mcast_xmit="false" > >>>>> [...] > >>>>> /> > >>>>> <UNICAST2 stable_interval="5000" > >>>>> [..] > >>>>> /> > >>>>> <pbcast.STABLE stability_delay="500" desired_avg_gossip="5000" > >>>> max_bytes="1m"/> > >>>>> <pbcast.GMS print_local_addr="false" join_timeout="3000" > >>>> view_bundling="true"/> > >>>>> <UFC max_credits="200k" min_threshold="0.20"/> > >>>>> <MFC max_credits="200k" min_threshold="0.20"/> > >>>>> <FRAG2 frag_size="60000"/> > >>>>> <RSVP timeout="60000" resend_interval="500" > ack_on_delivery="false" > >> /> > >>>>> </config> > >>>>> > >>>>> There two DB, db1 and db2 > >>>> > >>>> > >>>> > >>>> Why two databases? The point of JDBC_PING is to have a common DBMS > >>>> which > >>>> all members access... > >>>> > >>>> > >>>> > >>>>> and two processes p1 and p2. P1 uses db1 and p2 db2, they all are into > >> the > >>>> same subnet and they both share the same code, for instance > >>>>> they both create a channel with the same name. > >>>> > >>>> Again, are those 2 DBs in sync, so when P1 inserts its data into DB1, > >>>> will P2 read the same data out of DB2? > >>>> > >>>>> P1 starts up and it becomes a single node cluster, thereafter, for any > >>>> reason, the whole db1 is exported and imported into > >>>>> db2 thus causing the JDBC_PING table to be the same as p1 when p2 > >> starts. > >>>> > >>>> OK. Any reason for this? What if P1 and P2 start at the same time? > >>>> > >>>> Or what happens if P2 updates DB2? Will the update be exported back > to > >>>> DB1? What if P1 updates DB1 at *exactly the same time*? Exporting and > >>>> (probably) overwriting a DB's state seems very brittle IMO... > >>>> > >>>> > >>>>> The issue I am trying to get at the bottom of is that p2 keeps printing : > >>>>> > >>>>> 17:58:00.913 WARN OOB-2,LT1038-21441 org.jgroups.protocols.TCP > >> xxxxxx: > >>>> no physical address for 0eece642-4813-bb4a-934f-46d886cad55b, > dropping > >>>> message > >>>> > >>>> > >>>> This means that P2 probably wanted to send a message to P1 and didn't > >>>> find P1's IP address in DB2. DB2 should contain IP addresses for both P1 > >>>> and P2. You should enable tracing on JDBC_PING to see what's going on, > >>>> or dump the contents of each DB when starting P2. > >>>> > >>>> > >>>>> ---------------- TRACE (received) ---------------------- > >>>>> Event[type=MSG, arg=[dst: <null>, src: xxxxx (4 headers), size=1273 > bytes]] > >>>> (headers=MuxRequestCorrelator: MuxHeader(29318), > RequestCorrelator: > >>>> id=200, type=REQ, id=162, rsp_expected=false, NAKACK2: [MSG, > >> seqno=161], > >>>> TCP: [channel_name=ControlChannel]) > >>>>> -------------------------------------------------------- > >>>>> ---------------- TRACE (received) ---------------------- > >>>>> Event[type=MSG, arg=[dst: <null>, src: xxxxx (4 headers), size=1273 > bytes]] > >>>> (headers=MuxRequestCorrelator: MuxHeader(29318), > RequestCorrelator: > >>>> id=200, type=REQ, id=162, rsp_expected=false, NAKACK2: [MSG, > >> seqno=161], > >>>> TCP: [channel_name=ControlChannel]) > >>>>> -------------------------------------------------------- > >>>> > >>>> Do you need muxing? I assume you use a shared transport (in TCP)? This > >>>> was deprecated and removed a long time ago, in favor of FORK. > >>>> > >>>> > >>>>> ---------------- TRACE (received) ---------------------- > >>>>> Event[type=IS_MERGE_IN_PROGRESS, arg=null] > >>>>> -------------------------------------------------------- > >>>>> ---------------- TRACE (received) ---------------------- > >>>>> Event[type=IS_MERGE_IN_PROGRESS, arg=null] > >>>>> -------------------------------------------------------- > >>>>> > >>>>> The two tables hold, no surprise, a very different state of the cluster > and > >> p2 > >>>> is indeed receiving a message from somebody which is not part of its > table. > >>>> > >>>> Why do the tables hold different state? They should hold exactly the > >>>> same state, or else you're export/import function didn't function > correctly. > >>>> > >>>> I suggest try this with a single DB, and both P1 and P2 point to it. > >>>> > >>>>> I am not quite sure my reading of the situation is correct, it seems that > p2 > >>>> connects to p1 to notify its existence and thereafter p1, which is the > >>>> coordinator > >>>>> tries to perform a merge. > >>>> > >>>> Right: P2 starts up and becomes singleton because it cannot send a JOIN > >>>> request message to P1. So we have 2 clusters {P1} and {P2} and then they > >>>> try to merge. But this will probably fails, too, because P1 and P2 > >>>> cannot communicate... > >>>> > >>>>> Do you have any suggestion on how I could further debug what > happened > >> ? > >>>> > >>>> 1: use a more recent version of JGroups > >>>> 2: use a single DB > >>>> > >>>>> I would like to trace down the JOINS() being sent to the p1 to make > sure > >> the > >>>> above statement is correct, and then I would like to work > >>>>> out if there is any problem with the stack's configuration or if the > >> problem is > >>>> un-avoidable as the root of the rub is the DB being copied across. > >>>>> > >>>>> I am used to debug the view by setting a custom/dummy receiver which > >>>> would merely print the view without doing anything, I am a bit wary of > >> doing > >>>> that in my application > >>>>> and I am wondering if it is possible to programmatically enable the > >> logging > >>>> across all the stack so to have a whole view or what's going on without > >>>> impacting the stack's > >>>>> "functional" configuration. > >>>> > >>>> You could use probe.sh, e.g. probe.sh jmx=JDBC_PING.level="trace" > >>>> > >>>>> Thanks, > >>>>> P. > >>>>> > >>>>> > >>>>> > >>>>> Pietro Paolini > >>>>> Consultant > >>>>> > >>>>> Alfa > >>>>> ________________________________ > >>>>> e: [email protected] | w: > >>>> alfasystems.com<https://www.alfasystems.com> > >>>>> t: +44 (0) 20 7920-2643 | Moor Place, 1 Fore Street Avenue, London, > EC2Y > >>>> 9DT, GB > >>>>> ________________________________ > >>>>> > >>>>> The contents of this communication are not intended to be binding or > >>>> constitute any form of offer or acceptance or give rise to any legal > >> obligations > >>>> on behalf of the sender or Alfa. The views or opinions expressed > represent > >>>> those of the author and not necessarily those of Alfa. This email and any > >>>> attachments are strictly confidential and are intended solely for use by > the > >>>> individual or entity to whom it is addressed. If you are not the > addressee > >> (or > >>>> responsible for delivery of the message to the addressee) you may not > >> copy, > >>>> forward, disclose or use any part of the message or its attachments. At > >>>> present the integrity of email across the internet cannot be guaranteed > >> and > >>>> messages sent via this medium are potentially at risk. All liability is > >> excluded > >>>> to the extent permitted by law for any claims arising as a result of the > use > >> of > >>>> this medium to transmit information by or to Alfa or its affiliates. > >>>>> > >>>>> Alfa Financial Software Ltd > >>>>> Reg. in England No: 0248 2325 > >>>>> > >>>>> ------------------------------------------------------------------------------ > >>>>> Check out the vibrant tech community on one of the world's most > >>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot > >>>>> _______________________________________________ > >>>>> javagroups-users mailing list > >>>>> [email protected] > >>>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users > >>>>> > >>>> > >>>> -- > >>>> Bela Ban | http://www.jgroups.org > >>>> > >>>> > >>>> ------------------------------------------------------------------------------ > >>>> Check out the vibrant tech community on one of the world's most > >>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot > >>>> _______________________________________________ > >>>> javagroups-users mailing list > >>>> [email protected] > >>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users > >>> > >>> ------------------------------------------------------------------------------ > >>> Check out the vibrant tech community on one of the world's most > >>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot > >>> _______________________________________________ > >>> javagroups-users mailing list > >>> [email protected] > >>> https://lists.sourceforge.net/lists/listinfo/javagroups-users > >>> > >> > >> -- > >> Bela Ban | http://www.jgroups.org > >> > >> > >> ------------------------------------------------------------------------------ > >> Check out the vibrant tech community on one of the world's most > >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot > >> _______________________________________________ > >> javagroups-users mailing list > >> [email protected] > >> https://lists.sourceforge.net/lists/listinfo/javagroups-users > > > > ------------------------------------------------------------------------------ > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > _______________________________________________ > > javagroups-users mailing list > > [email protected] > > https://lists.sourceforge.net/lists/listinfo/javagroups-users > > > > -- > Bela Ban | http://www.jgroups.org > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > javagroups-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/javagroups-users ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot