Re: [jgroups-users] JDBC - Cloning DB
Questions/problems related to using JGroups <[email protected]> Thu, 21 Jun 2018 14:38:23 +0200
| Newsgroups | gmane.comp.java.javagroups.general |
|---|---|
| Message-ID | <mailman.14046.1529584718.978.javagroups-users@lists.sourceforge.net> |
On 21/06/18 14:23, Questions/problems related to using JGroups wrote: > Hi Bela, > >> -----Original Message----- >> From: Questions/problems related to using JGroups [mailto:javagroups- >> [email protected]] >> Sent: 21 June 2018 12:52 >> To: [email protected] >> Subject: Re: [jgroups-users] JDBC - Cloning DB >> >> With all due respect, I don't think MERGE2 or MERGE3 is the root cause, >> it is only used to fix the root cause... > > Given my poor knowledge on the matter I can only agree. > > Since we have updated, for other reasons, to MERGE3 I can't see the log lines being printed out, in > fact the problem "does not manifest itself", which does not mean I have understood what the problem was. > If I were to understand exactly what happened where do you reckon I should be starting from ? I think the problem may have disappeared just by chance. I would start looing into whether that export/import procedure creates exactly the same DB table on both sides, and what happens when the table is updated. I still don't understand why you have 2 DBs instead of 1. And if you need this for HA, then I suggest to use the DB's replication mechanism to replicate your data, and not a self-made and brittle export/import mechanism. > P2 writes: >>>>> 17:58:00.913 WARN OOB-2,LT1038-21441 org.jgroups.protocols.TCP >> xxxxxx: >>>> no physical address for 0eece642-4813-bb4a-934f-46d886cad55b, dropping >>>> message > > What could be the source of that message, I can only speculate :-( > > Maybe when p1 and p2 have started p2 establishes a TCP connection to p1 - as the JGroupsPing table is copied across, by mistake - and as a result of that > includes it among its cluster member and sending messages to it ? > > > Thanks, > P. > > >> >> On 21/06/18 12:55, Questions/problems related to using JGroups wrote: >>> Hi Bela, >>> >>> And indeed the "problem" manifest itself only if MERGE2 is used, while it >> disappears when upgrading to MERGE3. >>> >>> Thanks (again) >>> P. >>> >>> >>>> -----Original Message----- >>>> From: Questions/problems related to using JGroups [mailto:javagroups- >>>> [email protected]] >>>> Sent: 20 June 2018 15:28 >>>> To: [email protected] >>>> Subject: Re: [jgroups-users] JDBC - Cloning DB >>>> >>>> The config you show below suggest you're using an old version of JGroups >>>> (e.g. UNICAST2, MuxRequestCorrelator, TCP's shared transport etc), can >>>> you try this with the latest 4.x or 3.6.x release? >>>> >>>> Comments below >>>> >>>> On 19/06/18 19:21, Questions/problems related to using JGroups wrote: >>>>> Hello, >>>>> >>>>> The application I am working with makes use of JDBC_PING for discovery >>>> and I've recently stumbled upon a scenario which involves the following >> stack >>>> configuration : >>>>> >>>>> <config xmlns="urn:org:jgroups" >>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >>>>> xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-3.2.xsd"> >>>>> <TCP [..] >>>>> /> >>>>> >>>>> <JDBC_PING connection_url="jdbc:mysql://localhost:3306/jgroups" >>>>> connection_username="test" >>>>> connection_password="test" >>>>> connection_driver="com.mysql.jdbc.Driver" >>>>> /> >>>>> <MERGE2 max_interval="30000" min_interval="10000"/> >>>>> <FD_SOCK/> >>>>> <FD timeout="3000" max_tries="3"/> >>>>> <VERIFY_SUSPECT timeout="1500"/> >>>>> <pbcast.NAKACK2 use_mcast_xmit="false" >>>>> [...] >>>>> /> >>>>> <UNICAST2 stable_interval="5000" >>>>> [..] >>>>> /> >>>>> <pbcast.STABLE stability_delay="500" desired_avg_gossip="5000" >>>> max_bytes="1m"/> >>>>> <pbcast.GMS print_local_addr="false" join_timeout="3000" >>>> view_bundling="true"/> >>>>> <UFC max_credits="200k" min_threshold="0.20"/> >>>>> <MFC max_credits="200k" min_threshold="0.20"/> >>>>> <FRAG2 frag_size="60000"/> >>>>> <RSVP timeout="60000" resend_interval="500" ack_on_delivery="false" >> /> >>>>> </config> >>>>> >>>>> There two DB, db1 and db2 >>>> >>>> >>>> >>>> Why two databases? The point of JDBC_PING is to have a common DBMS >>>> which >>>> all members access... >>>> >>>> >>>> >>>>> and two processes p1 and p2. P1 uses db1 and p2 db2, they all are into >> the >>>> same subnet and they both share the same code, for instance >>>>> they both create a channel with the same name. >>>> >>>> Again, are those 2 DBs in sync, so when P1 inserts its data into DB1, >>>> will P2 read the same data out of DB2? >>>> >>>>> P1 starts up and it becomes a single node cluster, thereafter, for any >>>> reason, the whole db1 is exported and imported into >>>>> db2 thus causing the JDBC_PING table to be the same as p1 when p2 >> starts. >>>> >>>> OK. Any reason for this? What if P1 and P2 start at the same time? >>>> >>>> Or what happens if P2 updates DB2? Will the update be exported back to >>>> DB1? What if P1 updates DB1 at *exactly the same time*? Exporting and >>>> (probably) overwriting a DB's state seems very brittle IMO... >>>> >>>> >>>>> The issue I am trying to get at the bottom of is that p2 keeps printing : >>>>> >>>>> 17:58:00.913 WARN OOB-2,LT1038-21441 org.jgroups.protocols.TCP >> xxxxxx: >>>> no physical address for 0eece642-4813-bb4a-934f-46d886cad55b, dropping >>>> message >>>> >>>> >>>> This means that P2 probably wanted to send a message to P1 and didn't >>>> find P1's IP address in DB2. DB2 should contain IP addresses for both P1 >>>> and P2. You should enable tracing on JDBC_PING to see what's going on, >>>> or dump the contents of each DB when starting P2. >>>> >>>> >>>>> ---------------- TRACE (received) ---------------------- >>>>> Event[type=MSG, arg=[dst: <null>, src: xxxxx (4 headers), size=1273 bytes]] >>>> (headers=MuxRequestCorrelator: MuxHeader(29318), RequestCorrelator: >>>> id=200, type=REQ, id=162, rsp_expected=false, NAKACK2: [MSG, >> seqno=161], >>>> TCP: [channel_name=ControlChannel]) >>>>> -------------------------------------------------------- >>>>> ---------------- TRACE (received) ---------------------- >>>>> Event[type=MSG, arg=[dst: <null>, src: xxxxx (4 headers), size=1273 bytes]] >>>> (headers=MuxRequestCorrelator: MuxHeader(29318), RequestCorrelator: >>>> id=200, type=REQ, id=162, rsp_expected=false, NAKACK2: [MSG, >> seqno=161], >>>> TCP: [channel_name=ControlChannel]) >>>>> -------------------------------------------------------- >>>> >>>> Do you need muxing? I assume you use a shared transport (in TCP)? This >>>> was deprecated and removed a long time ago, in favor of FORK. >>>> >>>> >>>>> ---------------- TRACE (received) ---------------------- >>>>> Event[type=IS_MERGE_IN_PROGRESS, arg=null] >>>>> -------------------------------------------------------- >>>>> ---------------- TRACE (received) ---------------------- >>>>> Event[type=IS_MERGE_IN_PROGRESS, arg=null] >>>>> -------------------------------------------------------- >>>>> >>>>> The two tables hold, no surprise, a very different state of the cluster and >> p2 >>>> is indeed receiving a message from somebody which is not part of its table. >>>> >>>> Why do the tables hold different state? They should hold exactly the >>>> same state, or else you're export/import function didn't function correctly. >>>> >>>> I suggest try this with a single DB, and both P1 and P2 point to it. >>>> >>>>> I am not quite sure my reading of the situation is correct, it seems that p2 >>>> connects to p1 to notify its existence and thereafter p1, which is the >>>> coordinator >>>>> tries to perform a merge. >>>> >>>> Right: P2 starts up and becomes singleton because it cannot send a JOIN >>>> request message to P1. So we have 2 clusters {P1} and {P2} and then they >>>> try to merge. But this will probably fails, too, because P1 and P2 >>>> cannot communicate... >>>> >>>>> Do you have any suggestion on how I could further debug what happened >> ? >>>> >>>> 1: use a more recent version of JGroups >>>> 2: use a single DB >>>> >>>>> I would like to trace down the JOINS() being sent to the p1 to make sure >> the >>>> above statement is correct, and then I would like to work >>>>> out if there is any problem with the stack's configuration or if the >> problem is >>>> un-avoidable as the root of the rub is the DB being copied across. >>>>> >>>>> I am used to debug the view by setting a custom/dummy receiver which >>>> would merely print the view without doing anything, I am a bit wary of >> doing >>>> that in my application >>>>> and I am wondering if it is possible to programmatically enable the >> logging >>>> across all the stack so to have a whole view or what's going on without >>>> impacting the stack's >>>>> "functional" configuration. >>>> >>>> You could use probe.sh, e.g. probe.sh jmx=JDBC_PING.level="trace" >>>> >>>>> Thanks, >>>>> P. >>>>> >>>>> >>>>> >>>>> Pietro Paolini >>>>> Consultant >>>>> >>>>> Alfa >>>>> ________________________________ >>>>> e: [email protected] | w: >>>> alfasystems.com<https://www.alfasystems.com> >>>>> t: +44 (0) 20 7920-2643 | Moor Place, 1 Fore Street Avenue, London, EC2Y >>>> 9DT, GB >>>>> ________________________________ >>>>> >>>>> The contents of this communication are not intended to be binding or >>>> constitute any form of offer or acceptance or give rise to any legal >> obligations >>>> on behalf of the sender or Alfa. The views or opinions expressed represent >>>> those of the author and not necessarily those of Alfa. This email and any >>>> attachments are strictly confidential and are intended solely for use by the >>>> individual or entity to whom it is addressed. If you are not the addressee >> (or >>>> responsible for delivery of the message to the addressee) you may not >> copy, >>>> forward, disclose or use any part of the message or its attachments. At >>>> present the integrity of email across the internet cannot be guaranteed >> and >>>> messages sent via this medium are potentially at risk. All liability is >> excluded >>>> to the extent permitted by law for any claims arising as a result of the use >> of >>>> this medium to transmit information by or to Alfa or its affiliates. >>>>> >>>>> Alfa Financial Software Ltd >>>>> Reg. in England No: 0248 2325 >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> Check out the vibrant tech community on one of the world's most >>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >>>>> _______________________________________________ >>>>> javagroups-users mailing list >>>>> [email protected] >>>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users >>>>> >>>> >>>> -- >>>> Bela Ban | http://www.jgroups.org >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> Check out the vibrant tech community on one of the world's most >>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >>>> _______________________________________________ >>>> javagroups-users mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/javagroups-users >>> >>> ------------------------------------------------------------------------------ >>> Check out the vibrant tech community on one of the world's most >>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >>> _______________________________________________ >>> javagroups-users mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/javagroups-users >>> >> >> -- >> Bela Ban | http://www.jgroups.org >> >> >> ------------------------------------------------------------------------------ >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> _______________________________________________ >> javagroups-users mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/javagroups-users > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > javagroups-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/javagroups-users > -- Bela Ban | http://www.jgroups.org ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot