Re: [jgroups-users] JDBC - Cloning DB

Hi Bela,

> -----Original Message-----
> From: Questions/problems related to using JGroups [mailto:javagroups-
> [email protected]]
> Sent: 21 June 2018 12:52
> To: [email protected]
> Subject: Re: [jgroups-users] JDBC - Cloning DB
> 
> With all due respect, I don't think MERGE2 or MERGE3 is the root cause,
> it is only used to fix the root cause...

Given my poor knowledge on the matter I can only agree.

Since we have updated, for other reasons, to MERGE3 I can't see the log lines being printed out, in 
fact the problem "does not manifest itself", which does not mean I have understood what the problem was.
If I were to understand exactly what happened where do you reckon I should be starting from ?

P2 writes:
> >>> 17:58:00.913 WARN OOB-2,LT1038-21441 org.jgroups.protocols.TCP
> xxxxxx:
> >> no physical address for 0eece642-4813-bb4a-934f-46d886cad55b, dropping
> >> message

What could be the source of that message, I can only speculate :-(

Maybe when p1 and p2 have started p2 establishes a TCP connection to p1 - as the JGroupsPing table is copied across, by mistake -  and as a result of that 
includes it among its cluster member and sending messages to it ?


Thanks,
P.


> 
> On 21/06/18 12:55, Questions/problems related to using JGroups wrote:
> > Hi Bela,
> >
> > And indeed the "problem" manifest itself only if MERGE2 is used, while it
> disappears when upgrading to MERGE3.
> >
> > Thanks (again)
> > P.
> >
> >
> >> -----Original Message-----
> >> From: Questions/problems related to using JGroups [mailto:javagroups-
> >> [email protected]]
> >> Sent: 20 June 2018 15:28
> >> To: [email protected]
> >> Subject: Re: [jgroups-users] JDBC - Cloning DB
> >>
> >> The config you show below suggest you're using an old version of JGroups
> >> (e.g. UNICAST2, MuxRequestCorrelator, TCP's shared transport etc), can
> >> you try this with the latest 4.x or 3.6.x release?
> >>
> >> Comments below
> >>
> >> On 19/06/18 19:21, Questions/problems related to using JGroups wrote:
> >>> Hello,
> >>>
> >>> The application I am working with makes use of JDBC_PING for discovery
> >> and I've recently stumbled upon a scenario which involves the following
> stack
> >> configuration :
> >>>
> >>> <config xmlns="urn:org:jgroups"
> >>>           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> >>>           xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-3.2.xsd">
> >>>      <TCP    [..]
> >>>      />
> >>>
> >>>      <JDBC_PING connection_url="jdbc:mysql://localhost:3306/jgroups"
> >>>                 connection_username="test"
> >>>                 connection_password="test"
> >>>                 connection_driver="com.mysql.jdbc.Driver"
> >>>      />
> >>>      <MERGE2 max_interval="30000" min_interval="10000"/>
> >>>      <FD_SOCK/>
> >>>      <FD timeout="3000" max_tries="3"/>
> >>>      <VERIFY_SUSPECT timeout="1500"/>
> >>>      <pbcast.NAKACK2 use_mcast_xmit="false"
> >>>                     [...]
> >>>      />
> >>>      <UNICAST2 stable_interval="5000"
> >>>                [..]
> >>>      />
> >>>      <pbcast.STABLE stability_delay="500" desired_avg_gossip="5000"
> >> max_bytes="1m"/>
> >>>      <pbcast.GMS print_local_addr="false" join_timeout="3000"
> >> view_bundling="true"/>
> >>>      <UFC max_credits="200k" min_threshold="0.20"/>
> >>>      <MFC max_credits="200k" min_threshold="0.20"/>
> >>>      <FRAG2 frag_size="60000"/>
> >>>      <RSVP timeout="60000" resend_interval="500" ack_on_delivery="false"
> />
> >>> </config>
> >>>
> >>> There two DB, db1 and db2
> >>
> >>
> >>
> >> Why two databases? The point of JDBC_PING is to have a common DBMS
> >> which
> >> all members access...
> >>
> >>
> >>
> >>> and two processes p1 and p2. P1 uses db1 and p2 db2, they all are into
> the
> >> same subnet and they both share the same code, for instance
> >>> they both create a channel with the same name.
> >>
> >> Again, are those 2 DBs in sync, so when P1 inserts its data into DB1,
> >> will P2 read the same data out of DB2?
> >>
> >>> P1 starts up and it becomes a single node cluster,  thereafter, for any
> >> reason, the whole db1 is exported and imported into
> >>> db2 thus causing the JDBC_PING table to be the same as p1 when p2
> starts.
> >>
> >> OK. Any reason for this? What if P1 and P2 start at the same time?
> >>
> >> Or what happens if P2 updates DB2? Will the update be exported back to
> >> DB1? What if P1 updates DB1 at *exactly the same time*? Exporting and
> >> (probably) overwriting a DB's state seems very brittle IMO...
> >>
> >>
> >>> The issue I am trying to get at the bottom of is that p2 keeps printing :
> >>>
> >>> 17:58:00.913 WARN OOB-2,LT1038-21441 org.jgroups.protocols.TCP
> xxxxxx:
> >> no physical address for 0eece642-4813-bb4a-934f-46d886cad55b, dropping
> >> message
> >>
> >>
> >> This means that P2 probably wanted to send a message to P1 and didn't
> >> find P1's IP address in DB2. DB2 should contain IP addresses for both P1
> >> and P2. You should enable tracing on JDBC_PING to see what's going on,
> >> or dump the contents of each DB when starting P2.
> >>
> >>
> >>> ---------------- TRACE (received) ----------------------
> >>> Event[type=MSG, arg=[dst: <null>, src: xxxxx (4 headers), size=1273 bytes]]
> >> (headers=MuxRequestCorrelator: MuxHeader(29318), RequestCorrelator:
> >> id=200, type=REQ, id=162, rsp_expected=false, NAKACK2: [MSG,
> seqno=161],
> >> TCP: [channel_name=ControlChannel])
> >>> --------------------------------------------------------
> >>> ---------------- TRACE (received) ----------------------
> >>> Event[type=MSG, arg=[dst: <null>, src: xxxxx (4 headers), size=1273 bytes]]
> >> (headers=MuxRequestCorrelator: MuxHeader(29318), RequestCorrelator:
> >> id=200, type=REQ, id=162, rsp_expected=false, NAKACK2: [MSG,
> seqno=161],
> >> TCP: [channel_name=ControlChannel])
> >>> --------------------------------------------------------
> >>
> >> Do you need muxing? I assume you use a shared transport (in TCP)? This
> >> was deprecated and removed a long time ago, in favor of FORK.
> >>
> >>
> >>> ---------------- TRACE (received) ----------------------
> >>> Event[type=IS_MERGE_IN_PROGRESS, arg=null]
> >>> --------------------------------------------------------
> >>> ---------------- TRACE (received) ----------------------
> >>> Event[type=IS_MERGE_IN_PROGRESS, arg=null]
> >>> --------------------------------------------------------
> >>>
> >>> The two tables hold, no surprise, a very different state of the cluster and
> p2
> >> is indeed receiving a message from somebody which is not part of its table.
> >>
> >> Why do the tables hold different state? They should hold exactly the
> >> same state, or else you're export/import function didn't function correctly.
> >>
> >> I suggest try this with a single DB, and both P1 and P2 point to it.
> >>
> >>> I am not quite sure my reading of the situation is correct, it seems that p2
> >> connects to p1 to notify its existence and thereafter p1, which is the
> >> coordinator
> >>> tries to perform a merge.
> >>
> >> Right: P2 starts up and becomes singleton because it cannot send a JOIN
> >> request message to P1. So we have 2 clusters {P1} and {P2} and then they
> >> try to merge. But this will probably fails, too, because P1 and P2
> >> cannot communicate...
> >>
> >>> Do you have any suggestion on how I could further debug what happened
> ?
> >>
> >> 1: use a more recent version of JGroups
> >> 2: use a single DB
> >>
> >>> I would like to trace down the JOINS() being sent to the p1 to make sure
> the
> >> above statement is correct, and then I would like to work
> >>> out if there is any problem with the stack's configuration or if the
> problem is
> >> un-avoidable as the root of the rub is the DB being copied across.
> >>>
> >>> I am used to debug the view by setting a custom/dummy receiver which
> >> would merely print the view without doing anything, I am a bit wary of
> doing
> >> that in my application
> >>> and I am wondering if it is possible to programmatically enable the
> logging
> >> across all the stack so to have a whole view or what's going on without
> >> impacting the stack's
> >>> "functional" configuration.
> >>
> >> You could use probe.sh, e.g. probe.sh jmx=JDBC_PING.level="trace"
> >>
> >>> Thanks,
> >>> P.
> >>>
> >>>
> >>>
> >>> Pietro Paolini
> >>> Consultant
> >>>
> >>> Alfa
> >>> ________________________________
> >>> e: [email protected] | w:
> >> alfasystems.com<https://www.alfasystems.com>
> >>> t: +44 (0) 20 7920-2643 | Moor Place, 1 Fore Street Avenue, London, EC2Y
> >> 9DT, GB
> >>> ________________________________
> >>>
> >>> The contents of this communication are not intended to be binding or
> >> constitute any form of offer or acceptance or give rise to any legal
> obligations
> >> on behalf of the sender or Alfa. The views or opinions expressed represent
> >> those of the author and not necessarily those of Alfa. This email and any
> >> attachments are strictly confidential and are intended solely for use by the
> >> individual or entity to whom it is addressed. If you are not the addressee
> (or
> >> responsible for delivery of the message to the addressee) you may not
> copy,
> >> forward, disclose or use any part of the message or its attachments. At
> >> present the integrity of email across the internet cannot be guaranteed
> and
> >> messages sent via this medium are potentially at risk. All liability is
> excluded
> >> to the extent permitted by law for any claims arising as a result of the use
> of
> >> this medium to transmit information by or to Alfa or its affiliates.
> >>>
> >>> Alfa Financial Software Ltd
> >>> Reg. in England No: 0248 2325
> >>>
> >>> ------------------------------------------------------------------------------
> >>> Check out the vibrant tech community on one of the world's most
> >>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> >>> _______________________________________________
> >>> javagroups-users mailing list
> >>> [email protected]
> >>> https://lists.sourceforge.net/lists/listinfo/javagroups-users
> >>>
> >>
> >> --
> >> Bela Ban | http://www.jgroups.org
> >>
> >>
> >> ------------------------------------------------------------------------------
> >> Check out the vibrant tech community on one of the world's most
> >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> >> _______________________________________________
> >> javagroups-users mailing list
> >> [email protected]
> >> https://lists.sourceforge.net/lists/listinfo/javagroups-users
> >
> > ------------------------------------------------------------------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> > _______________________________________________
> > javagroups-users mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/javagroups-users
> >
> 
> --
> Bela Ban | http://www.jgroups.org
> 
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot