Re: [jgroups-users] JDBC - Cloning DB

The config you show below suggest you're using an old version of JGroups 
(e.g. UNICAST2, MuxRequestCorrelator, TCP's shared transport etc), can 
you try this with the latest 4.x or 3.6.x release?

Comments below

On 19/06/18 19:21, Questions/problems related to using JGroups wrote:
> Hello,
> 
> The application I am working with makes use of JDBC_PING for discovery and I've recently stumbled upon a scenario which involves the following stack configuration :
> 
> <config xmlns="urn:org:jgroups"
>          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>          xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-3.2.xsd">
>     <TCP    [..]
>     />
> 
>     <JDBC_PING connection_url="jdbc:mysql://localhost:3306/jgroups"
>                connection_username="test"
>                connection_password="test"
>                connection_driver="com.mysql.jdbc.Driver"
>     />
>     <MERGE2 max_interval="30000" min_interval="10000"/>
>     <FD_SOCK/>
>     <FD timeout="3000" max_tries="3"/>
>     <VERIFY_SUSPECT timeout="1500"/>
>     <pbcast.NAKACK2 use_mcast_xmit="false"
>                    [...]
>     />
>     <UNICAST2 stable_interval="5000"
>               [..]
>     />
>     <pbcast.STABLE stability_delay="500" desired_avg_gossip="5000" max_bytes="1m"/>
>     <pbcast.GMS print_local_addr="false" join_timeout="3000" view_bundling="true"/>
>     <UFC max_credits="200k" min_threshold="0.20"/>
>     <MFC max_credits="200k" min_threshold="0.20"/>
>     <FRAG2 frag_size="60000"/>
>     <RSVP timeout="60000" resend_interval="500" ack_on_delivery="false" />
> </config>
> 
> There two DB, db1 and db2



Why two databases? The point of JDBC_PING is to have a common DBMS which 
all members access...



> and two processes p1 and p2. P1 uses db1 and p2 db2, they all are into the same subnet and they both share the same code, for instance
> they both create a channel with the same name.

Again, are those 2 DBs in sync, so when P1 inserts its data into DB1, 
will P2 read the same data out of DB2?

> P1 starts up and it becomes a single node cluster,  thereafter, for any reason, the whole db1 is exported and imported into
> db2 thus causing the JDBC_PING table to be the same as p1 when p2 starts.

OK. Any reason for this? What if P1 and P2 start at the same time?

Or what happens if P2 updates DB2? Will the update be exported back to 
DB1? What if P1 updates DB1 at *exactly the same time*? Exporting and 
(probably) overwriting a DB's state seems very brittle IMO...


> The issue I am trying to get at the bottom of is that p2 keeps printing :
> 
> 17:58:00.913 WARN OOB-2,LT1038-21441 org.jgroups.protocols.TCP xxxxxx: no physical address for 0eece642-4813-bb4a-934f-46d886cad55b, dropping message


This means that P2 probably wanted to send a message to P1 and didn't 
find P1's IP address in DB2. DB2 should contain IP addresses for both P1 
and P2. You should enable tracing on JDBC_PING to see what's going on, 
or dump the contents of each DB when starting P2.


> ---------------- TRACE (received) ----------------------
> Event[type=MSG, arg=[dst: <null>, src: xxxxx (4 headers), size=1273 bytes]] (headers=MuxRequestCorrelator: MuxHeader(29318), RequestCorrelator: id=200, type=REQ, id=162, rsp_expected=false, NAKACK2: [MSG, seqno=161], TCP: [channel_name=ControlChannel])
> --------------------------------------------------------
> ---------------- TRACE (received) ----------------------
> Event[type=MSG, arg=[dst: <null>, src: xxxxx (4 headers), size=1273 bytes]] (headers=MuxRequestCorrelator: MuxHeader(29318), RequestCorrelator: id=200, type=REQ, id=162, rsp_expected=false, NAKACK2: [MSG, seqno=161], TCP: [channel_name=ControlChannel])
> --------------------------------------------------------

Do you need muxing? I assume you use a shared transport (in TCP)? This 
was deprecated and removed a long time ago, in favor of FORK.


> ---------------- TRACE (received) ----------------------
> Event[type=IS_MERGE_IN_PROGRESS, arg=null]
> --------------------------------------------------------
> ---------------- TRACE (received) ----------------------
> Event[type=IS_MERGE_IN_PROGRESS, arg=null]
> --------------------------------------------------------
> 
> The two tables hold, no surprise, a very different state of the cluster and p2 is indeed receiving a message from somebody which is not part of its table.

Why do the tables hold different state? They should hold exactly the 
same state, or else you're export/import function didn't function correctly.

I suggest try this with a single DB, and both P1 and P2 point to it.

> I am not quite sure my reading of the situation is correct, it seems that p2 connects to p1 to notify its existence and thereafter p1, which is the coordinator
> tries to perform a merge.

Right: P2 starts up and becomes singleton because it cannot send a JOIN 
request message to P1. So we have 2 clusters {P1} and {P2} and then they 
try to merge. But this will probably fails, too, because P1 and P2 
cannot communicate...

> Do you have any suggestion on how I could further debug what happened ?

1: use a more recent version of JGroups
2: use a single DB

> I would like to trace down the JOINS() being sent to the p1 to make sure the above statement is correct, and then I would like to work
> out if there is any problem with the stack's configuration or if the problem is un-avoidable as the root of the rub is the DB being copied across.
> 
> I am used to debug the view by setting a custom/dummy receiver which would merely print the view without doing anything, I am a bit wary of doing that in my application
> and I am wondering if it is possible to programmatically enable the logging across all the stack so to have a whole view or what's going on without impacting the stack's
> "functional" configuration.

You could use probe.sh, e.g. probe.sh jmx=JDBC_PING.level="trace"

> Thanks,
> P.
> 
> 
> 
> Pietro Paolini
> Consultant
> 
> Alfa
> ________________________________
> e: [email protected] | w: alfasystems.com<https://www.alfasystems.com>
> t: +44 (0) 20 7920-2643 | Moor Place, 1 Fore Street Avenue, London, EC2Y 9DT, GB
> ________________________________
> 
> The contents of this communication are not intended to be binding or constitute any form of offer or acceptance or give rise to any legal obligations on behalf of the sender or Alfa. The views or opinions expressed represent those of the author and not necessarily those of Alfa. This email and any attachments are strictly confidential and are intended solely for use by the individual or entity to whom it is addressed. If you are not the addressee (or responsible for delivery of the message to the addressee) you may not copy, forward, disclose or use any part of the message or its attachments. At present the integrity of email across the internet cannot be guaranteed and messages sent via this medium are potentially at risk. All liability is excluded to the extent permitted by law for any
  claims arising as a result of the use of this medium to transmit information by or to Alfa or its affiliates.
> 
> Alfa Financial Software Ltd
> Reg. in England No: 0248 2325
> 
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> javagroups-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/javagroups-users
> 

-- 
Bela Ban | http://www.jgroups.org


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot