RE: [Opendlm-devel] ODLM/OGFS Recovery

Comments below.

> -----Original Message-----
> From: [email protected]=20
> [mailto:[email protected]] On Behalf=20
> Of Cahill, Ben M
> Sent: Friday, April 23, 2004 12:41 PM
> To: [email protected]
> Cc: [email protected]
> Subject: RE: [Opendlm-devel] ODLM/OGFS Recovery
>=20
>=20
> =20
>=20
> > -----Original Message-----
> > From: [email protected]
> > [mailto:[email protected]] On Behalf=20
> > Of Stanley Wang
> > Sent: Friday, April 23, 2004 4:20 AM
> > To: [email protected]
> > Cc: [email protected]
> > Subject: Re: [Opendlm-devel] ODLM/OGFS Recovery
> >=20
> > Hi Ben,
> >=20
> > Thanks for your docs, it really helps a lot!
> >=20
> > Current solution in my mind:
> >=20
> > Once a node deads, all other nodes are notified by deadman
> > locks. After
> > being notified, every node's lock module hold all lock=20
> > request (put them in
> > a wait  queue) except NOEXP lock request.=20
>=20
> That's the basic idea I've been thinking about, also,=20
> (inspired by the work you've already done, and CA's=20
> comments), but I think there might be some problems with=20
> timing.  There's no guarantee of processing order when doing=20
> lock recovery.  The deadman lock will be granted during=20
> ODLM's lock recovery process, but might be granted after many=20
> other locks have been granted (and therefore mistakenly=20
> passed to OGFS).  That is, there's nothing special about the=20
> deadman lock (from OpenDLM's point of view) that would cause=20
> it to be processed first.
>=20
That is correct.

> CA uses the DLM_VALNOTVALID invalid LVB status as a possible=20
> earlier indicator.  I haven't convinced myself that that is=20
> early enough, or conclusive enough (discussion was in the=20
> attachment, "false positive/negative"), to serve as the=20
> trigger for withholding locks from OGFS (but it might be, and=20
> I might not understand it well enough!). Plus, it forces us=20

When a node dies I see three possible scenarios:
1)  a lock is granted withOUT the DLM_VALNOTVALID flag set
  - to me this indicates the master was/is on a live node, no possible =
data corruption occurred
2)  a lock is granted with the DLM_VALNOTVALID flag set
  - to me this indicates the master of that resource was forcefully =
removed (mostly likely died), the new master is now the current lock, =
and its data is probably corrupted.  Therefore rebuild from the journal.
3)  a deadman lock is granted.  Enough said, rebuild from the journal.

As for your false positive, this is scenario 2 described above.  If it =
accidentally marks the value block as invalid, then does it really =
matter?  It is playing better safe than sorry.  It is very difficult to =
determine what mode the lock on the dead node was in before it died.  =
Unless the locks are designed to be used a certain way such that they =
are never in EX/PW/CW, you can never really tell if you have a false =
positive or not.  Any notification is better than no notification.
As for your false negative, if no one knows about the resource, then yes =
the resource is destroyed.  However the scenario you propose seems =
unlikely.  Before the opendlm returns to the run state, I believe it =
grants all the newly acquired locks (in this case deadman locks).  Then =
it reaches the run state and processes new locks.  If those new locks =
are granted (say they are of the aforementioned resource) before the =
deadman locks, then I would think opendlm has some issues. =20
I could probably run some test cases early next week, as we are =
currently testing a 3/4 node cluster while pulling its power cord. =20


> to use LVBs with every lock (maybe not a huge problem, but=20
> nice to avoid if possible).  And, it prevents us from using=20
> the LKM_INVVALBLK flag!  (but I don't think there's a need=20
> to, so not a problem).

I don't know how OpenGFS works, but I thought there would a wrapper =
function around opendlm calls such that adding code for the lock value =
blocks would be quick and painless.

>=20
> The other tricky part is, if we withhold *all* locks from=20
> OpenGFS (not just the ones that were blocked by the dead=20


> node), will surviving nodes be able to finish their write=20
> transactions (and release their CR transaction locks)?  They=20

A write while holding a read lock??

> need to do this, or else the node that attempts the journal=20
> recovery will never get the (EX) transaction lock ....
>=20
>  .... Again (false positive/negative), how do we conclusively=20
> tell which locks were blocked by dead node vs. not?
>=20
>=20
> > After the node who
> > replays journal
> > completes its work, it notifies (though a dedicate=20
> "recover_complete"=20
> > lock?) all
> > the others that the blocked requests can continue now. And=20
> > then the OGFS=20
> > cluster
> > resumes.
>=20
> Yes, "recover_complete" locks might work.  The good thing=20
> about this is that each lock would be filesystem-specific (a=20
> concern if the user mounts multiple OGFS filesystems).  I=20
> think that each lock would need to be node- or=20
> journal-specific as well (multiple nodes might need to=20
> recover multiple journals before continuing operation).  I'd=20
> rather use locks than the callbacks, etc., ... I'll keep=20
> working on this.
>=20
> More comments/suggestions from anyone????

I was just curious how opengfs retrieved the memexp storage area of a =
dead node? =20
This seems counter-intuitive.  Unless every node had a copy of each =
others storage area. =20

Best Regards,
Don


>=20
> -- Ben --
>=20
> Opinions are mine, not Intel's
>=20
>=20
> >=20
> > Any comments?
> >=20
> > Best Regards,
> > Stan
> >=20
> > Cahill, Ben M wrote:
> >=20
> > >Hi all,
> > >
> > >Attached please find some discussion I'm trying to write to
> > understand
> > >the problems with OpenDLM/OpenGFS recovery after a node=20
> dies.  Memexp=20
> > >had this all integrated, but OpenDLM works quite differently.
> > >
> > >Please take a look and comment ... I'm hoping that I'm wrong
> > about some
> > >of the things I wrote, especially about DLM_VALNOTVALID.
> > Please give me
> > >a sanity check.
> > >
> > >I'm thinking about using a list within the lock module to
> > retain locks
> > >that get granted during lock recovery, but must not be=20
> forwarded to=20
> > >OpenGFS yet.
> > >
> > >There's more that I need to write, about timing and recovery=20
> > >notification ... Running the journal recovery after lock=20
> recovery is=20
> > >complete, then notifying all nodes when journal recovery is done.
> > >
> > >I'm thinking (just an idea) about using a callback from
> > OpenDLM to the
> > >lock module to indicate:
> > >
> > >-- when lock recovery begins (and which node died)
> > >-- when lock recovery ends
> > >-- when journal recovery ends (in response to a call from the lock=20
> > >module), perhaps using a new BARRIER state in the ODLM=20
> recovery state=20
> > >machine, to wait until all nodes' journal recovery ("client"
> > recovery,
> > >more generically) had occurred.
> > >
> > >Also wrestling with multiple filesystems ... Each must=20
> recover before=20
> > >moving on.  Memexp had separate instances and storage=20
> areas for each
> > >filesystem, but there's just one ODLM.   Arrgh.
> > >
> > >Stan's been thinking about this also, but I haven't captured his=20
> > >thoughts ... Please add/comment ... And anyone else, too.
> > >
> > >-- Ben --
> > >
> > >Opinions are mine, not Intel's
> > >
>=20
>=20
>=20
> -------------------------------------------------------
> This SF.net email is sponsored by: The Robotic Monkeys at=20
> ThinkGeek For a limited time only, get FREE Ground shipping=20
> on all orders of $35 or more. Hurry up and shop folks, this=20
> offer expires April 30th!=20
> http://www.thinkgeek.com/freeshipping/?cpg=12297
>=20
> _______________________________________________
> Opendlm-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/opendlm-devel
>=20
>=20


-------------------------------------------------------
This SF.net email is sponsored by: The Robotic Monkeys at ThinkGeek
For a limited time only, get FREE Ground shipping on all orders of $35
or more. Hurry up and shop folks, this offer expires April 30th!
http://www.thinkgeek.com/freeshipping/?cpg=12297