Re: Several questions about enbd

Michael Ulitskiy <[email protected]> wrote:
> I understand that all devices are up-to-date under the normal working conditions.
> I'm talking about high-availability scenario with fail-over. The question is how would

Yes - there are heartbeat failover scripts distributed with enbd, but
be aware that I'm only "uncle" to them.  I probably did write some
scripts originally but then others wrote their definitive versions, and
I collected them in the distro ...

So feel free to improve them!

Certainly which "end" is uptodate is a question for the failover
algorithm (heartbeat? Plus scripts?) to decide. 

> networked RAID determine which device is the source for resync, i.e. which one 
> has most up-to-date data.
> Let's take an example provided in DRBD paper assuming we have 2 nodes. Anyway
> this is configuration I'm going to start with.
> 1. enbd-client (primary) fails and we switch to enbd-server (secondary), i.e. we stop 

Hmmm. OK. We were writing into a mirror, one of whose partitions was
local and the other hosted remotely, via enbd. Then the mirror dies and
we have to start using the remote ...

> the enbd-server, mount partition rw etc.
> 2. data modified on the secondary

Well, now you start using this reserve server in earnest ...

> 3. cluster is restarted, i.e. cluster manager lost information about nodes state (like power

I don't follow this. I think you are saying that all the involved nodes go
down. Your question now is how we can tell who was the operative server
when we went down, so that we can restart in that same configuration?

This is the startup problem. As far as I know it's not solved in raid1,
so you would want to solve it there before doing so elsewhere.

But if you want to know which end of the pair was serving last, just
pass a token in the heartbeat config. Let it put down a marker on the
side that is serving.

(yeah, I'm sure the existing scripts don't do that - so add it).

> outage on both nodes or manual cluster manager restart etc.)
> 4. both enbd-client (primary) and enbd-server(secondary) comes up and enbd-client starting
> raid. At this point raid has to figure out whether the resync direction - from client to server 

Well, no, raid doesn't, YOU do. You want to start things up the way you
want them to be. You can force the direction of the resync as you like.
But if you trust raid1 (and I don't see why you should, given that I
just gone tole' you that it's only approximately correct), you would
leave it to raid1 to decide.

In more detail, rai1 writes an "event" count into the superblocks every
so often. If one event count is higher than the other, it knows.

The problem is that it doesn't write the event count at every request.

But that shoouldn't matter, since the reasion it would have been doing
a resync (if interrupted in resync last time) would have been because
the event counts were imbalanced in one direction.

And if the all-node crash did not happen during a resync, then the
event counts would have been equal, so you don't care.

Can the event counts be equal and the mirrors be unbalanced? Well, the
user would never have got an ack until both mirrors were written. But
yes, it can happen that only one mirror is written and the other not,
and the user is not acked, but then on restart the wrong mirror is
chosen and the write is seen as registered even though it was never
acked.

That's just raid1.

I guess you can run a (metadata only) journal fs in order to help cut
that error down.


> or backwards. DRBD has provisioning for this in its meta-data. If enbd makes no 
> provisioning for this, I guess resync will always be done from client to server and
> all modifications made on the server will be lost.

It'll be done the way you want it done. I don't see that there is any
problem in distinguishing which end was serving when all nodes went
down. The problem is other, and is connected with a setup in which the 
primary is serving and mirroring to the secondary. In principle the
secondary could be slightly more recent that the primary at times, but
that will never happen in practice.  So when the nodes come back up
and the primary was serving befre, sync from primary to secondary!

Care to improve the scripts?


> > > especially because man mkraid says: "Note that initializing RAID devices destroys  
> > > all  of  the data on the consituent devices".
> > 
> > No it doesn't. The call is with no-resync.
> > 
> > > Is it really necessary to recreate raid
> > > superblock on each client start?
> > 
> > No, nothing is necessary. But I don't think I wrote those scripts!
> > Certainly mkraid with the no-resync just assembles.
> 
> Ok. What's the purpose of reassemble? Wouldn't just raidstart be enough?

I am afraid you are currently more familiar with the scripts than I am.
I would have to go look, and it is late. I'll answer over the w/e.


> > Why not convert the scripts to use mdadm?  Then I might be able to
> > comment more.
>  
> I'm probably too lazy to move to mdadm. I'm still using raidtools. I'm used to them.

mdadm really is much more suitable. I'll do the script conversion.


> > > 3. Is it possible to mount read-only and access data on the enbd-server? DRBD FAQ says
> > > "most likely your filesystem will be confused because it will not be aware about 
> > > changes in the underlying device". Is it also true for ENBD?
> > 
> > You can always mount the fs ro and read what was on it at the time of
> > the mount. But you will not see any updates because they don't pass
> > through the kernel fs cache, and yoru fs will rapidly become very
> > confused as things change underneath it. I imagine the kernel would bug
> > out in short order if you continue writing to it. Depend on what you
> > do.
>  
> Ok. So it's the same as with DRBD. Too bad :(

It' nothing to do with either of us. It's a fact about the kernel. File
systems have a cache and so if you change the disk underneath while the
cache contents stay the same, the fs will become very confused!

Would would be nice would be a noncaching filesystem l-).

>
> > > 4. Because both local drive and nbd device are made into RAID-1 is there a way
> > > to ensure that data reads will be done locally and not from the network.
> > 
> > Using fr1, reads come from the fastest device.
> > 
> > > AFAIK md driver will distribute reads over all device and having one of them
> > > networked can have serious performance implications.
> > 
> > True.
> 
> So it's a problem with standard kernel raid. Is there a plan to make a way to enforce
> it, like device priority or something?

Well, you can use the fr1 patches - they cure things like that. The
patch to change wihich read device is used should be standalone in the
distribution, as I recall.


Peter