Re: Several questions about enbd

Michael Ulitskiy <[email protected]> wrote:
> I need a data mirroring solution for some high-available setups. 
> I've made some research and found 2 packages that seem to be 
> a viable option as they are in use and actively developed - ENBD and DRBD. 
> So far I would favor ENBD over DRBD as it allows to have several mirrors
> from one master data (one client RAID-1 - multiple servers). Also it seems
> to me that using standard RAID driver is more promising and can potentially
> allow to do very interesting things.

If you are going to use rai1 you should want the fr1 version (see
freshmeat and places ...) which adapts it for enbd in particular. Ditto
fr5 (very experimental).

> I'm still at the stage of studying, but there're a few questions that concerns
> me:
> 
> 1. are there any precautions taken to ensure that resyncs in networked RAID 
> will be done from a node that has most up-to-date information?

That's up to raid1. Resyncs consist of a read-1 followed by a write-all
(as far as I recall). The choice of which device to read from in
classical raid1 is made by effectively switching devices round-robin
every 128 blocks.

In 2.6 kernels the choice is made by minimising over the head movement
required (silly, in my opinion, since we don't really know where the
heads are, so the correlation is statistical, but never mind).

However, in all kernels, all devices are equally up to date. They have
to be!

In fr1 the disk that responds fastest is always chosen. But in fr1
resyncs are only of non-uptodate blocks, not all blocks (unless, of
course, all blocks are not up to date, as in the case of a new disk).


> DRBD has a
> well-defined algorithm to determine the source of up-to-date information 

In raid1 all devices are up to date.

> described here http://www.drbd.org/fileadmin/drbd/publications/drbd_paper_for_NLUUG_2001.pdf
> (meta-data section). What about ENBD? What will happen in worst-case 
> scenario described in the paper (client crashes, enbd-server becomes primary 
> and have modified data, then the whole cluster restarts.)?

Oh, you are talking about failover? It depends how you do failover.  I
would usually prefer that nothing inverts again when the original comes
back up, but you can switch back again if you choose.

You'd have to describe the scenario in detail for me to comment.


> I can't imagine how
> would raid1 know that it needs to resync from server to client unless ENBD
> takes some special precautions. In other words whatever ENBD does is it
> enough to reasonably guarantee data integrity?

I don't yet understand the question exactly enough to comment. It seems to
me that you are talking about the failover mechanisms used, not enbd!


> 2. this line of nbd-2.4.31/nbd/etc/ha.d/resource.d/enbd-primary scares the heck

Ah - you are looking at the provided heartbeat scripts?


> out of me:
> primaryside_client_start () {
>     ...
>    mkraid --really-force --dangerous-no-resync $MD
>     ...
> }

I don't recall the scripts in detail. Fell free to improve them! That
looks like it just assembles the raid without doing a sync. 


> especially because man mkraid says: "Note that initializing RAID devices destroys  
> all  of  the data on the consituent devices".

No it doesn't. The call is with no-resync.

> Is it really necessary to recreate raid
> superblock on each client start?

No, nothing is necessary. But I don't think I wrote those scripts!
Certainly mkraid with the no-resync just assembles.

Why not convert the scripts to use mdadm?  Then I might be able to
comment more.


> 3. Is it possible to mount read-only and access data on the enbd-server? DRBD FAQ says
> "most likely your filesystem will be confused because it will not be aware about 
> changes in the underlying device". Is it also true for ENBD?

You can always mount the fs ro and read what was on it at the time of
the mount. But you will not see any updates because they don't pass
through the kernel fs cache, and yoru fs will rapidly become very
confused as things change underneath it. I imagine the kernel would bug
out in short order if you continue writing to it. Depend on what you
do.


> 4. Because both local drive and nbd device are made into RAID-1 is there a way
> to ensure that data reads will be done locally and not from the network.

Using fr1, reads come from the fastest device.

> AFAIK md driver will distribute reads over all device and having one of them
> networked can have serious performance implications.

True.

Peter