Re: [RFC] ext3 multi-mount protection

On Jul 15, 2006  19:57 -0400, Theodore Tso wrote:
> Like Stephen, I'm pretty nervous about trying to effectively add a
> heartbeat/keepalive mechanism into the filesystem, and effectively
> utilizing a disk block as a the communication path.  It's adding a
> fair amount of complexity into the filesystem, and it's not even a
> complete solution; without STONITH, we can easily end up with a
> situation where the FC cable gets temporarily disconnected, the backup
> takes over, and then when the FC cable gets reconnected, the master
> thinks it can still write to the disk, and.... oops!

By all means, we aren't trying to implement a full-scale HA mechanism.
We're just trying to prevent the inadvertent mount on the backup system.
Yes, STONITH is supposed to be involved in failover, but there are lots
of cases of human error where someone is manually trying to mount the
filesystem on the backup server, etc, that are well suited to fixing.

> But if we are going to do this, we can do it without adding an
> INCOMPAT_PO flag.  What we do is we define a new superblock field
> which defines the block that should be used as the "poor-man's
> heartbeat block", and then reserve the block in the bad block inode.

The reason the INCOMPAT flag is used in order to prevent some other
node which knows nothing about the heartbeat block from trying to
mount the filesystem.  This needs to be INCOMPAT because even an r/o
mount on a second node will still replay the journal and corrupt the
filesystem.

> It is actually possible to do a STONITH like thing, but it's
> **really** gross.  It requires that the kalived thread to every five
> seconds, to try reading from the heartbeat block, and if the read does
> not succeed within 3 seconds, the kernel must abort all filesystem I/O
> operations and revoke all file descriptors.

Yeah, I also thought about reading the block back, and this might be
desirable at some point in the future.  The current goal is just to
prevent another node from trying to mount the filesystem.  This can
also happen in non-HA scenarios where an FC SAN has disks that are
visible on nodes they shouldn't be.  Some clueless newbie thinks
"Wow, there's 2TB of disk space on this node that isn't even mounted,
I'm going to store my pr0n collection there...".

> When a system tries to grab the filesystem, it must first check the
> heartbeat block, wait 10 seconds, and check the heartbeat block again.
> If the sequence number has changed, then it must refuse the mount.
> Once it believes the system is dead, it must write its bootid UUID
> into the heartblock, and wait another 10 seconds, and recheck the
> heartbeat block.  If the bootid UUID changes, then something has gone
> wrong and the acquiring system must refuse the mount.  Only after the
> second 20 second wait should the backup system assume that it can
> safely assume that the other system is dead (or will refuse to touch
> the filesystem, in any case), and at that point, it can run the
> journal and mount the filesystem.

This is essentially what the current patch implements.  Having the boot
UUID in there would make it a bit safer than the current random 32-bit
sequence number, but it is nearly the same.

> The question though is how much of this should be done in the kernel,
> and how much should just be deferred to a userspace helper utility
> program.

IMHO, having this done in userspace is just asking for trouble.  You
first have to handle the INCOMPAT flag, then this has to be checked
after the filesystem is mounted but before journal replay (doing a
"pre check" is still racy with some other node mounting), then start
a daemon after the fs is mounted, and that daemon might never start or
die for any number of reasons.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.



-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642