Replication and cold failover

Hi folks,

Sorry for the long message... I'm attempting to flesh out some
replication / failover / High Availability plans to see whether we can
get by with only some scripting of DirectoryStorage's features, or if
we in fact need to fork out the money for ZRS from Zope Corp. Comments
from anyone who's extensively used the replication feature would be
much appreciated.

Even more welcome would be working example scripts to make it all
happen!

There was a "cliff's notes" message about HA here
http://mail.zope.org/pipermail/zope-cmf/2003-February/017255.html
which gives some idea of how it might be done, but I'm unclear whether
that was a "we've done this" message or a "this might work" message.

In another mail thread on this list from June 10-11 2004, titled
"Replicating", we established that it is not possible to get "warm" or
"hot" failover from a master to a replica.  The reason is that, as
Toby explained, "Replication works by transferring a tar file of
changes onto the replica machine. ... This all happens outside the
ZODB transaction mechanisms, so its not possible to have a storage
process running on the replica at the same time."  So the secondary
can't be up and running as long as we're replicating to it.

But! Maybe we can get automated cold failover? That should do the
trick for us. Meaning, as described at http://tinyurl.com/7ublt :

"""After a failure, the standby system takes over the applications
formerly running on the failed system. ...  [T]he standby system
senses a heartbeat from the production system on a frequent and
regular basis. If the heartbeat consistently stops for a period of
time, the standby system automatically assumes the IP address ...
associated with the failed system. The standby can then run any
applications ... [e.g. ZEO] that were on the failed system.
"""

That actually doesn't sound too hard to set up on the face of it.  But
there are possible complications.

Let's say that our master is called A and our replica is called B.
When failover occurs, A is temporarily dead and B becomes the master.
When A is working again, we want to get any new transactions from B
back to A before we make it the master again.

The latter requirement means that before we can restart the master, we
need to treat it as a temporary replica and pull the new transactions
to it.  For efficiency and minimum downtime, I imagine this could be
done in a sequence something like:

* A replicates from B

* when done, A calls via ssh some script that tells B to
  shut down

* A replicates from B again to get the very latest transactions.  This
  should be fast. (Question: Can you replicate from a master storage
  that is not running a storage process? I don't see why not.)

* A starts up.

* failover scripts need to somehow be told to wait until now to switch
  the IP back to A.


The one big flaw here is that AFAICT, if the failover coincides with
any network problems, OR occurs between a transaction commit on A and
a replication to B, we could well have a few transactions on A that
aren't on B.  Which we'd like to preserve, but is it possible?  I
suspect not.


The next complication is packing.  Again, let's call the master A and
the replica B.  Once I pack A, on the next replication B gets the new
transactions but I assume it doesn't automagically pack itself. In
other words, once I've packed A, and then we have failover and I later
need to replicate from B back to A, have I violated the requirement
that "the only difference between the two storages is that the master
contains some newer transactions that are not present on the replica"?
Taken literally, it seems the answer is yes, because B (the temporary
master) never got packed so it has some OLD transactions that are not
present on A.  Or maybe the replication tool simply ignores those and
only copies over the new transactions, in which case we should be OK?

(Which leads to a more general question:  How can I pack my replicas?
I don't want 'em to just keep growing forever.)

One approach to the fail-back problems might be to only automate
failover, and bring A back up by hand, following the sequence
described above. But I suspect this is a false economy, since even
then I'd want some scripts in place so I don't have to teach the
sysadmin the constraints of properly replicating a
DirectoryStorage. Not that it's that complicated, but in a crisis,
things are often forgotten.

-- 

Paul Winkler
http://www.slinkp.com


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click