Re: Replication and cold failover

On Tuesday 26 Jul 2005 22:04, Paul Winkler wrote:
> Hi folks,
>
> Sorry for the long message... I'm attempting to flesh out some
> replication / failover / High Availability plans to see whether we can
> get by with only some scripting of DirectoryStorage's features, or if
> we in fact need to fork out the money for ZRS from Zope Corp. Comments
> from anyone who's extensively used the replication feature would be
> much appreciated.

I know nothing about ZRS, but I can answer for DirectoryStorage.

> That actually doesn't sound too hard to set up on the face of it.  But
> there are possible complications.
>
> Let's say that our master is called A and our replica is called B.
> When failover occurs, A is temporarily dead and B becomes the master.
> When A is working again, we want to get any new transactions from B
> back to A before we make it the master again.

Another option is to configure the storage running on B during this failover 
period to be read-only. B never holds any transactions that are not also 
present on A, so this problem scenario never arises.

This is the only type of ZODB cold standby that I have used (using heartbeat 
on linux)

> The latter requirement means that before we can restart the master, we
> need to treat it as a temporary replica and pull the new transactions
> to it.  For efficiency and minimum downtime, I imagine this could be
> done in a sequence something like:
>
> * A replicates from B

DirectoryStorage replication currently raises an error if there are new 
transactions in the replica destination which are not present on the replica 
source. This is the "one big flaw" scenario you describe below. I can see two 
options:

a. Use rsync from B to A. This will be somewhat slow - time proportional to 
storage size. And not transactional. If you do this, think carefully about 
what happens if A fails again half way through.

b. Use the special failover-recovery mode of the DirectoryStorage replication 
tool which does not raise an error in this scenario. It atomicly peels back 
the doomed transactions on A (back to some common ancestor transaction) then 
replicates as normal. This is fast - time proportional to the number of 
outstanding transactions. Unfortunately this mode has not yet been 
implemented ;-) Patches or sponsorship gratefully accepted.

> * when done, A calls via ssh some script that tells B to
>   shut down
>
> * A replicates from B again to get the very latest transactions.  This
>   should be fast. (Question: Can you replicate from a master storage
>   that is not running a storage process? I don't see why not.)

yes, you can.

> * A starts up.
>
> * failover scripts need to somehow be told to wait until now to switch
>   the IP back to A.
>
>
> The one big flaw here is that AFAICT, if the failover coincides with
> any network problems, OR occurs between a transaction commit on A and
> a replication to B, we could well have a few transactions on A that
> aren't on B.  Which we'd like to preserve, but is it possible?  I
> suspect not.

Automatic preservation will be difficult. It will be worth saving the raw 
pickle files of those doomed transactions just in case they contain some 
critical information which justifies a manual salvage process.

> The next complication is packing.  Again, let's call the master A and
> the replica B.  Once I pack A, on the next replication B gets the new
> transactions but I assume it doesn't automagically pack itself. 

Yes, thats correct. Replication ever deletes anything. That is necessary to 
prevent permanent data loss by packing too much on the master - either though 
human error or a bug in the packing code.

> In 
> other words, once I've packed A, and then we have failover and I later
> need to replicate from B back to A, have I violated the requirement
> that "the only difference between the two storages is that the master
> contains some newer transactions that are not present on the replica"?
> Taken literally, it seems the answer is yes, because B (the temporary
> master) never got packed so it has some OLD transactions that are not
> present on A.  Or maybe the replication tool simply ignores those and
> only copies over the new transactions, in which case we should be OK?

The replication tool ignores "old history".

The minimal constraint is that "the most recent transaction in the replication 
destination must exist unchanged on the source". Any transactions before that 
are "old history" and can be ignored.

The result for the operator is that when packing you must keep enough days 
history to cover your previous replication event. Using '0 days' is asking 
for trouble.

[ That constraint is still not quite true. You can skip this paragraph if you 
dont care for storage implementation details.....  "exist unchanged" is 
determined by md5 checksum of the transaction file, but each transaction file 
contains a pointer to the previous transaction. This means that "old history" 
starts one transaction earlier that the naive definition above. As of version 
1.1.8 DirectoryStorage packing is aware of this, and the operating 
instructions above are the whole truth. ]

> (Which leads to a more general question:  How can I pack my replicas?
> I don't want 'em to just keep growing forever.)

import sys
from DirectoryStorage.Full import Full
from DirectoryStorage.Filesystem import Filesystem
from DirectoryStorage.utils import storage_pack_days
f = Filesystem(sys.argv[1])
s = Full(f,synchronous=True)
storage_pack_days(s,30)
s.close()



-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click