Re: Packing, again

On Saturday 04 Sep 2004 11:57, Richard Waid wrote:

> I'm currently exporting the entire database to a file (via the
> export mechanism in ZODB) to mitigate against any possible dataloss.

I recommend doing this to a different filesystem on a different disk 
controller, to guard against hardware problems. Ideally, please take a tar 
backup of both replica and master storage directories.

> I've just tried a pack of our largest dirstorage, 

You mentioned you have been trying the 'Minimal' packing method. Was that in 
use here, or with this storage in the past?

> running at somewhere 
> between 8-10 gig. It aborted with a 'FileDoesNotExist' error, which
> worries me quite a bit.
>
> """
> FileDoesNotExist: DirectoryStorage file 'A/o000/000/000/012/A5D6.c' does
> not exist
> """

Thats not good.

> What could cause this?

> The missing file _does_ exist in our replica, 

Can you check whether there are any files matching 
A/o000/000/000/012/A5D6* on the master. If this file has been removed by 
packing then there will still be a file
A/o000/000/000/012/A5D6.c-XXXXXX-deleted
where XXXXXXX is an integer timestamp of the time that packing decided to 
remove it.

> and  
> interestingly enough the date seems to correspond exactly to where
> someone accidently copied about 1.5GB of data to another part of the
> ZODB, which we discovered and subsequently deleted a couple of days ago.
> The database hasn't been packed since that event (neither the copying or
> the deletion).

DirectoryStorage only deletes data files during a pack. Maybe there was a pack 
that you dont know about.....

Please verify this by:
1. using dumpdsf.py on the time-of-last-packing file. example usage below.
2. checking that for any A/o000/000/000/012/A5D6.c-XXXXXX-deleted file

We know the file was written to disk because it appears in the replica. If you 
can verify that there has been no pack since the data was written then I 
suspect a cause outside of DirectoryStorage.

> Perhaps more importantly, what can be done to fix it (without data
> loss)? 

If it does look like packing has gone wrong, then it is definitely safe to 
repair a storage by 'undeleting' files by renaming them to remove the 
-XXXXXX-deleted suffix or copying individual files from the replica. This is 
completely safe provided you do not overwrite any file. (overwriting a 
damaged file might be the right way to fix a problem, but you need to take 
care)

checkds.py will tell you if any problems remain.

Please let us know what happens.

dumpdsf example:

$ python dumpdsf.py ~/projects/Zope/var/ds/A/x/packed
/home/toby/projects/Zope/var/ds/A/x/packed
  current rev 03577F55D34D77CC
  transaction timestamp Mon Aug 30 16:17:49 2004



-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click