Re: How to get out of snapshot mode?

On Monday 20 Mar 2006 17:02, Ed Leafe wrote:
> On Mar 20, 2006, at 11:29 AM, Toby Dickenson wrote:
> >> Snapshot mode
> >> The storage is currently in snapshot mode recombining/startup.
> >
> > OK, so you already have restarted the storage server.
> > DirectoryStorage has
> > started recombining changes made during that previous snapshot
> > mode, and wont
> > let another snapshot mode start until that process is complete.
>
> 	FWIW, I had started the server yesterday afternoon. This morning I
> got an error message for the backup process, which is what initiated
> my email to you. 

> Shouldn't it have finished recombining after more than 12 hours?

On a lightly loaded machine it should be very quick... a few seconds at most. 
(The amount of time taken is proportional to the number and size of all the 
write transactions since snapshot mode was entered)

> 2006-03-20T11:46:28 INFO ZServer HTTP server started at Mon Mar 20
> 11:46:28 2006
>      Hostname: 0.0.0.0
>      Port: xxxx

Zope starts.....

> 2006-03-20T11:46:40 INFO DirectoryStorage Flushing 35 transactions
> (recovery)

Twelve seconds later we have confirmation that DirectoryStorage has started 
its flusher thread, and started flushing 35 transactions from its journal/ 
directory into the B/ directory.

The problem is that nothing follows it. No log messages indicating progress, 
problems, or anything that might cause the process to stall. Recreating the 
same situation on my development machine I see:

2006-03-21T21:10:29 INFO DirectoryStorage Flushing 6 transactions (recovery)
------
2006-03-21T21:10:30 INFO DirectoryStorage Left snapshot mode 
'recombining/startup'


The next area for investigation is to see what happened to those 35 
transactions. Check the files under the various directories under the storage 
directory, and check which files are out of place. Hopefully that will give a 
clue about which part of the process has stalled. The lifecycle should be:

a. There would have been 35 directories named journal/working_xxxxx_done
b. The files in those directories are moved to subdirectories of B/
c. Then those directories in journal/ are unlinked
d. Then all the files under B/ are moved into A/
e. Then the subdirectories in B/ are unlinked
f. And snapshot mode is reopened.

The number of files that need to be moved in step d is proportional to the 
number and size of all the write transactions since snapshot mode was 
entered. If there are many files in B/ then this may be a clue that this 
process has been broken for a while.


> >> 	How would I be able to tell if this is the case? And if so, what
> >> 	steps would I take?
> >
> > lsof, then kill it.
> >
> > However Im now sure thats not the case.
>
> 	You're correct. Running that shows that the current Zope process has
> the files open, and killing Zope releases the lock.

Thats good. That confirms that the DirectoryStorage in the Zope process knows 
that no other process is relying on A/ being a snapshot, therefore it should 
be proceeding with steps d, e, and f above.

Im sorry I dont have a more definite answer.


-- 
Toby Dickenson


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642