Re: Packing, again

On Thu, 2004-09-02 at 04:21 +0100, Toby Dickenson wrote:
> On Thursday 02 Sep 2004 03:08, Richard Waid wrote:
> > I'm currently running several dirstorage instances, pretty much
> > flawlessly I might add. The only problem is packing -- one of our
> > instances is currently using 8-10 gig of storage (it's actually hard to
> > measure exactly, because doing a du takes a very, very long time on
> > reiser3 for some reason :)). It takes well over 12 hours to do _stage 1_
> > of a pack cycle, 
> 
> Pass 1 is a tidy-up cycle that is only run for the *first* pack cycle after 
> startup. Once that cycle has completed it knows it is in that tidy state, so 
> can skip pass 1 next time. Yes, taking 12 hours for that is poor.

The problem is that we restart Zope in between packs -- we have to it
consumes all memory eventually :)

> Using the permission marker method, This phase is effectively.....
> chmod -R u=rwX,g=r,o= /path/to/my/storage/A
> ....but in python, rather than using an external chmod command. Does running 
> that command (when there is no pack running!) complete faster than phase 1? 
> if it does, there is room for improvement in DirectoryStorage.

I'm picking that command would take a very long time still -- I _do_
expect it to take a long time, I'm just looking for ways to keep it
sane :)

> Which kernel are you using? the permissions marker method seems to trigger 
> some particularly bad performance of reiserfs. There are some patches to 
> reiserfs that should improve this greatly.... included as part of a "data 
> logging" or "small transactions" patch set from Chris Mason. These patches 
> have been in the SuSe kernel for ages, and are in 2.6.6 too. Ive not actually 
> confirmed that it improves packing performance, but it may be a useful 
> experiment if you have some time for testing and the possibility of 
> upgrading. I plan to benchmark on 2.6 sometime soon.

It's  2.4.x kernel, which won't change till we completely rebuild the
host. It would be worth experimenting, I just can't do it at the
moment :)

> Imagine: from cron, start packing at 11PM, then send it SIGINT it at 4AM. The 
> full pack cycle will be complete at the end of the week. Thats vaporware at 
> the moment. Ive got some implementation ideas. They wont get implemented 
> without sponsorship or a volunteer.

That would be nice, very nice.

> > The MinimalStorageMarker method 
> > claims to be between 20% and 80% faster -- is it just as stable? I note
> > that I could just create a 'misc/packing' and link it to a different
> > drive.
> 
> Memory usage patterns are different, so I wouldnt guarantee success.
> 
> It is still layered on top of the normal disaster-recovery mechanisms, so you 
> should never unrecoverably loose data.  If all goes wrong, just rename the 
> *-deleted files. I think its safe to try.

I tried it on a much smaller storage (250M) and it completed in 85
seconds, deleting 50k transactions. That's interesting, since even if we
assume that the storage is 50x larger (it isn't), it should be
completing in 6000 seconds. Stage one only took a second.

> > It would also really, really help to have some kind of 'checkpoint'
> > messages in the packing process. At some stages in the packing it's
> > really difficult to tell if it's actually doing something, if it has
> > locked up, or if it is still going.
> 
> Has it ever actually locked up?

At least once Zope _has_ refused to return anything after 8+ hours of
packing. It's difficult to say whether it was dirstorage or something
else going on as well.

> > Maybe every 10,000 transactions it 
> > could just write something like 'marked x0,000 transactions, last was:
> > xxxxxx'. If there is some way to figure out _actual_ progress that would
> > be even better.
> >
> > Another issue is replicas. Once the storage has finished packing, I need
> > to re-tar it, rsync it, then resume replication
> 
> Do you? the normal replication tool shouldnt need this seperate tar+rsync step 
> - it will pick up where it left off when packing is complete.

I'm getting:

Preparing local storage....
Fetching increment....
ReplicaError: storage has been packed since the last replica
( 2004-05-26 13:02:55 GMT > 2004-05-26 11:10:20 GMT )
ReplicaError: Error from remote replica

If it just picks up where it leaves off ... wouldn't that mean that
you'd have to pack the replica separately anyway? (in which case, the
rsync might simply be quicker anyway).


--Richard




-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click