Re: Packing, again

On Thursday 02 Sep 2004 03:08, Richard Waid wrote:
> I'm currently running several dirstorage instances, pretty much
> flawlessly I might add. The only problem is packing -- one of our
> instances is currently using 8-10 gig of storage (it's actually hard to
> measure exactly, because doing a du takes a very, very long time on
> reiser3 for some reason :)). It takes well over 12 hours to do _stage 1_
> of a pack cycle, 

Pass 1 is a tidy-up cycle that is only run for the *first* pack cycle after 
startup. Once that cycle has completed it knows it is in that tidy state, so 
can skip pass 1 next time. Yes, taking 12 hours for that is poor.

Using the permission marker method, This phase is effectively.....
chmod -R u=rwX,g=r,o= /path/to/my/storage/A
....but in python, rather than using an external chmod command. Does running 
that command (when there is no pack running!) complete faster than phase 1? 
if it does, there is room for improvement in DirectoryStorage.

Which kernel are you using? the permissions marker method seems to trigger 
some particularly bad performance of reiserfs. There are some patches to 
reiserfs that should improve this greatly.... included as part of a "data 
logging" or "small transactions" patch set from Chris Mason. These patches 
have been in the SuSe kernel for ages, and are in 2.6.6 too. Ive not actually 
confirmed that it improves packing performance, but it may be a useful 
experiment if you have some time for testing and the possibility of 
upgrading. I plan to benchmark on 2.6 sometime soon.

> which is very bad when you need to pack, but have a 
> busy site. I've had to abort the last couple of attempts because it was
> having a significant impact on site performance, even though the pack
> _started_ well outside of peak load hours.
>
> I'm looking for an alternative here. 

The solution to doing anything efficiently with big storages is to perform the 
work incrementally.......

Imagine: from cron, start packing at 11PM, then send it SIGINT it at 4AM. The 
full pack cycle will be complete at the end of the week. Thats vaporware at 
the moment. Ive got some implementation ideas. They wont get implemented 
without sponsorship or a volunteer.

> The MinimalStorageMarker method 
> claims to be between 20% and 80% faster -- is it just as stable? I note
> that I could just create a 'misc/packing' and link it to a different
> drive.

Memory usage patterns are different, so I wouldnt guarantee success.

It is still layered on top of the normal disaster-recovery mechanisms, so you 
should never unrecoverably loose data.  If all goes wrong, just rename the 
*-deleted files. I think its safe to try.

> It would also really, really help to have some kind of 'checkpoint'
> messages in the packing process. At some stages in the packing it's
> really difficult to tell if it's actually doing something, if it has
> locked up, or if it is still going.

Has it ever actually locked up?

> Maybe every 10,000 transactions it 
> could just write something like 'marked x0,000 transactions, last was:
> xxxxxx'. If there is some way to figure out _actual_ progress that would
> be even better.
>
> Another issue is replicas. Once the storage has finished packing, I need
> to re-tar it, rsync it, then resume replication

Do you? the normal replication tool shouldnt need this seperate tar+rsync step 
- it will pick up where it left off when packing is complete.

>, which means there is a 
> significant period of time when the replica is out of sync.

Yes, Replication will certainly be offline while the pack is running.

At one time I considered fixing this.... in theory we dont need to block 
snapshot mode except during the last phase of packing. 

However in my current configuration there isnt much to be gained by doing this 
because I synchronise packing to occur on the replica storage at the same 
time as on the master. Making it possible to replicate *into* a storage with 
an active pack process would be a more fundamental change.






-------------------------------------------------------
This SF.Net email is sponsored by BEA Weblogic Workshop
FREE Java Enterprise J2EE developer tools!
Get your free copy of BEA WebLogic Workshop 8.1 today.
http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click