Performance data points

Just for those out there with idle curiosity I thought I'd report some of
the results of some recent work we've done on our ZODB server to speed
things up a bit.

We are running DirectoryStorage 1.1.9 on FreeBSD 4-STABLE on a
UFS+softupdates,noatime partition on a HP/Compaq Proliant DL380G2 (dual
1.4Ghz PIIIs, 2GB RAM).  Our ZODB is roughly 6GB.

The general performance of DiretoryStorage was fairly good, but I wanted
to try and speed up some of the bigger operations, such as packing and
catalog re-indexing that do sustained hammering of the ZODB and lots of
random IO on the disks.

Our original setup was 3 x 36GB 15K SCSI U320 disks in a RAID5 array on a
SmartArray 5i controller with battery backed write cache.  We made the
following upgrades:

* Added 3 more (identical) disks
* Re-arranged the array to:
    System/Zope/Logs/etc: 2 disks, RAID 1
    ZODB: 4 disks, RAID 1+0
  (both still on the same controller and SCSI bus)
* Reduced the filesystem block/frag size from 16K/2K -> 8K/1K
* Copied the entire ZODB to another machine, did newfs with above
  options, then back again (rsync)

To test the setup we just did:
time find . -name="foo"
in the zodb/A directory to measure to time it takes to traverse the entire
ZODB filesystem tree (several millions of files).

This took around 12 minutes on the original system, and only around 3-4
minutes on the new setup.  So we managed to get a 3-fold increase in
random disk read IO.

I'm not sure which part of the changes made the most significant
difference.  Reducing the block size should reduce wastage on the disk
from many small files (we also increased the number of inodes).  Changing
from 3 disks to 4 disks for the ZODB should in theory only increase
performance by a 25% as I don't think RAID5 and RAID1+0 would have any
significant differences in read performance beyond the number of spindles
you have.

Copying the ZODB to another machine and back again is an interesting one. 
I don't see how this would make that much difference to UFS.  IIRC, UFS
attempts to spread files out evenly across a disk so I don't think it
could be anything to do with the order of files.  My only thought is I
don't know how well UFS handles updates to directories, maybe multiple
changes over time cause the directory entries to bloat and a rewrite of
all this compacts this.

I have noticed that before according to iostat we would max out at around
300-400 IO ops/sec whereas now we can reach 1000-2000 ops/sec.  Hopefully
that is a benefit of having a smaller block size and not an adverse
symptom of having one (ie. needing to do more IO ops to achieve the same
result).

These tests were done when the system was idle, so the seperation of the
system and zodb onto different sets of disks shouldn't make much
difference, however in real use we see it does help quite a lot.

Food for thought :)

-Matt


-- 
Matt Hamilton                                         [email protected]
Netsight Internet Solutions, Ltd.          Business Vision on the Internet
http://www.netsight.co.uk                               +44 (0)117 9090901
Web Design | Zope/Plone Development and Consulting | Co-location | Hosting




-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click