Re: NFS Help! Terrible performance with sync, fast performance with async

On 11/20/06, Christopher McCrory <[email protected]> wrote:
> Hello...
>

Sorry for misparsing which Chris(topher) I was talking to.

> On Mon, 2006-11-20 at 15:06 -0700, Stephen John Smoogen wrote:
> > On 11/20/06, Christopher McCrory <[email protected]> wrote:
> > > Hello...
> > >
> > > On Sun, November 19, 2006 1:58 am, Chris Wornell wrote:
> > > > I've got a problem that I've spent quite a bit of time on, though I'm
> > > > not an expert at NFS. In summary, operations that require meta-data
> > > > changes (such as file/directory creations/deletions), perform extremely
> > > > slow over sync, but over 10x faster using async.
> > > >
> > >
> > > The same thing happens with RHEL4; software raid and lvm/lvm2 make it worse.
> > > I opened a support ticket with RH, but it didn't really go anywhere.
> > > I'm considering switching my NFS server to FreeBSD 6.2
> >
> > I am going to assume that you did this a couple of days ago.. as
> > opening a ticket on a Sunday nigth and expecting a resolution by
> > Monday morning is a bit tight.
>
> Actually I opened a ticket last June.  It went something like:
> Me: examples ( like below)
> RH: We replicated in lab, please do these other tests
> Me: ( busy ) figured if they can see it in their lab, why do they need
> me?  There must be lots of other people seeing the same thing.  I
> probably should have followed up.
>

yeah. The reason for usually asking for more tests is trying to figure
out what might be causing it if there are multiple ways it could be
triggered.

The problems look to be in the kernel hardware interaction. THe fact
that your system gets slower on software-raid(SR), LVM, SR+LVM, etc
shows that the NFS server is having a hard time navigating the
interactions.

To be honest, this may be a place where Linux is just going to suck.
The NFS server stuff has a lot of issues to deal with.. and reading
http://nfsworld.blogspot.com/ (which I am guessing is from Sun ) there
are a lot of places where it hasnt been optimized.


> ( Again, I was not the OP but,) here are my numbers:
> ( done on live systems with people using them so MMV )
> nfs mount opts: wsize=8192,rsize=8192,hard,tcp
> GigE connects
> NFS export is LVM2 over software raid
> lots of 15k U320 scsi drives
> with RHEL3 client and RHEL4 server
> roughly the same with RHEL5
>
> summary: untar file with overwrite takes same amount of time as copying
> a single file 10x as large.  File creation seems to be the bottleneck,
> not data transfer.
>
> Not shown, but non-raid, non-lvm filesystems are worse than single
> drives, but single drives are still painful over NFS.  e.g. "cvs co" or
> "svn co"
>
>
> # copy from NFS server to local disk: 2.1G iso
> [[email protected] iso]$ time cp
> RHEL5-Server-20060830.1-SRPMS-DVD.iso /tmp/
>
> real    1m17.591s
> user    0m0.340s
> sys     0m19.680s
>
> # copy from local disk to NFS server 2.1G iso
> [[email protected] iso]$ time
> cp /tmp/RHEL5-Server-20060830.1-SRPMS-DVD.iso
> RHEL5-Server-20060830.1-SRPMS-DVD.iso.copy
>
> real    3m25.563s
> user    0m0.140s
> sys     0m16.000s
>
> # untar from local disk to NFS server 214M uncompressed tarball
> [[email protected] iso]$ time tar
> -xf /tmp/mysql-debug-5.0.24a-linux-i686.tar
>
> real    2m43.387s
> user    0m0.040s
> sys     0m1.890s
>
> # untar from local disk to NFS server 214M uncompressed tarball with
> overwrite
> [[email protected] iso]$ time tar
> -xf /tmp/mysql-debug-5.0.24a-linux-i686.tar
>
> real    3m26.998s
> user    0m0.050s
> sys     0m2.300s
>
>
> ## just for comparison, async exports:
> # async iso test:
> [email protected] async]$ time
> cp /tmp/RHEL5-Server-20060830.1-SRPMS-DVD.iso
> RHEL5-Server-20060830.1-SRPMS-DVD.iso.copy
>
> real    0m41.678s
> user    0m0.270s
> sys     0m13.380s
>
> # async tar test
> [[email protected] async]$ time tar
> -xf /tmp/mysql-debug-5.0.24a-linux-i686.tar
>
> real    0m8.259s
> user    0m0.050s
> sys     0m1.540s
>

-- 
Stephen J Smoogen. -- CSIRT/Linux System Administrator
How far that little candle throws his beams! So shines a good deed
in a naughty world. = Shakespeare. "The Merchant of Venice"

--
Taroon-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/taroon-list