Re: NFS Help! Terrible performance with sync, fast performance with async

Hello...

On Mon, 2006-11-20 at 15:06 -0700, Stephen John Smoogen wrote:
> On 11/20/06, Christopher McCrory <[email protected]> wrote:
> > Hello...
> >
> > On Sun, November 19, 2006 1:58 am, Chris Wornell wrote:
> > > I've got a problem that I've spent quite a bit of time on, though I'm
> > > not an expert at NFS. In summary, operations that require meta-data
> > > changes (such as file/directory creations/deletions), perform extremely
> > > slow over sync, but over 10x faster using async.
> > >
> >
> > The same thing happens with RHEL4; software raid and lvm/lvm2 make it worse.
> > I opened a support ticket with RH, but it didn't really go anywhere.
> > I'm considering switching my NFS server to FreeBSD 6.2
> 
> I am going to assume that you did this a couple of days ago.. as
> opening a ticket on a Sunday nigth and expecting a resolution by
> Monday morning is a bit tight.

Actually I opened a ticket last June.  It went something like:
Me: examples ( like below)
RH: We replicated in lab, please do these other tests
Me: ( busy ) figured if they can see it in their lab, why do they need
me?  There must be lots of other people seeing the same thing.  I
probably should have followed up.


For Guil Barros: it was Service Request #920132



>  If you are more comfortable with
> FreeBSD then I would suggest it as you would know how to tune it
> better.
> 

Actually I know linux better than FreeBSD and would prefer a RHEL
solution.  So far FreeBSD 6.2 is much faster with older slower hardware.
For me, in many cases the speed does not matter (reads are fast), but
the subversion client is unusable over NFS. 



> To answer your earlier question,

I was not the OP, but do appreciate you responding. 

>  I have found that going to TCP on
> links faster than 100mb always gives better performance than UDP...
> even with crossover cables. I tested this with:
> Netapp Server -> AIX 5.3
> Netapp Server -> Solaris 2.8/2.9
> Netapp Server -> Linux 2.4.20+
> Netapp Server -> Linux 2.6.9+
> 
> Also a test with Solaris <-> Solaris. The UDP packets only seem to
> work ok with jumbo packets on a smart switch that could optimize data
> transfers.
> 
> File creation is going to always be faster with async... sometimes by
> a factor of 10. You can improve your other performance with a Linux
> NFS server, you need to be judicious in the network card (I like the
> e1000 over the broadcom which has always had weird latencies for me),
> and your kernel tuning. The Linux kernel is not set up to be an NFS
> server by default and you need to set up the size of the buffers and
> network tuning to meet your needs.
> 
> The problems sound like more about writing to disk. What does the
> local bonnie say when writing to RHEL-4 disks? and which version of
> RHEL-4?
> 

( Again, I was not the OP but,) here are my numbers:
( done on live systems with people using them so MMV )
nfs mount opts: wsize=8192,rsize=8192,hard,tcp
GigE connects
NFS export is LVM2 over software raid
lots of 15k U320 scsi drives
with RHEL3 client and RHEL4 server
roughly the same with RHEL5

summary: untar file with overwrite takes same amount of time as copying
a single file 10x as large.  File creation seems to be the bottleneck,
not data transfer.

Not shown, but non-raid, non-lvm filesystems are worse than single
drives, but single drives are still painful over NFS.  e.g. "cvs co" or
"svn co"


# copy from NFS server to local disk: 2.1G iso
[[email protected] iso]$ time cp
RHEL5-Server-20060830.1-SRPMS-DVD.iso /tmp/

real    1m17.591s
user    0m0.340s
sys     0m19.680s

# copy from local disk to NFS server 2.1G iso
[[email protected] iso]$ time
cp /tmp/RHEL5-Server-20060830.1-SRPMS-DVD.iso
RHEL5-Server-20060830.1-SRPMS-DVD.iso.copy

real    3m25.563s
user    0m0.140s
sys     0m16.000s

# untar from local disk to NFS server 214M uncompressed tarball
[[email protected] iso]$ time tar
-xf /tmp/mysql-debug-5.0.24a-linux-i686.tar

real    2m43.387s
user    0m0.040s
sys     0m1.890s

# untar from local disk to NFS server 214M uncompressed tarball with
overwrite
[[email protected] iso]$ time tar
-xf /tmp/mysql-debug-5.0.24a-linux-i686.tar

real    3m26.998s
user    0m0.050s
sys     0m2.300s


## just for comparison, async exports:
# async iso test:
[email protected] async]$ time
cp /tmp/RHEL5-Server-20060830.1-SRPMS-DVD.iso
RHEL5-Server-20060830.1-SRPMS-DVD.iso.copy

real    0m41.678s
user    0m0.270s
sys     0m13.380s

# async tar test
[[email protected] async]$ time tar
-xf /tmp/mysql-debug-5.0.24a-linux-i686.tar

real    0m8.259s
user    0m0.050s
sys     0m1.540s



-- 
Christopher McCrory
 "The^W One of the guys that keeps the servers running"

[email protected]
 http://www.pricegrabber.com

To the optimist, the glass is half full.
To the pessimist, the glass is half empty.
To the engineer, the glass is twice as big as it needs to be.


--
Taroon-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/taroon-list