RE: NFS Help! Terrible performance with sync, fast performance with async

>A couple of things:
>
>1) Is this RHEL-3? The server and client in RHEL-3 default to UDP
>packets which are the 'worst' for high bandwidth networks. You can
>find out what the client is mounting things as by looking in
>/proc/mounts. I forget exactly what needs to be done on the server
>side (RHEL-4 supposedly has a better TCP server but I am not sure).
>Once you have both server and client on TCP... you should see an
>improvement.

>From my understanding, UDP should be fine in this scenario because there
is only one switch between the devices and no packet loss is occurring.
I ran my tests with tcp though (RHEL3 uses NFSv3 which does support TCP,
it just has to be explicitly enabled) and the results were no better.  

>2) What is your switch set up as? A dumb switch should be ok, but some
>switches will try to do things like 'grouping' packets etc which can
>cause bad performance.

It's the cheapest Dell Powerconnect switch that can do jumbo frames
(i.e. sub $100 switch).  I've run my tests over a GigE network and a
100Mbps network using a cheap D-Link switch and same results between the
two.

>3) Look at using iozone as a second test. It may be able to show where
>the problems are  better.

I've seen iozone but I haven't tried it yet.  Its just meta-data changes
seem to take much longer than they need to due to the commit process.  I
should also mention that the tests for reading/writing sequential
problems run perfectly using sync.  Its only file creation/deletion that
exhibit the extremely slow speeds.

Thanks,

Chris Wornell
Network Administrator, Information Technology
Peerless Systems Corporation
http://www.peerless.com
office: 310.727.5723 
fax: 310.727.5715 
mailto:cwornell-KWjJJj/cdYZWk0Htik3J/[email protected]

-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Stephen John
Smoogen
Sent: Sunday, November 19, 2006 9:01 AM
To: Discussion of Red Hat Enterprise Linux 3 (Taroon)
Subject: Re: NFS Help! Terrible performance with sync,fast performance
with async

On 11/19/06, Chris Wornell <CWornell-KWjJJj/cdYZWk0Htik3J/[email protected]> wrote:
>

> I've got a problem that I've spent quite a bit of time on, though I'm
not an
> expert at NFS. In summary, operations that require meta-data changes
(such
> as file/directory creations/deletions), perform extremely slow over
sync,
> but over 10x faster using async.
>
>  I have two systems, connected to a GigE switch using intel pro 1000
NICs
> (jumbo frames is currently not enabled on any of the points).
>
>  The NFS server is a dual-core opteron system with 1GB of RAM and
3x300 SAS
> disk RAID-5 on a Perc5/i controller with 256MB battery backed cache
(write
> cache is enabled). The file system is ext3. I've configured nfsd to
spawn 32
> processes upon startup. I'm using defaults for export the nfs shares,
no
> changes to rsize or wsize.
>
>  The NFS client is a dual Xeon with 4GB of RAM and a single 7200rpm
SATA
> disk. Both systems are running RHEL WS 3 Update 8 and kernel
> 2.4.21-47.0.1.ELsmp.
>
>  For testing, I'm using bonnie++. The following are some sample test
results
> that sum up the problem:
>
>  Test on NFS server directly (not NFS loopback)
>  -Sequential File Creation: 2976
>  -Sequential File Deletion: N/A
>  -Random File Creation: 3077
>  -Random File Deletion: 9922
>
>  NFS test with sync enabled
>  -Sequential File Creation: 39
>  -Sequential File Deletion: 79
>  -Random File Creation: 39
>  -Random File Delection: 65
>
>  NFS test with async enabled
>  -Sequential File Creation: 575
>  -Sequential File Deletion: 1718
>  -Random File Creation: 543
>  -Random File Deletion: 1228
>
>  Based on the local performance of the NFS server, it does not appear
the IO
> setup is the culprit. My understanding of the sync operation is a
commit
> happens which means the NFS server doesn't reply back until the change
has
> actually been committed to stable storage. There is something
happening
> behind the scenes though which is causing a huge delay before the NFS
server
> replies back the commit was complete.
>
>  This question is actually work related and I'm planning to put the
NFS
> server into production, but I'd rather not use async, even with a UPS
and
> dual PSU's on the server. With the newer nfs-utils, sync is the
default
> option as well so it seems like sync should perform relatively well.
>
>  Another question is I don't quite understand how the data corruption
> happens if a power loss occurs on an NFS server using async. Even with
sync,
> data transferred over the wire maybe loss if the nfs server gets shut
down
> before that data is committed. Can anyone go into more detail on how
the
> data corruption happens?
>
>  Thanks a bunch!
>

A couple of things:

1) Is this RHEL-3? The server and client in RHEL-3 default to UDP
packets which are the 'worst' for high bandwidth networks. You can
find out what the client is mounting things as by looking in
/proc/mounts. I forget exactly what needs to be done on the server
side (RHEL-4 supposedly has a better TCP server but I am not sure).
Once you have both server and client on TCP... you should see an
improvement.

2) What is your switch set up as? A dumb switch should be ok, but some
switches will try to do things like 'grouping' packets etc which can
cause bad performance.

3) Look at using iozone as a second test. It may be able to show where
the problems are  better.

-- 
Stephen J Smoogen. -- CSIRT/Linux System Administrator
How far that little candle throws his beams! So shines a good deed
in a naughty world. = Shakespeare. "The Merchant of Venice"

--
Taroon-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/taroon-list

--
Taroon-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/taroon-list