Re: CIFS slow on gigabit, doesn't support sockopt=TCP_NODELAY ?

On Mon, 24 Aug 2009 16:29:52 -0400
Timothy Normand Miller <[email protected]> wrote:

> On Mon, Aug 24, 2009 at 3:42 PM, Jeff Layton<[email protected]> wrote:
> > On Mon, 24 Aug 2009 14:36:37 -0400
> > Timothy Normand Miller <[email protected]> wrote:
> >
> >> On Mon, Aug 24, 2009 at 9:50 AM, Jeff Layton<[email protected]> wrote:
> >> > On Sun, 23 Aug 2009 20:21:10 -0400
> >> > Timothy Normand Miller <[email protected]> wrote:
> >> >
> >> >> Hi, everyone.
> >> >>
> >> >> I've noticed that the cifs client for Linux is slow over gigabit
> >> >> ethernet.  It seems to max out at about 10 megs/sec, while the drives
> >> >> can go a lot faster.  Also, if I mount the same network share from a
> >> >> Mac, it's a lot faster.  When _sharing_ via samba, one can set the
> >> >> TCP_NODELAY option (among others), which fixes the problem.  But with
> >> >> the cifs client, I find that there appears to be no way to set the
> >> >> option.  When mounting manually, you can use "-o sockopt=TCP_NODELAY",
> >> >> and you can also put that into /etc/fstab.  Either way, the option
> >> >> appears to be ignored.
> >> >>
> >> >
> >> > Yeah. The sockopt= option is completely ignored. I proposed a patch
> >> > to remove it not too long ago...
> >> >
> >> > I'm unclear...so you're saying here that turning on TCP_NODELAY on the
> >> > server it helps throughput when using the Linux CIFS client?
> >>
> >> No.  I think I'll give you excess detail just to make sure that I
> >> don't leave out something important.  :)
> >>
> >> I have a Linux server, an Apple AEBS with disks hanging off of it, and
> >> various clients (some Macs).  On the Linux server, I can configure the
> >> SMB server to enable those options so that when a client accesses the
> >> server, it's fast.  The AEBS is also a server, but I cannot configure
> >> it with those options.  I can get a Mac to get fast access to the AEBS
> >> by configuring those TCP options on the client side.
> >>
> >> The problem pertains to having the Linux box access the disks on the
> >> AEBS.  In that case, I can't configure the TCP options on either the
> >> client (the Linux box in this case) or the server (the AEBS), so
> >> transfers are slow.  This is important, because a large part of my
> >> large file copies are between the Linux box and the AEBS.
> >>
> >> On the Linux server, SMB configuration has no impact on CIFS performance.
> >>
> >
> > If you wanted to experiment with it, it doesn't sound too hard to hack
> > up a patch to do so. You'd just need to add a kernel_setsockopt call to
> > the ipv4_connect (and/or ipv6_connect) functions. The place where the
> > sk_rcvtimeo/sk_sndtimeo get set is probably around the right spot. In
> > fact, those should probably be changed to kernel_setsockopt calls too.
> >
> > If that shows notable improvement then we could discuss the best way to
> > set up an option for it. Probably we'd want to flesh out the sockopt=
> > option.
> 
> I don't have much experience with Linux kernel drivers, so maybe you
> can give me some hints.
> 
> (1) There's a userspace component (mount.cifs).  Is the problem in there?

Nope. Sockets are opened and configured within the kernel.

> (2) Or is it in the kernel driver?

Yep.

> (3) How are the options parsed?  Is this done for us?  That is, do I
> need to look at existing data structures, or do I need to add some
> parsing code?

I wouldn't worry about option parsing at the moment. I'd just hardcode
those functions (ipv4_connect and/or ipv6_connect) to call
kernel_setsockopt for TCP_NODELAY. Once you've tested it, if you find
that it's helpful then you can worry about cleaning it up and adding a
mount option (or doing something else).

kernel_setsockopt is somewhat similar to its userspace counterpart.
There are examples of its use in other areas of the kernel.

> (4) Is it okay to discuss this on this list, or should I move to a
> different one?
> 

Yes. This is the place to discuss the linux-cifs client development.

-- 
Jeff Layton <[email protected]>