Re: LWP::UserAgent and TCP_NODELAY

Michael Curtis <[email protected]> writes:

> Hello, I have a LWP application that POSTs thousands of small files to
> a mod_perl server. All POSTs *under* a certain size (around 3k) seem
> to take about 200x as long as those over. In this case about 0.09
> seconds as opposed to 0.004 seconds.
> 
> After much debugging I narrowed the issue down to TCP latency and
> discovered I can get around the problem by setting TCP_NODELAY on the
> socket.
> 
> My workaround is redefining the LWP::Protocol::http::_check_sock()
> method and adding a setsockopt($sock, 6, 1, pack("l", 1)). This tweak
> significantly improves performance on small POSTs but feels like a
> total hack.
> 
> Can I be sure that the _check_sock() method will always be there? The
> function itself doesn't do anything so I assume its just a hook. Is
> there another, better way to do this? Thanks much.

This message was written in July, so I'm sorry about the long delay
before answering it.  I still hope some might find the example code
below useful.

The _check_sock hook will always be there, as it's part of the
(undocumented) subclassing API.  It is for instance overridden by the
LWP::Protocol::https module.  Just redefining the function is a hack
and might disable code if the default _check_sock() would start do
something in the future, so I don't recommend that approach.

This is how I would suggest you do it by subclassing the protocol
handler:

    #!perl -w
    
    use strict;
    use LWP::UserAgent;
    
    {
       package MyHTTP;
       use base 'LWP::Protocol::http';
       LWP::Protocol::implementor(http => __PACKAGE__);
    
       package MyHTTP::Socket;
       use base 'LWP::Protocol::http::Socket';
       use Socket qw(IPPROTO_TCP TCP_NODELAY);
    
       sub new {
           my $class = shift;
           my $sock = $class->SUPER::new(@_);
           if ($sock) {
               setsockopt($sock, IPPROTO_TCP, TCP_NODELAY, 1);
           }
           return $sock;
       }
    }
    
    # test it
    my $ua = LWP::UserAgent->new;
    my $res = $ua->post("http://example.com", [foo => 1]);
    print $res->as_string;

The MyHTTP class just subclass from LWP::Protocol::http and register
itself as the one to handle 'http' requests for LWP.  MyHTTP will as a
subclass of LWP::Protocol::http by default create sockets of the class
'MyHTTP::Socket' and we also provide this class and override its
constructor to set the socket options we want after the socket is
created.

Regards,
Gisle