Re: High SFTP CPU usage

Alexander V. Lukyanov wrote:
> On Tue, May 11, 2004 at 01:37:51AM -0700, Andy Lutomirski wrote:
> 
>>>I have put out lftp-3.0.4rc1 at ftp://ftp.yar.ru/lftp/devel/ , please test.
>>
>>This is a lot better (CPU usage is down from 27% to 11%).
> 
> 
> lftp-3.0.4rc3 should be even faster thanks to Speedometer optimization.
> Please test.
> 
> --
>    Alexander.

Disclaimer: these results are patently absurd.  So I'll retest later.

The fast server that I've been testing against is down, so I tested 
against localhost.  I did a ~/test3/bin/lftp -c 'open sftp://localhost; 
get 040331.rm' (from a reiserfs partition to an XFS partition on another 
physical disk).  Both of these disks can sustain 40-60MBps.

I got a whopping 1.45MBps.  lftp used ~46% CPU.  ssh and sshd each used 
another 5% or so (so I had a lot of idle time).  I reran this with 
3.0.4-rc3, 3.0.4-rc1 with -O3, 3.0.4-rc1 with profiling and debug 
(attached), and dag's 3.0.3 build.  The results were indistinguisable. 
For reference, openssh's sftp got 12-13MBps with almost zero CPU -- ssh 
and sshd used all the rest.

For fun, I redid this with a 'put' instead of a 'get' and I got closer 
to 9MBps.  So something's funny here.

BTW, 3.0.4-rc4 seemed to use somewhat more user time than 3.0.4-rc1 for 
the put.

One possible issue is that I ran this test over another ssh connection 
over a wireless network.  So there could be some funny dependence on the 
terminal speed.  Except that getting with output redirected to /dev/null 
  didn't help.

lftp 3.0.4-rc3 put XFS-->reiserfs:

real    0m11.445s
user    0m1.709s
sys     0m0.539s

See rc3-put-profile.bz2

lftp 3.0.4-rc1 put XFS-->reiserfs:

real    0m11.254s
user    0m1.694s
sys     0m0.568s

See rc1-put-profile.bz2

The regression was more dramatic with -O3.

Enjoy :)

When the server I use comes back up, I'll retest.  If there are any 
other useful tests I can run, let me know.

--Andy

(bz2 this time b/c it compressed a lot better)