Re: [Proftpd-user] mod_sftp vs sshd performance

I installed proftpd on a VM where I could tweak the settings more easily and got some trace data. It looks like the client (scp) is sending 32777 byte payloads, which I think is the default sftp buffer size: Here’s an small excerpt:

...
2019-05-08 18:37:01,500 [29914] <ssh2:19>: waiting for max of 600 secs while polling socket 0 for reading using select(2)
2019-05-08 18:37:01,500 [29914] <ssh2:3>: received SSH_MSG_CHANNEL_DATA (94) packet
2019-05-08 18:37:01,500 [29914] <ssh2:17>: processing 32768 bytes of data for channel ID 0
2019-05-08 18:37:01,500 [29914] <ssh2:19>: waiting for max of 600 secs while polling socket 0 for reading using select(2)
2019-05-08 18:37:01,500 [29914] <ssh2:20>: SSH2 packet len = 32796 bytes
2019-05-08 18:37:01,500 [29914] <ssh2:20>: SSH2 packet padding len = 18 bytes
2019-05-08 18:37:01,500 [29914] <ssh2:20>: SSH2 packet payload len = 32777 bytes
2019-05-08 18:37:01,500 [29914] <ssh2:19>: waiting for max of 600 secs while polling socket 0 for reading using select(2)
2019-05-08 18:37:01,501 [29914] <ssh2:20>: SSH2 packet MAC len = 8 bytes
2019-05-08 18:37:01,501 [29914] <ssh2:19>: waiting for max of 600 secs while polling socket 0 for reading using select(2)
2019-05-08 18:37:01,501 [29914] <ssh2:3>: received SSH_MSG_CHANNEL_DATA (94) packet
2019-05-08 18:37:01,501 [29914] <ssh2:17>: processing 32768 bytes of data for channel ID 0
2019-05-08 18:37:01,501 [29914] <ssh2:19>: waiting for max of 600 secs while polling socket 0 for reading using select(2)
2019-05-08 18:37:01,501 [29914] <ssh2:20>: SSH2 packet len = 32796 bytes
2019-05-08 18:37:01,501 [29914] <ssh2:20>: SSH2 packet padding len = 18 bytes
2019-05-08 18:37:01,501 [29914] <ssh2:20>: SSH2 packet payload len = 32777 bytes
2019-05-08 18:37:01,501 [29914] <ssh2:19>: waiting for max of 600 secs while polling socket 0 for reading using select(2)
2019-05-08 18:37:01,501 [29914] <ssh2:20>: SSH2 packet MAC len = 8 bytes
2019-05-08 18:37:01,501 [29914] <ssh2:19>: waiting for max of 600 secs while polling socket 0 for reading using select(2)
2019-05-08 18:37:01,501 [29914] <ssh2:3>: received SSH_MSG_CHANNEL_DATA (94) packet
2019-05-08 18:37:01,501 [29914] <ssh2:17>: processing 32768 bytes of data for channel ID 0
2019-05-08 18:37:01,501 [29914] <ssh2:19>: waiting for max of 600 secs while polling socket 0 for reading using select(2)
2019-05-08 18:37:01,501 [29914] <ssh2:20>: SSH2 packet len = 32796 bytes
2019-05-08 18:37:01,501 [29914] <ssh2:20>: SSH2 packet padding len = 18 bytes
...


Does the above look normal?

> On May 8, 2019, at 11:18 AM, Sergei Gerasenko <[email protected]> wrote:
> 
> Any idea why the below directives at the beginning of the main config file don’t generate anything in /var/log/proftpd/sftp_trace.log.
> 
> TraceLog /var/log/proftpd/sftp_trace.log
> Trace scp:20 sftp:20 ssh2:20
> 
>> On May 8, 2019, at 9:43 AM, Sergei Gerasenko <[email protected] <mailto:[email protected]>> wrote:
>> 
>> Hi Dieter,
>> 
>> Here are the details from the sftp log on the server:
>> 
>> Proftpd:
>> 
>> In the sftp log:
>> 
>> 2019-05-07 04:47:59,080 mod_sftp/1.0.0[37533]:  + Session client-to-server encryption: aes128-cbc
>> 2019-05-07 04:47:59,080 mod_sftp/1.0.0[37533]:  + Session server-to-client encryption: aes128-cbc
>> 2019-05-07 04:47:59,080 mod_sftp/1.0.0[37533]:  + Session client-to-server MAC: hmac-sha1
>> 2019-05-07 04:47:59,080 mod_sftp/1.0.0[37533]:  + Session server-to-client MAC: hmac-sha1
>> 2019-05-07 04:47:59,080 mod_sftp/1.0.0[37533]:  + Session client-to-server compression: none
>> 2019-05-07 04:47:59,080 mod_sftp/1.0.0[37533]:  + Session server-to-client compression: none
>> 
>> In the output of scp:
>> 
>> debug1: SSH2_MSG_KEXINIT sent
>> debug1: SSH2_MSG_KEXINIT received
>> debug1: kex: algorithm: ecdh-sha2-nistp256
>> debug1: kex: host key algorithm: ssh-rsa
>> debug1: kex: server->client cipher: aes128-ctr MAC: [email protected] <mailto:[email protected]> compression: none
>> debug1: kex: client->server cipher: aes128-ctr MAC: [email protected] <mailto:[email protected]> compression: none
>> debug1: kex: ecdh-sha2-nistp256 need=16 dh_need=16
>> debug1: kex: ecdh-sha2-nistp256 need=16 dh_need=16
>> debug1: sending SSH2_MSG_KEX_ECDH_INIT
>> debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
>> 
>> 
>> SSHD
>> 
>> In the output of scp:
>> 
>> debug1: Authenticating to internal102.ord.clientftp.core.cnvr.net:22 <http://ord.clientftp.core.cnvr.net:22/> as 'sgerasenko'
>> debug1: SSH2_MSG_KEXINIT sent
>> debug1: SSH2_MSG_KEXINIT received
>> debug1: kex: algorithm: curve25519-sha256
>> debug1: kex: host key algorithm: ecdsa-sha2-nistp256
>> debug1: kex: server->client cipher: [email protected] <mailto:[email protected]> MAC: <implicit> compression: none
>> debug1: kex: client->server cipher: [email protected] <mailto:[email protected]> MAC: <implicit> compression: none
>> debug1: kex: curve25519-sha256 need=64 dh_need=64
>> debug1: kex: curve25519-sha256 need=64 dh_need=64
>> debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
>> 
>> Thank you for responding!
>>   Sergei
>> 
>> 
>>> On May 8, 2019, at 9:04 AM, Dieter Bloms <[email protected] <mailto:[email protected]>> wrote:
>>> 
>>> Hello Sergei,
>>> 
>>> which ciphers were used, when you use sshd and proftpd ?
>>> You will see it on command line with scp/sftp -v
>>> 
>>> Be sure both connections use the same one.
>>> 
>>> 
>>> On Wed, May 08, Sergei Gerasenko wrote:
>>> 
>>>> Also, for some reason I’m having trouble enabling tracing for the sftp traffic. The sftp log is empty. I’ve put this in my main config at the very beginning:
>>>> 
>>>> # (http://www.proftpd.org/docs/howto/Tracing.html <http://www.proftpd.org/docs/howto/Tracing.html>)
>>>> TraceLog                        /var/log/proftpd/sftp_trace.log
>>>> Trace scp:20 sftp:20 ssh2:20
>>>> 
>>>> 
>>>> Attaching my proftpd -V and sftp config
>>>> 
>>>> Compile-time Settings:
>>>>  Version: 1.3.6 (stable)
>>>>  Platform: LINUX [Linux 3.10.0-957.1.3.el7.x86_64 x86_64]
>>>>  Built: Mon Oct 22 2018 18:48:06 UTC
>>>>  Built With:
>>>>    configure  '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--libexecdir=/usr/libexec/proftpd' '--localstatedir=/run/proftpd' '--disable-strip' '--enable-ctrls' '--enable-dso' '--enable-facl' '--enable-ipv6' '--enable-memcache' '--enable-nls' '--enable-openssl' '--disable-pcre' '--disable-redis' '--disable-xattr' '--enable-shadow' '--enable-tests=nonetwork' '--with-libraries=/usr/lib64/mysql' '--with-includes=/usr/include/mysql' '--with-modules=mod_readme:mod_auth_pam:mod_tls' '--with-shared=mod_sql:mod_sql_passwd:mod_sql_mysql:mod_sql_postgres:mod_sql_sqlite:mod_quotatab:mod_quotatab_file:mod_quotatab_ldap:mod_quotatab_radius:mod_quotatab_sql:mod_ldap:mod_ban:mod_wrap:mod_ctrls_admin:mod_facl:mod_load:mod_vroot:mod_radius:mod_ratio:mod_rewrite:mod_site_misc:mod_exec:mod_shaper:mod_geoip:mod_wrap2:mod_wrap2_file:mod_wrap2_sql:mod_copy:mod_deflate:mod_ifversion:mod_qos:mod_sftp:mod_sftp_pam:mod_sftp_sql:mod_tls_shmcache:mod_tls_memcache:mod_ifsession' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1  -m64 -mtune=generic' 'LDFLAGS=-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1  -m64 -mtune=generic'
>>>> 
>>>>  CFLAGS: -g2 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1  -m64 -mtune=generic -Wall -fno-omit-frame-pointer -Werror=implicit-function-declaration
>>>>  LDFLAGS: -L$(top_srcdir)/lib -Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -rdynamic -L/usr/lib64/mysql -L/usr/lib64/mysql -L/usr/lib64
>>>>  LIBS: -lacl  -lssl -lcrypto -lcap  -lmemcached -lmemcachedutil  -lssl -lcrypto  -lpam -lsupp -lnsl -lresolv -lresolv -lcrypt -ldl 
>>>> 
>>>>  Files:
>>>>    Configuration File:
>>>>      /etc/proftpd.conf
>>>>    Pid File:
>>>>      /run/proftpd/proftpd.pid
>>>>    Scoreboard File:
>>>>      /run/proftpd/proftpd.scoreboard
>>>>    Header Directory:
>>>>      /usr/include/proftpd
>>>>    Shared Module Directory:
>>>>      /usr/libexec/proftpd
>>>> 
>>>>  Info:
>>>>    + Max supported UID: 4294967295
>>>>    + Max supported GID: 4294967295
>>>> 
>>>>  Features:
>>>>    - Autoshadow support
>>>>    + Controls support
>>>>    + curses support
>>>>    - Developer support
>>>>    + DSO support
>>>>    + IPv6 support
>>>>    + Largefile support
>>>>    - Lastlog support
>>>>    + Memcache support
>>>>    + ncursesw support
>>>>    + NLS support
>>>>    - Redis support
>>>>    - Sodium support
>>>>    + OpenSSL support (FIPS enabled)
>>>>    - PCRE support
>>>>    + POSIX ACL support
>>>>    + Shadow file support
>>>>    + Sendfile support
>>>>    + Trace support
>>>>    - xattr support
>>>> 
>>>>  Tunable Options:
>>>>    PR_TUNABLE_BUFFER_SIZE = 1024
>>>>    PR_TUNABLE_DEFAULT_RCVBUFSZ = 8192
>>>>    PR_TUNABLE_DEFAULT_SNDBUFSZ = 8192
>>>>    PR_TUNABLE_ENV_MAX = 2048
>>>>    PR_TUNABLE_GLOBBING_MAX_MATCHES = 100000
>>>>    PR_TUNABLE_GLOBBING_MAX_RECURSION = 8
>>>>    PR_TUNABLE_HASH_TABLE_SIZE = 40
>>>>    PR_TUNABLE_LOGIN_MAX = 256
>>>>    PR_TUNABLE_NEW_POOL_SIZE = 512
>>>>    PR_TUNABLE_PATH_MAX = 4096
>>>>    PR_TUNABLE_SCOREBOARD_BUFFER_SIZE = 80
>>>>    PR_TUNABLE_SCOREBOARD_SCRUB_TIMER = 30
>>>>    PR_TUNABLE_SELECT_TIMEOUT = 30
>>>>    PR_TUNABLE_TIMEOUTIDENT = 10
>>>>    PR_TUNABLE_TIMEOUTIDLE = 600
>>>>    PR_TUNABLE_TIMEOUTLINGER = 10
>>>>    PR_TUNABLE_TIMEOUTLOGIN = 300
>>>>    PR_TUNABLE_TIMEOUTNOXFER = 300
>>>>    PR_TUNABLE_TIMEOUTSTALLED = 3600
>>>>    PR_TUNABLE_XFER_SCOREBOARD_UPDATES = 10
>>>> 
>>>> Thanks again,
>>>>  Sergei
>>>> 
>>>> P.S.: I noticed that forums.proftpd.org <http://forums.proftpd.org/> <http://forums.proftpd.org/ <http://forums.proftpd.org/>> has an invalid SSL cert. Can we chip in to fix that or is it prohibitively expensive?
>>>> 
>>>>> On May 7, 2019, at 3:05 PM, Sergei Gerasenko <[email protected] <mailto:[email protected]>> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> I’m running proftpd version 1.3.6. When comparing the sftp transfer speed between sshd and proftpd, I recently noticed that proftpd sftp transfers are half as fast. My test is:
>>>>> 
>>>>> 858M file
>>>>> same machine
>>>>> scp as the transfer client
>>>>> via sshd is 9.2 seconds
>>>>> via mod_sftp: 24 seconds
>>>>> 
>>>>> I’ve tried turning off compression and adjusting socket options but with practically no difference. My sftp config is basically this:
>>>>> 
>>>>> <VirtualHost ...>
>>>>> ...
>>>>> SFTPENgine on
>>>>> SFTPCompression delayed
>>>>> ...
>>>>> </VirtualHost>
>>>>> 
>>>>> Should the transfer speeds be comparable? Both sshd and proftpd are on the same machine.
>>>>> 
>>>>> Thanks for any ideas,
>>>>>  Sergei
>>>> 
>>> 
>>> 
>>>> _______________________________________________
>>>> ProFTPD Users List   <[email protected] <mailto:[email protected]>>
>>>> Unsubscribe problems?
>>>> http://www.proftpd.org/list-unsub.html <http://www.proftpd.org/list-unsub.html>
>>> 
>>> 
>>> -- 
>>> Gruß
>>> 
>>>  Dieter
>>> 
>>> --
>>> I do not get viruses because I do not use MS software.
>>> If you use Outlook then please do not put my email address in your
>>> address-book so that WHEN you get a virus it won't use my address in the
>>> From field.
>>> 
>>> 
>>> _______________________________________________
>>> ProFTPD Users List   <[email protected] <mailto:[email protected]>>
>>> Unsubscribe problems?
>>> http://www.proftpd.org/list-unsub.html <http://www.proftpd.org/list-unsub.html>
>

_______________________________________________
ProFTPD Users List   <[email protected]>
Unsubscribe problems?
http://www.proftpd.org/list-unsub.html