Re: SCTP <= 500 pps unless SCTP_NODELAY set (was: Expected SCTP DATA chunk per second performance)

> Unfortuantely it seems that the problems I reported in March 2020 on kernel 5.4.19
> still persist at least up to 5.10.46

I ran into this problem as well when doing some testing.
And the problem still exists as far as I can tell.
After looking into the details it seems to be related to pathmtu and
sndbuf size.
Since the messages sent out are quite small, the overhead for bundling
the outgoing data eats up the sndbuf before hitting the pathmtu size boundary.

This makes the send block with only 1 packet in flight,
which is not acked by the remote side until SACK Delay or number of packets
set with SACK freq, usually set to 2.
Sender is blocked by sndbuf and can't fill the Nagle bundle buffer
for the second packet to go out.

Verified send is blocked in sctp_wait_for_sndbuf by:
Small change to make the function traceable.

 /* Helper function to wait for space in the sndbuf.  */
-static int sctp_wait_for_sndbuf(struct sctp_association *asoc,
+static noinline int sctp_wait_for_sndbuf(struct sctp_association *asoc,
                                struct sctp_transport *transport,
                                long *timeo_p, size_t msg_len)

root@virtme-ng:/home/jonas/tmp# bpftrace -e
'kprobe:sctp_wait_for_sndbuf { @callstack[kstack] = count();}'
Attaching 1 probe...
^C

@callstack[
    sctp_wait_for_sndbuf+1
    sctp_sendmsg_to_asoc+362
    sctp_sendmsg+1619
    ____sys_sendmsg+376
    ___sys_sendmsg+153
    __sys_sendmsg+136
    do_syscall_64+270
    entry_SYSCALL_64_after_hwframe+119
]: 47


Then made a small change where the max size for bundling is
min(pathmtu, SCTP_DEFAULT_MAXSEGMENT).
Would such a patch be of interest?
Or add a max data chunk counter to cap on?

$ ./client -n 1000000
About to send 1000000 chunks of each 150 bytes
1000000 DATA chunks of 150 bytes each in  0.95 seconds: 1051674.12
DATA chunks per second

The problem only manifests itself when sending data where you can fill
the sndbuf before hitting pathmtu size.
Bumping the sndbuf to to approx 5,6x pathmtu should work as well.

-Jonas