Re: Surprising observations on the NetBSD TCP stack

Thanks for the feedback see below.

On 2025-10-08 19:45, Michael van Elst wrote:
> [email protected] (Frank Kardel) writes:
>
>> I just got my Fiber 1GBit/sec internet connection (don't ask).
>> When testing the throughput with breitbandmessung.de I got a meager
>> 88Mbit/sec download rate.
> There are lots of things that do have an effect.
>
> Download speeds are mostly affected by local buffering.
> Upload speeds are mostly affected by congestion control.
>
I played with the buffersizes. It is still asymetric between gateway and 
notebook.

notebook -> gateway

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   935 Mbits/sec 0             sender
[  5]   0.00-10.00  sec  1.09 GBytes   934 Mbits/sec                  
receiver

gateway -> notebook (iperf3 -R)

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  1.02 GBytes   871 Mbits/sec 0             sender
[  5]   0.00-10.01  sec  1.01 GBytes   871 Mbits/sec                  
receiver

>> Testing local connectivity on the gigbit interfaces found:
>>         gateway->notebook xfer rate: 61201.87 KB/sec !!! should be more
>> like 113000 KB/sec
>>         notebook->gateway xfer rate: 113958.41 KB/sec
>> This begs the question what is limiting the transfer rate asymmetrically?
> Usually different network interfaces on either side, bad or nonexistent
> interrupt mitigation, bad use of hardware queues, etc..

Maybe there are issues in the igc driver. sending from there does not 
look completely perfect.

> This is two NetBSD machines (Xeon+wm vs Ryzen+wm) on the local network.
>
> iperf
> [  7]   0.00-10.00  sec  1.06 GBytes   914 Mbits/sec    0             sender
> [  7]   0.00-10.04  sec  1.06 GBytes   909 Mbits/sec                  receiver
>
> iperf -R
> [  7]   0.00-10.02  sec  1.07 GBytes   920 Mbits/sec    0             sender
> [  7]   0.00-10.00  sec  1.07 GBytes   919 Mbits/sec                  receiver
>
>
>> The NetBSD performance is not competative at all - barely able to keep
>> up with ADSL modems, though
>> the send rate is ok.
> I don't have a fibre connection, so testing on the internet gets
> difficult.
>
> Instead I put an RPI running Linux as a "WAN simulator" between
> two NetBSD hosts to analyze the issues. I now have a couple of
> changes in the TCP stack and workaround in the wm driver to
> get almost 1Gbit/s over a simulated WAN with 200ms TTL.
>
> This also helped a bit with internet upload speed, but at
> 40Mbit/s (VDSL) the effect is small.
>
Good to hear.
>> net.inet.tcp.recvbuf_max = 262144
>> net.inet.tcp.sendbuf_max = 262144
> You may need larger buffers (and larger increments).
>
I now have

net.inet.tcp.keepinit = 150
net.inet.tcp.recvbuf_auto = 1
net.inet.tcp.recvbuf_inc = 131072
net.inet.tcp.recvbuf_max = 1048576
net.inet.tcp.sendbuf_auto = 1
net.inet.tcp.sendbuf_inc = 131072
net.inet.tcp.sendbuf_max = 1048576