Re: Poor TCP performance as latency increases

On 2/1/25 06:24, Jared McNeill wrote:

> It seems that there is a performance issue in the TCP stack, and I’m
> hoping someone else can reproduce my findings as I’m not an expert in
> this area.
> 
> Simple test: Download a file from a web server. As latency between
> the client and server increases, there is a drastic reduction in
> throughput. I don’t see this with other (Linux, macOS) TCP stacks.
> 
> To rule out any issues with the NIC driver (I’ve tried both wm(4) and
> ixg(4) here), I can fetch a file hosted on a local Linux NAS and
> performance is reasonable. As the latency to the web server increases
> (20ms, 30ms), we see a drop off in performance
My guess is the same as Havard's:
On 2/8/25 05:31, Havard Eidnes wrote:
> If you find that your performance scales inversely with RTT, that
> is probably the issue: your TCP sessions will then be "window
> size limited".

That is likely what is going on.

Bandwidth-delay product (BDP) refers to the product of a data link’s
capacity (in bits per second) and its propagation time (in seconds).

A network with a large bandwidth-delay product is commonly known as a
long fat network (LFN).

* A high bandwidth-delay product is an important problem case in the
design of protocols such as Transmission Control Protocol (TCP).

* TCP requires a receiver to send messages (ACKs) back to the sender to
acknowledge receipt of data.

* The time elapsed between sending a packet and receiving the
acknowledgement is called the round-trip time (RTT).

* A TCP receiver has a receive window, the size of which is announced to
the sender. The sender may send up to that many bytes of data before
seeing any acknowledgment.

* If the amount of sent but unacknowledged data exceeds the receive
window size, the sender will stop sending, and wait for some to be
acknowledged.

* TCP can only achieve optimum throughput if a sender can send a
sufficiently large quantity of data before being required to stop and
wait until an acknowledgement message is received from the receiver.

* In order for a single TCP session to utilize the full bandwidth of a
link it should at all times have within the transmission path a full RTT
worth of sent but not yet acknowledged data.

Example:
* The end-to-end delay for a cross-country fiber optic link is about 30 ms.

* The bandwidth-delay product of a cross-country OC-48 (2.4 Gb/s) fiber
link would be .030 s * 2400 Mb/s = 72 megabits.

* This means up to 9 megabytes can be in transit between the two
endpoints.

Not all operating systems have TCP stacks which handle the LFN
problem well (having large enough TCP receive window buffers, and
adapting TCP window size adjustments to the problem). Maybe NetBSD does
not handle it in a way which results in good performance. It's a
complicated problem to solve.

A pair of packet captures (one from each end) will reveal exactly what
is going on.