Re: Increasing TCP TSO size support

On Fri, Feb 2, 2024 at 1:21 AM Scheffenegger, Richard <[email protected]>
wrote:

>
> Hi,
>
> We have run a test for a RPC workload with 1MB IO sizes, and collected the
> tcp_default_output() len(gth) during the first pass in the output loop.
>
> In such a scenario, where the application frequently introduces small
> pauses (since the next large IO is only sent after the corresponding
> request from the client has been received and processed) between sending
> additional data, the current TSO limit of 64kB TSO maximum (45*1448 in
> effect) requires multiple passes in the output routine to send all the
> allowable (cwnd limited) data.
>
> I'll try to get a data collection with better granulariy above 90 000
> bytes - but even here the average strongly indicates that a majority of
> transmission opportunities are in the 512 kB area - probably also having to
> do with LRO and ACK thinning effects by the client.
>
> With other words, the tcp output has to run about 9 times with TSO, to
> transmit all elegible data - increasing the FreeBSD supported maximum TSO
> size to what current hardware could handle (256kB..1MB) would reduce the
> CPU burden here.
>
>
> Is increasing the sofware supported TSO size to allow for what the NICs
> could nowadays do something anyone apart from us would be interested in (in
> particular, those who work with the drivers)?
>
Reposted after joining freebsd-net@...

 A factor here is the if_hw_tsomaxsegcount limit. For example, a 1Mbyte NFS
write request
or read reply will result in a 514 element mbuf chain. Each of these
(mostly 2K mbuf clusters)
are non-contiguous data segments. (I suspect most NICs do not handle this
many segments well,
if at all.)

The NFS code does know how to use M_EXTPG mbufs (for NFS over TLS, for the
ktls), but I do not
know what it would take to make these work for non-KTLS TSO?
I do not know how the TSO loop in tcp_output handles M_EXTPG mbufs.
Does it assume each M_EXTPG mbuf is one contiguous data segment?
I do see that ip_output() will call mb_unmapped_to_ext() when the NIC does
not have IFCAP_MEXTPG set.
(If IFCAP_MEXTPG is set, do the pages need to be contiguous so that it can
become
a single contiguous data segment for TSO or ???)

If TSO and the code beneath it (NIC and maybe mb_unmapped_to_ext() being
called) were to
all work ok for M_EXTPG mbufs, it would be easy to enable that for NFS
(non-TLS case).

I do not want to hijack this thread, but do others know how TSO interacts
with M_EXTPG
mbufs?

rick


> Best regards,
>
>   Richard
>
>
>
>
> tso size (transmissions < 1448 would not be accounted here at all)
>
>                     # count
>
> <1000 0
> <2000 23
> <3000 111
> <4000 40
> <5000 30
> <7000 14
> <8000 134
> <9000 442
> <10000 9396
> <20000 46227
> <30000 25646
> <40000 33060
> <60000 23162
> <70000 24368
> <80000 19772
> <90000 40101
> >=90000 75384169
> Average: 578844.44
>
> CAUTION: This email originated from outside of the University of Guelph.
> Do not click links or open attachments unless you recognize the sender and
> know the content is safe. If in doubt, forward suspicious emails to
> [email protected].
>
>