Re: Increasing TCP TSO size support

On Fri, Feb 2, 2024, at 9:05 PM, Rick Macklem wrote:
> > But the page size is only 4K on most platforms.  So while an M_EXTPGS mbuf can hold 5 pages (..from memory, too lazy to do the math right now) and reduces socket buffer mbuf chain lengths by a factor of 10 or so (2k vs 20k per mbuf), the S/G list that a NIC will need to consume would likely decrease only by a factor of 2.  And even then only if the busdma code to map mbufs for DMA is not coalescing adjacent mbufs.  I know busdma does some coalescing, but I can't recall if it coalesces physcally adjacent mbufs.
> 
> I'm guessing the factor of 2 comes from the fact that each page is a
> contiguous segment?

Actually, no, I'm being dumb.  I was thinking that pages would be split up, but that's wrong.  Without M_EXTPGS, each mbuf generated by sendfile (or nfs) would be an M_EXT with a wrapper around a single 4K page.  So the scatter/gather list would be exactly the same.

The win would be if the pages themselves were contiguous (which they often are), and if the bus_dma mbuf mapping code coalesced those segments, and if the device could handle DMA across a 4K boundary.  That's what would get you shorter s/g lists.

I think tcp_m_copy() can handle this now, as if_hw_tsomaxsegsize is set by the driver to express how long the max contiguous segment they can handle is.

BTW, I really hate the mixing of bus dma restrictions with the hw_tsomax stuff.  It always makes my head explode..

Drew