RE: Please read - proposed WG termination

At 12:06 AM 9/3/2005, Dror Goldenberg wrote:
>
>From: Michael Krause [mailto:[email protected]]
>Sent: Saturday, September 03, 2005 1:24 AM
>Large send can be implemented entirely in software and local to the 
>injection point making it rather trivial to accomplish whille reaping the 
>performance benefits.
>
>[dg] This is really interesting ! Why would the performance of a NIC 
>driver doing LSO/TSO in SW be better than the TCP stack doing it on its 
>own ? I tend to believe that the TCP stack is highly optimized, especially 
>in the mainstream path where packet is being constructed. When moving the 
>LSO/TSO from the OS to the NIC driver, you essentially take the same piece 
>of SW and just move it somewhere else in the stack. Am I missing something ?

The primary reason LSO has benefit is it reduces the number of passes made 
through the network stack - even a very tuned stack repeated multiple times 
still has cost.  SAR occurs at the bottom.  Yes there is increased 
performance from off-loading the SAR functionality to the NIC but having a 
driver create N headers and attach N data blocks which it already has to 
mapped, etc. to work descriptors isn't that much additional work compared 
to its normal job.

>  Large receive generally needs some hardware assist to work well but 
> given a large percentage of the protocol off-load implementations rely 
> upon firmware for various operations, it will vary as to whether there is 
> a legitimate execution problem or not.
>
>[dg] There was a paper at OLS talking about LRO 
>(<http://www.linuxsymposium.org/2005/linuxsymposium_procv1.pdf>http://www.linuxsymposium.org/2005/linuxsymposium_procv1.pdf).
>My recollection from the talk is that also LRO implementation requires 
>changes from the OS too. For example, in TCP, you're supposed to send an 
>ACK every other incoming segments. If you have LRO implemented in the NIC 
>driver, then you loose track of segment count, unless you can pass that 
>along to the OS (new indication).

Really have to go back to the end-to-end argument quite frankly and whether 
a TCP ACK should indicate receipt to the host memory subsystem or receipt 
at to the NIC (some would argue that it should be receipt to the 
application but majority of implementations assume just to the host memory 
subsystem.  The issue for LRO is really about where state is tracked which 
is ideally in the host stack thus the ACK is issued from the host 
stack.  It is correct that this presents some challenges but isn't that 
difficult to comprehend in the network stack as one knows the MSS which is 
on a per connection basis and can generate appropriate ACKs.  That is 
fairly trivial but is a network stack implementation change and there are a 
variety of ways to implement network / driver communications to deliver 
performance.

The CM spec is basically stating that the segments are large thus the host 
stack will only communicate via large MSS where both sides see large 
packets.  However, majority of workloads are not large buffer focused - 
only a subset which still intermix large and small exchanges quite 
often.  I agree that there will be performance improvements seen in micro 
benchmarks and perhaps a selective set of workloads but given the majority 
of workloads over an IB fabric are not dominated by their network stack 
implementations, only a small sliver of customers will see real benefit and 
only if these customers are running fully loaded systems (likely for a 
number of HPC applications but hardly a drop in the bucket for the 
commercial markets).

>There is another issue which has to do with duplication of TCP stack work 
>while doing LRO in SW. The paper describes performing many of the TCP/IP 
>checks in the NIC driver in order to take an LRO decision... So, I'm not 
>sure whether this doesn't degrade mainstream operation.
>There is also a strong assumption on locality for LRO. The paper describes 
>coalescing of subsequent frames. It obviously does great on synthetic 
>benchmarks. To some systems, the locality assumption may not perform well.
>
>  The same cannot be said for the OS infrastructure changes which is what 
> I and a couple of others have touched upon.
>
>[dg] Putting aside for a second the different MTU issue. Can't we just run 
>ipoib-cm on any OS that supports Ethernet jumbo frames ? If yes, then we 
>can say that ipoib-cm cleanly falls into the domain of the IHV NIC driver, 
>and can therefore be supported in a larger variety of OSes.

Anything is possible but not everything will be supported.   There is cost 
with every spec permutation in terms of validation, support, etc.  Adding 
yet another spec to the plethora already in the IB suite is just a low ROI 
cost adder to me.  Again, feel free to continue the work.  My 
recommendation to my company is to spend resources on where there is 
customer visible value-add that goes beyond the micro benchmark.  My push 
is to get the RDMA infrastructure in place along with the ULP to take 
advantage of it and not work to make IP over IB better since it already 
meets the original intent when I spoke at the first IETF BOF to get the 
workgroup chartered.  The workgroup has met its objectives and should 
cease.  I suspect the hum in the virtual room may disagree but such is life.

Mike

_______________________________________________
IPoverIB mailing list
[email protected]
https://www1.ietf.org/mailman/listinfo/ipoverib