Re: The tcpip layer is really frustrating me

Keith Whitwell wrote:

> Mike Houston wrote:
>
>> The main issue here is time.  I'll be at ATI this summer, so I don't 
>> have the time to deal with a network layer rewrite just now.
>>
>> The main issue with the tcpip layer is how we handle mothership 
>> connections.  We really should use special calls for mothership 
>> connections so that we don't pollute the rest of the network layer.  
>> The mothership connections do not stay live, but the tcpip layer 
>> still checks them in a tight loop.  This isn't a huge hit, but is 
>> does have an impact on other layers.
>
>
> I don't understand - why are connections that are known not to be live 
> ever examined again?

Because num_conns at line ~782 also counts mothership connections made 
at startup.  Usually, the connections are dropped and we bail out on the 
connection at ~line 785.  However, we still run through all connections 
and do a pointer dereference.  Where things get interesting are in non 
connection-based layers like IB, GM, and maybe Quadrics (Joel Welling 
would be the person to ask here).  The problem is that the connections 
are NOT closed becuase the connection semantics are being handled 
through the mothership.  The tcpip layer holds a connection open so that 
we can try to cleanly take down other nodes.

We do need to monitor these, but since data is not being sent, we get 
stuck on timeouts in the tcpip layer which creams performance.

>
>> At the moment, I have some really hacky code that forces the tcpip 
>> layer to bypass if the only connections are mothership connections.  
>> This helps to speed things up a little. 
>
>
> Can you post that code?  Or send it to me privately if you'd rather?

I won't have access to the code until this weekend as I'm officially at 
ATI this summer. 

Basically I hacked the conn structs to mark whether or not things were 
mothership connections.  If they were, I skip over them in crTCPIPRecv.  
This does speed things up on IB and GM, but completely fubars their 
ability to cleanly exit when a node is killed.  I have to manually kill 
all of the crservers/crappfakers when this happens.  This is obviously 
not an acceptable solution.

What would be nice to see would be a mothership connection library.  
This would *hopefully* remove some of the complexity from the tcpip 
layer, and allow us to better handle faking connection managers.  The 
tcpip layer is currently a little brittle because of all of the 
hackiness to get the mothership to behave.

-Mike


-------------------------------------------------------
This SF.Net email sponsored by Black Hat Briefings & Training.
Attend Black Hat Briefings & Training, Las Vegas July 24-29 - 
digital self defense, top technical experts, no vendor pitches, 
unmatched networking opportunities. Visit www.blackhat.com