Re: Connection reset

On Thu, Aug 28, 2008 at 04:03:06AM -0700, Robert Gibson wrote:
> I'm coming back to this very old thread, since after a few years and
> a couple of changes of network config we have started to get the
> problem again, but this time we understand a bit more about it.
> I've also had a few off-list follow-ups which I wasn't able to reply
> helpfully to, maybe re-opening the thread can help some more.

FWIW, here is a link to the previous messages of this thread in the
archives:

http://archives.java.sun.com/cgi-bin/wa?A2=ind0603&L=RMI-USERS&P=377

> This time round I was able to occasionally reproduce the problem
> with a small test case, and take packet dumps.  The dumps showed
> that the server machine wasn't responding correctly to MTU
> negociation and certain packets were just getting unceremoniously
> (but consistently) swallowed.

Could you elaborate?  Do you mean that the server was ignoring ICMP
messages from routers that fragmentation was needed instead of sending
the data in smaller packets, or something else?  Which packets were
getting swallowed where?

Are there non-trivial network elements between the client and server
involved here, like a firewall?  Firewalls configured to drop ICMP
packets have been known to interfere with path MTU discovery.

>> Have you gotten thread dumps from the client and server VMs during
>> this two hour period?  "netstat" output on the client and server
>> machines during this period?
> 
> I have thread dumps, but I'm not sure they are very interesting.
> I'll put them anyway, since you asked.

One thing I notice is that in this case, the client thread is blocked
attempting to read the very first byte of the remote invocation
response, whereas previously the connection reset exception showed the
client somewhere in the middle of deserializing the returned HashMap.
This may not be a significant distinction, though.

I'm guessing that the server thread is blocked trying to write the
contents of the HashMap because the TCP connection's output buffers
are full.

> After about 10 minutes, this server thread disappears.
> Netstat on both machines shows an established connection at the
> beginning, although after a while only the client machine shows an
> established connection.

I'm guessing that the server side considered the TCP connection broken
at this point because all retransmissions of a data segment timed out,
i.e. no acknowledgment received.  (As Esmond said, server-side RMI
logging should show an exception at that point.)  Perhaps these data
segments were never delivered to the client because they exceeded the
MTU of some hop, but the resulting ICMP messages necessary for path
MTU discovery were dropped?

> This time I have some slightly different questions:
> 
> 1) Where should I look to try and sort out why the MTU negociation
> isn't happening properly, in the O/S, Java or the network setup?
> Does Java rewrite any of the network stack that would be implicated
> in this kind of problem or is it 100% based on the underlying O/S
> implementation?  (We're using Sun JDK 1.6.0_06 and RH Linux FC6
> kernel 2.6.18 on x86_64.)

As Esmond said, this is below the Java (and thus RMI) level.

I can't say exactly where to look without knowing more about the
network configuration, but I would try to understand at what point
packets are getting dropped, and if that seems consistent with the
MTUs of the adjacent networks, and whether ICMP messages are being
generated by the element dropping the packets, and if so, what is
happening to those messages.

> 2) My network admins tell me they are surprised to see that the RMI
> packets are all sent with the Don't Fragment bit set.  (This
> apparently explains why we are only seeing the problem with our RMI
> traffic and not SMTP, etc.)  Is there any way to adjust this?

Again as Esmond said, this is not controllable at the Java (and thus
RMI) level.

The "do not fragment" bit being set for TCP packets is not surprising
at all-- in fact, it is essential to how path MTU discovery works (see
RFC 1191).  So, I would say that that's not a cause of the problem.

> 3) While I wait for the underlying problem to be fixed, in the face
> of systematically dropped packets, would you still advise adjusting
> the property sun.rmi.transport.tcp.responseTimeout assuming we can
> find a value that allows "normal" calls to complete?

If it is helpful, then I suppose that it is fine to do.  But it
doesn't seem related to the problem.  A possible workaround, although
hacky, could be to set a server-side socket factory that wraps the
socket output stream in a layered output stream that (re)buffers the
data being written and forces all writes to the underlying socket
output stream to be less than the maximum TCP data payload for the
true path MTU of the connection (whatever that really is).

-- Peter

===========================================================================
To unsubscribe, send email to [email protected] and include in the body
of the message "signoff RMI-USERS".  For general help, send email to
[email protected] and include in the body of the message "help".

For a list of frequently asked RMI questions please refer to:
http://java.sun.com/j2se/1.3/docs/guide/rmi/faq.html

To view past RMI-USERS postings, please see:
http://archives.java.sun.com/archives/rmi-users.html