Re: Software Caused Connection Abort (10053) on Client

The more I dig into this the more it just seems to be a possible
artifact of using a SmartProxy around our services stubs. I can make it
fail pretty reliably by restarting the server that the agents are a
client of. After the restart, the agents will contain an invalid stub.
The first time the Proxy invokes a method on that cached stub I will get
this error on the server side:

 java.rmi.NoSuchObjectException: no such object in table
        at sun.rmi.transport.Transport.serviceCall(Transport.java:112)
        at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
        at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.ja
va:701)
        at java.lang.Thread.run(Thread.java:534)

This results in a NoSuchObjectException on the agent (as you would
expect). So the proxy determines our stub is out of date and uses the
Registry on the server to get a fresh one and retries the method
invocation. This time there is an invalid op error in the TCPTransport
layer of the server and the client gets this exception:

 Caused by: java.rmi.UnmarshalException: Error unmarshaling return
header; nested exception is: 
        java.net.SocketException: Connection reset
        at
sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:203
)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
        at
com.tripwire.space.services.server.ServerService_Stub.exec(Unknown
Source)
      <deleted>
        ... 2 more
 Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:168)
        at com.sun.net.ssl.internal.ssl.InputRecord.a(DashoA12275)
        at com.sun.net.ssl.internal.ssl.InputRecord.read(DashoA12275)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA12275)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA12275)
        at com.sun.net.ssl.internal.ssl.AppInputStream.read(DashoA12275)
        at
java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
        at
java.io.BufferedInputStream.read(BufferedInputStream.java:201)
        at java.io.DataInputStream.readByte(DataInputStream.java:331)
        at
sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:189
)
        ... 15 more

Now all of this seems to be inline with bug #4716483. But what I don't
understand is this. Don't tons of people use Reloading Proxies for
service stubs? Why isn't this bug rampant with them? Also, the call we
are making on the server does not have large arguments, basically one
object (with no instance data), and another remote stub to use as a call
back to the agent. Also note that this is an RMISSLSocketFactory (not
sure if that figures into the argument size issue).

Finally, I see two ways to address this:

        1.) Never cache stubs on the agent, get rid of the reloading
proxy.
        2.) Modify the reload proxy so that if the first exception is a
NoSuchObjectException retry like normal. If the next exception is a
Connection reset exception, retry yet another time assuming this is an
artifact of bug #4716483.

Any idea on which approach is best? Since this is so rare, #2 would
minimize network traffic (for registry lookups) when the system is in a
good state. So it seems like a better choice at the moment.

Also, for your scrutiny, here is the offending proxy code, is there
something inherently wrong with our initial retry logic?

    private class SmartHandler implements InvocationHandler
    {
        /** The type we use to lookup the RMI stub from the Registry */
        private Class m_type;

        /** The real RMI stub */
        private Object m_stub;

        public SmartHandler( Class type ) throws RemoteException,
NotBoundException
        {
            m_type = type;

            // Look up the stub in the registry
            m_stub = getRegistry().lookup( m_type.getName() );
        }

        /** {@inheritDoc} */
        public Object invoke( Object proxy, Method method, Object[] args
) throws Throwable
        {
            // Try and invoke the method, allowing for a retry if
applicable.
            return invokeMethod( method, args, true );
        }

        private Object invokeMethod( Method method, Object[] args,
boolean retry )
            throws Throwable
        {
            try
            {
                // Try to make the RMI method invocation and return the
result.
                return method.invoke( m_stub, args );
            }
            catch (Throwable e)
            {
                e.printStackTrace();

                Throwable realThrowable = e;

                // The real exception is almost always wrapped, but we
want the original.
                if ( e instanceof InvocationTargetException &&
e.getCause() != null )
                     realThrowable = e.getCause();

                if ( retry )
                    reloadWrapped();

                if ( retry && shouldRetryOn( realThrowable ) )
                {
                    return invokeMethod( method, args, false );
                }
                else
                {
                    throw realThrowable;
                }
            }
        }

        private void reloadWrapped()
        {
            try
            {
                // Relook it up in the registry
                m_stub = getRegistry().lookup( m_type.getName() );
            }
            catch (Throwable e)
            {
                e.printStackTrace();
                // If we can't get a new one, we just stick with what
we've got.
            }
        }

        private boolean shouldRetryOn( Throwable e )
        {
            // This one normally happens if the JVM of the remote object
has restarted.
            if ( e instanceof NoSuchObjectException )
                return true;

            // Otherwise we shouldn't risk a retry.
            return false;
        }
    }

#getRegistry just returns the real RemoteRegistry. The local caching
registry that the agents use looks basically like this:

    public synchronized Object getService( final Class type ) throws
RemoteException
    {
        // look in the cache first
        Object service = m_serviceCacheByInterface.get( type );
        if ( service != null )
            return service;

        try
        {
            service = 
                Proxy.newProxyInstance( type.getClassLoader(),
                    new Class[] { type },
                    new SmartHandler( type ) );

            // Now cache it for subsequent retrievals
            m_serviceCacheByInterface.put( type, service );

            return service;
        }
        catch ( ... );
    }

Anything look unreasonable here? The only thing I'm going to change is
to allow a third retry if the second retry resulted in a Connection
reset error. Thoughts?

R.

-----Original Message-----
From: Bob Scheifler [mailto:[email protected]] 
Sent: Tuesday, November 29, 2005 6:58 AM
To: Robert DiFalco
Cc: [email protected]
Subject: Re: Software Caused Connection Abort (10053) on Client

> In the case of 4716483 wouldn't I have seen something on the client 
> besides just the 10053 socket error?

Not if the arguments of the call are sufficiently large.  Read the 2nd
and 3rd paragraphs of the bug report again.  The server may have
marshalled the exception, but it will immediately process what it thinks
is the next call, see the bad transport op, and close the connection.
The client could still be marshalling its arguments, and not yet gotten
around to reading the exception that was marshalled.

- Bob

===========================================================================
To unsubscribe, send email to [email protected] and include in the body
of the message "signoff RMI-USERS".  For general help, send email to
[email protected] and include in the body of the message "help".

For a list of frequently asked RMI questions please refer to:
http://java.sun.com/j2se/1.3/docs/guide/rmi/faq.html

To view past RMI-USERS postings, please see:
http://archives.java.sun.com/archives/rmi-users.html