Re: Software Caused Connection Abort (10053) on Client

Bob, 

Thanks for all your help. We've been running load tests for several days
now (with various combinations of restarts causing stubs to get new
ObjIDs) with no more problems. The support on this list is pretty
amazing. We are well on our way to switching out JRMP for JERI. For JERI
this issue will be largely mitigated because we will be using well-known
ObjIDs for our services and many will not require DGC. We will be
setting up some tests soon to compare max open socket handles, max
thread count, average number of socket calls, etc to compare JERI and
JRMP. When we have some results I will summarize them over on the Jini
list.

>> ...if you're willing to use implementation-specific
>> knowledge and dig into stack traces...

Well, I'm not happy to, but we control the distribution of our JRE so it
is better to have things working properly.

FWIW, I think the same bug (once I fixed the retry conditions) had moved
its effect over to the registry lookup with this exception:

java.rmi.ConnectIOException: error during JRMP connection establishment;
nested exception is: 
        java.net.SocketException: Connection reset
        at
sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:274)
        at
sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)
        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:313)
        at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
      ....

So I added a retry to the reloading code as well, which seemed to remedy
the problem. Here is our final ReloadHandler code in case anyone can
benefit from it.

    /**
     * A proxy that should wrap cached remote stubs and reload them on
invocation
     * errors that indicate the cached stub is stale. This code has been
written
     * to specifically deal with
     * http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4716483.
     */
    class ReloadHandler implements InvocationHandler
    {
        /** The type we use to lookup the RMI stub from the Registry */
        private final Class m_type;

        /** The real RMI stub */
        private Object m_wrapped;
        
        /** The registry */
        private final Registry m_registry;

        public ReloadHandler( 
            final Registry registry
            final Class type ) throws RemoteException, NotBoundException
        {
            /* NOTE: introduce an abstract class here to key stubs on
            something other than type */
            m_type = type; 
            m_registry = registry;

            // Look up the stub in the registry
            refreshWrapped();
        }

        /** {@inheritDoc} */
        public Object invoke( Object proxy, Method method, Object[] args
) throws Throwable
        {
            // Try and invoke the method, allowing for a retry if
applicable.
            return invokeMethod( method, args, true );
        }

        /**
         * This methods attempt to perform the invocation on the stub
and return the
         * result to the caller.  If the retry parameter is true it may
attempt to
         * perform the invocation again if the first attempt fails and 
         * {@link #shouldRetry} returns true.
         *
         * @param method the method to invoke
         * @param args it's arguments
         * @param retry true to retry, otherwise false
         * @return the method invocation result
         * @throws Throwable
         */
        private Object invokeMethod( Method method, Object[] args,
boolean retry )
            throws Throwable
        {
            try
            {
                // Try to make the RMI method invocation and return the
result.
                return method.invoke( getWrapped(), args );
            }
            catch ( Throwable e )
            {
                if ( TRACE )
                    e.printStackTrace();

                Throwable realThrowable = e;

                // The real exception is almost always wrapped
                // and we want the original.
                if ( e instanceof InvocationTargetException 
                    && e.getCause() != null)
                {
                     realThrowable = e.getCause();
                }

                if ( retry )
                    refreshWrapped();

                if ( retry && shouldRetryOn( realThrowable ) )
                {
                    // We only want to retry once so on this attempt we
set the 'retry'
                    // parameter to false.
                    return invokeMethod( method, args, false );
                }
                else
                {
                    throw realThrowable;
                }
            }
        }

        private synchronized Object getWrapped()
        {
            return m_wrapped;
        }
            
        /**
         * Attempts to assign m_wrapped to a fresh stub from the
Registry, but 
         * eats any exceptions that might be thrown. You can override
this to
         * use custom load behavior. There is clearly an abstract class
here.
         */
        protected synchronized void refreshWrapped() 
            throws RemoteException, NotBoundException
        {
            // block access to m_wrapped until we are done.
            
            for ( int tries = 1;; ++tries )
            {
                try
                {
                    // Relook it up in the registry
                    m_wrapped = getRegistry().lookup( m_type.getName()
);
                    return;
                }
                catch ( ConnectIOException e )
                {
                    // We sometimes get these, assuming bug #4716483 and
retry...
                    if ( tries == 2 )
                        throw e;

                    try { Thread.sleep( 500 ); } catch ( Exception ie )
{/**/}

                    if ( TRACE )
                        e.printStackTrace();
                }
            }
        }

        /**
         * @param e The <code>Throwable</code> to check.
         * @return true if the caller should reattempt the invocation.
         */
        protected boolean shouldRetryOn( final Throwable e )
        {
//
// NOTE: 
// Uncomment if you don't mind using JDK version specific code. With
this
// commented out, there will be side-effect of the bug
//     http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4716483
// that will not be handled.
//
//            if ( e instanceof UnmarshalException )
//            {
//                if ( e.getMessage().indexOf( "return header" ) == -1 )
//                    return false;
//
//                StringWriter swriter = new StringWriter();
//                PrintWriter pwriter = new PrintWriter( swriter );
//                e.printStackTrace( pwriter );
//                return 
//                    swriter.getBuffer().indexOf( 
//                        "releaseOutputStream" ) != -1;
//            }
//
            return ( 
                e instanceof NoSuchObjectException 
                    || e instanceof MarshalException );
        }
    }


-----Original Message-----
From: Bob Scheifler [mailto:[email protected]] 
Sent: Monday, December 05, 2005 8:17 AM
To: Robert DiFalco
Cc: [email protected]
Subject: Re: Software Caused Connection Abort (10053) on Client

> Hmmm...but with a default SSL/RMI Socket factory, is it possible that 
> the server would have invoked the call before the client had flushed 
> the output stream? I was thinking of something very hacky like this:
>       ...

Hmm, if you're willing to use implementation-specific knowledge and dig
into stack traces, I would think that if the exception came from
releaseOutputStream then the upcall has not occurred.

- Bob

===========================================================================
To unsubscribe, send email to [email protected] and include in the body
of the message "signoff RMI-USERS".  For general help, send email to
[email protected] and include in the body of the message "help".

For a list of frequently asked RMI questions please refer to:
http://java.sun.com/j2se/1.3/docs/guide/rmi/faq.html

To view past RMI-USERS postings, please see:
http://archives.java.sun.com/archives/rmi-users.html