Re: Software Caused Connection Abort (10053) on Client

Still confirming your questions.

>> I don't immediately see why this should matter. 

No, me either. Seems mysterious to me but I notice everywhere else we
pass around "this" rather than using the stub directly. Then I noticed
this old bug report and decided to make the change.

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4114579

Ok, so when an agent starts up it registers with the server. During this
process it does two things (a) rebinds itself to the registry it hosts
and (b) sends a AgentDescriptor to the server. This object has a
reference to the stub (not "this") and it is serialized to the server
where the server saves it persistently. When I changed the rebind to use
"this" I also changed the AgentDescriptor to use "this". Anyway, if the
server ever has a problem contacting the station it uses the same trick
of re-looking up the agent's stub in the Smart Proxy's invocation
handler. Maybe that is more the issue than the server side stub. But it
seems unlikely since when the server side exception occurs, the agent is
making a call on the server and not the server on the agent.

>> (I notice that you can only see server-side logging
>> of the NoSuchObjectException by using the sun.rmi.server.call
>> logger (or setting java.rmi.server.logCalls) 

Yup saw that when I was checking out the code. Of course it would be
nice to know what that object is since we have so many callbacks when
operating on many agents. The trace is only this:

 FINE: Rule Run Launch Queue.0: free connection (reuse = true)
 Nov 29, 2005 7:21:57 AM sun.rmi.transport.Transport serviceCall
 FINE: RMI TCP Connection(13)-192.168.54.104: [192.168.54.104]
exception: 
 java.rmi.NoSuchObjectException: no such object in table
        at sun.rmi.transport.Transport.serviceCall(Transport.java:112)
        at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
        at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.ja
va:701)
        at java.lang.Thread.run(Thread.java:534)

I'll have to setup this tracing on all our agents and then correlate it
(timewise) with the agent log to see if it is actually a call to the
server stub (from the agent) that is causing this exception. It could
also be a remote object that is being marshaled over a call but that
seems unlikely.

R.

-----Original Message-----
From: Bob Scheifler [mailto:[email protected]] 
Sent: Tuesday, November 29, 2005 11:10 AM
To: Robert DiFalco
Cc: [email protected]
Subject: Re: Software Caused Connection Abort (10053) on Client

> 1. Our server was binding a reference to its stub instead of "this" in

> it's Registry.

I don't immediately see why this should matter.  If it's an external
registry, impl-to-stub replacement should happen automatically as part
of the bind remote call anyway.  If it's an in-VM registry obtained
using createRegistry, then impl-to-stub replacement should happen
automatically as part of the return from a lookup remote call.
So what the client sees should be the same in the end.

> 2. The agent registry used a caching mechanism to only lookup the 
> server stub once. It then wrapped the cached stubs in a SmartProxy 
> that would re-lookup the stub in the server's registry only if an 
> invocation handler produced a NoSuchObjectException.

Ah, on the server side, NoSuchObjectException is treated like a
server-side failure, and is subject to the same problem described in bug
4716483: the NoSuchObjectException will be marshalled back to the
client, but the arguments of the call will not be consumed, and will be
treated incorrectly as bits of a subsequent call.
(I notice that you can only see server-side logging of the
NoSuchObjectException by using the sun.rmi.server.call logger (or
setting java.rmi.server.logCalls), you won't see if by setting
sun.rmi.server.exceptionTrace.)

> For some reason it looks like either one of these alone was ok but 
> together they introduced unreliable behavior (oddly just on windows so

> far).

How certain are you of the correlation with how the registry bind is
performed?

> The best way to see it would be to restart the server (but not the 
> agent). Then have the agent invoke a call on the server after it had 
> restarted. It's cached stub would be invalid, so the first attempt to 
> invoke a method on it would cause the SmartProxy to re-lookup the stub

> from the server's registry.

Are you certain the SmartProxy is actually seeing the
NoSuchObjectException?  Because if it does receive it, the client side
should be treating the connection as not reusable, and the next call
should be using a new connection.  The simpler explanation would be that
you have enough argument data that, depending on scheduling, you can't
reliably receive NoSuchObjectException.

- Bob

===========================================================================
To unsubscribe, send email to [email protected] and include in the body
of the message "signoff RMI-USERS".  For general help, send email to
[email protected] and include in the body of the message "help".

For a list of frequently asked RMI questions please refer to:
http://java.sun.com/j2se/1.3/docs/guide/rmi/faq.html

To view past RMI-USERS postings, please see:
http://archives.java.sun.com/archives/rmi-users.html