AW: Socket factories with JERI

Hello Mark,

thanks for this really interesting information. 

> Also  I don't  expect a  timeout constraint  to be  very accurate,  making > it  a soft[*] constraint. I don't expect people  to make a decision
> between a  timeout of 100ms versus 150ms, but more as a mechanism to
> protect itself from hanging on a system it can't influence and to specify 
> the range in which it should  return.
> I expect people to use a time-out constraint with a timeout of 500ms or 
> 1sec for the  general call  that would  take ~50ms.

With respect to our current project, your expectations are exactly correct. As i already wrote, we have realtime requirements: If a remote call, that normally takes ~50 millisecs, does not return within ~2 seconds, the results of that call are useless in our system. We must assume in this case that the network or the called server is broken, and we must immediately react to this failure. We also want to free ressources used by this remote call immediately (i.e., close OS sockets and make objects eligible for GC)

> I believe that the server side doesn't need to be notified, in my
> opinion this case is similar to a connection failure while the return
> part of remote method invocation is in progress.

Obviously, in the 'broken network cable' scenario it is not possible at all to notify the server. In my opinion, the server should however free it's resources that are no longer useful. Perhaps this could be done this way: If a method with round-trip-timeout constraint is invoked, this timeout is transmitted to the server within the call. The server logically starts it's own timeout when it has unmarshalled the call. If the result of the call cannot be computed or written to the network within the timeout, the server can be sure that the client has already timed out anyways and can abort the computation or sending of the result.

BTW, i have just coded SocketFactories for 'traditional' RMI that use nonblocking NIO and select() for reading and writing and this way implement read and write timeouts on sockets. This way, i can prevent the client as well as the server from hanging if anything fatal happens to the network or to the peer machine during a remote invocation. I'm now going to modify this factory to make it usable with JERI, perhaps this provides a usable workaround.

> Improvements to the Jini core are supposed to be developed as part of the
> Apache River project and and any input for ideas to materialize is
> welcomed.

I looked at http://incubator.apache.org/river/roadmap.html and subscribed to the developers mailing list of the project. For our current project (that is scheduled to be completed within 2008), the detection of broken cables/servers within a few seconds is an absolute knockout requirement. Do you expect that river will support 'round trip timeout constraints' in this timescale ? 

Finally, i'd like to raise some general, somewhat heretical questions. Just having read the report about the JINI based 'Neuromancer project', which was discontinued, i'm asking myself: If we are looking for a networking technology to be deployed generally in our (more or less) embedded products, that should survive one decade, could JINI/River be the choice? Who is supporting this technology (SUN any longer ?), and who is actually using it? Are there any 'reference' projects one could cite to convince coworkers or even managers? I'd really appreciate any hint or report about this issues, regardless it's tenor.

Cheers, Martin

-----Ursprüngliche Nachricht-----
Von: Mark Brouwer [mailto:[email protected]] 
Gesendet: Sonntag, 2. Dezember 2007 14:50
An: Cornelius, Martin (DWBI)
Cc: [email protected]
Betreff: Re: Socket factories with JERI

Hi Martin,

Cornelius, Martin (DWBI) wrote:
> Hi Mark, thanks for taking your time.
> 
>  
> 
>>Not that it shouldn't be handy to have a constraint to indicate the
>>maximum time one is willing to spend waiting for a response.
> 
>  
> 
> Actually, this constraint is exactly what i was looking for. In my 
> application, i have to react to physical network interruptions quickly 
> (< 10 seconds) In CORBA, a constraint with this semantics can be 
> enforced via the RoundTripTimeoutPolicy. One of the reasons i am 
> experimenting with JERI is the fact, that open-source JAVA ORBs either 
> do not implement this policy at all (Sun's ORB) or do not implement it 
> correctly. (JACORB).
> 
> Skimming through the JINI/JERI API's, i had already found the 
> ConnectionRelativeTime constraint, and was quite astonished that there 
> is no constraint with semantics similar to CORBA's RoundTripTimeout. Of 
> course, it is a major task to implement this semantic in presence of 
> multiplexing, (what may be the reason why it is not implemented or not 
> working in many CORBA implementations).
> 
> My eager hope was , that RMI resp. JERI would at least measure up to 
> CORBA in this respect, AFAIKS it does not. Perhaps, it makes no sense at 
> all to use RMI or JERI under the given requirements (detect broken 
> network within 10 seconds). Of course, that would make me wonder, as the 
> very first of the famous "eight fallacies" is : "The Network is 
> reliable". Perhaps i missed an essential point so far ?

During the Porter project (of which unfortunately the mailing list
archives are lost) this constraints has been discussed (as part of the
discussion with subject "supporting asynchronous remote calls" for those
who archived the mails). Below you can find a collections of the
relevant postings for your interest. Although I'm not able to give a
satisfactory response to some of the issues raised by the Jini team, I
still believe there is need for a mechanism that would 'abort' a remote
method call in case a certain time limit has passed and that should free
up the resources involved with such a remote method invocation.

It is an interesting topic and I hope the discussions provides you
answers related to the problem you try to tackle. If you have opinions
yourself, please share them with us. Improvements to the Jini core are 
supposed to be developed as part of the Apache River project and and any 
input for ideas to materialize is welcomed.

For you interest what follows is my initial question to the Jini team 
and the discussion that followed:

-----------------------------------------------------------------------

Mark Brouwer asked:

"We also encountered cases in which clients hung op on remote method
calls that were in a dead-lock situation at the server side, requiring
an abort of the server side to free the client. Debugging these type of 
problems is in general also a rather 'interesting' task.

Therefore I would like to see a 'ResponseTimeOut' constraint that allows
me to define a time limit (probably on top of the ConnectionRelativeTime
or ConnectionAbsoluteTime constraint) in which a remote method call must
return. Without giving it much thought I would say that when the timeout
limit is reached and the method has not returned the remote method call
is aborted and a new type of RemoteException is thrown."

-----------------------------------------------------------------------

The reaction of Bob Scheifler was:

"Internally we've discussed the notion of a remote call timeout
on and off for quite some time. In the abstract it seems desirable,
but as usual there are many specifics to figure out. For example,
what computation should it cover ("response" or complete call;
what if a smart proxy's method does multiple remote calls, or does
significant local pre/post processing; does a smart proxy have to
know about this constraint), how do you implement it safely, how
do you implement it reliably (what if the hangup is in a class loader
attempting to download code), should an attempt be made to notify the
server side (should perhaps the remote call be leased), how would
you propagate that notification up the server side stack, do you want
to keep waiting if you somehow know forward progress is being made
or communication is still possible."

-----------------------------------------------------------------------

Based on the above response my reaction was:

Bob Scheifler wrote:

 > Internally we've discussed the notion of a remote call timeout
 > on and off for quite some time. In the abstract it seems desirable,
 > but as usual there are many specifics to figure out.

Thanks for that info. I take the liberty to provide you my opinion with
regard to the 'specifics' you mentioned. As I'm rather virgin with
regard to this subject be gentle with me :-)

 >                                                      For example,
 > what computation should it cover ("response" or complete call;

Complete call I would say, by setting a time-out constraint I indicate
that the computation the service interface promises me must/should be
satisfied within my max time-out, no matter what that involves under the
hood. For that reason I was also not sure whether the timeout should be
on top of the connection constraints already available.

 > what if a smart proxy's method does multiple remote calls,

For these difficult and minority cases a smart proxy could throw its
hands into the air and indicate it won't support that constraint, or it
could implement a mechanism in which it mimics the semantics of such a
timeout constraint. Or in case you consider the constraints not as a
'hard' constraint [*] do the best it can and (for example) spread the
constraint over the various calls, in a way that the total of the
individual constraints would be equal to the one set by the client.

 > or does
 > significant local pre/post processing; does a smart proxy have to
 > know about this constraint),

In such a case the proxy has to know about the constraints and can deal
with it in two ways, makes a rough estimate about the time for the
computation in the proxy and subtracts that from the one set on the
actual (remote) proxy, or implements a mechanism that mimics the
time-out semantics.

 >                                   how do you implement it safely,

Can you elaborate what safely exactly means here?

 >                                                                   how
 > do you implement it reliably (what if the hangup is in a class loader
 > attempting to download code),

Downloading classes the first time is indeed something that is tricky to
count in. It could result in never being able to make the first
invocation due to an unlucky case where the download jar file is 3 MB
and must be loaded within 20ms. But in real live I think that the
problems can be managed, as I expect that as result of downloading the
service proxy the jar file will be cached at the client side and that
applying the time-out constraint is done after most of the download jar
files are locally to the client. But agreed I'm making some assumptions
here for which there are cases I'm totally wrong.

I fail to see yet what reliability has to do with a hangup in a class
loader as I believe that the timeout would be implemented by the client
side of the RMI runtime and that any class downloading issues wouldn't
interfere with the ability to throw the specific timeout exception (but
this is complete due to the fact that I have no idea what the best
approach would be for implementing such a feature).

 >                           should an attempt be made to notify the
 > server side (should perhaps the remote call be leased), how would
 > you propagate that notification up the server side stack,

I believe that the server side doesn't need to be notified, in my
opinion this case is similar to a connection failure while the return
part of remote method invocation is in progress. If this happens too
often to the like of the server (due to a too short timeout set by the
client) it can find out and take its measurements. I don't envision
support here by the RMI runtime.

With regard to leasing, not a good idea I think, if synchronous calls
are expected to take a long time (expected within the design space for
that interface) I think the business interface should be designed for
that, meaning that a short call should be made that result in a
'computation ticket' that will contain a lease and that has some
mechanism to communicate the result back. As in the original post of Ann
I think these case are best served with interfaces that are designed for
such purposes.

 >                                                        do you want
 > to keep waiting if you somehow know forward progress is being made
 > or communication is still possible.

No too much complexity, and I doubt whether in general the server side
is able to find out how long it would take to return as it has probably
dependencies for which on forehand it can't say that much. If it can and
is important to have this sort of communication I would suggest design
your interface for that.
Also I don't expect a timeout constraint to be very accurate, making it
a soft[*] constraint. I don't expect people to make a decision between a
timeout of 100ms versus 150ms, but more as a mechanism to protect itself
from hanging on a system it can't influence and to specify the range in
which it should return. I expect people to use a time-out constraint
with a timeout of 500ms or 1sec for the general call that would take
~50ms. If a client really wants to make decisions based on the actual
round trip times it can always measure them and if it believes they are
too long it can go after another services, if any. But at least it will
be able to free/protect itself without switching to its own asynchronous
programming model and this IMHO would be a great asset to the remote
programming model currently available.

[*] In general I would say that a time-out constraint shouldn't be seen
as a 'hard' constraint but as a 'soft' constraint. The difference
between 'hard' and 'soft' is that the hard constraints must be satisfied
for 100% (such as the security constraints, you have permissions or not)
and that soft constraints are more like an indication (they have to be
supported but the actual behavior can slightly deviate from the actual
parameters given). Within time I got the impression that 'quality of
service' constraints are better served by seeing them as soft
constraints, as implementing them as a hard constraint requires
enormous investments in writing complex smart proxy code, if even possible.

-----------------------------------------------------------------------

For which the response of Bob Scheifler was:

 >> what computation should it cover ("response" or complete call;
 > >
 > > Complete call I would say

OK ("ResponseTimeout" in your original message had me guessing).

 >> >>what if a smart proxy's method does multiple remote calls,
 > >
 > > For these difficult and minority cases a smart proxy could throw its
 > > hands into the air and indicate it won't support that constraint, or it
 > > could implement a mechanism in which it mimics the semantics of such a
 > > timeout constraint.

Well, this goes to a fundamental question that was raised by the
async constraint: is it ever OK to introduce a new constraint
that is supported by JERI (or other RMI implementations), if smart
proxies must also be aware of it to provide correct semantics?
There may be a difference in how fatal the consequences are of
injecting such constraints into existing smart proxies, but should
fatality be the discriminator?

 > > Can you elaborate what safely exactly means here?

Do you have sufficient visibility into and control over all places
where a thread might block, to cause it to safely unblock? As an
example, simply forcing interrupts on the thread until it pops
back out might not be guaranteed to result in consistent state.

 > > Downloading classes the first time is indeed something that is 
tricky to
 > > count in. It could result in never being able to make the first
 > > invocation due to an unlucky case where the download jar file is 3 MB
 > > and must be loaded within 20ms. But in real live I think that the
 > > problems can be managed, as I expect that as result of downloading the
 > > service proxy the jar file will be cached at the client side and that
 > > applying the time-out constraint is done after most of the download jar
 > > files are locally to the client. But agreed I'm making some assumptions
 > > here for which there are cases I'm totally wrong.

Yes, for example that there isn't more than one jar file, or that
they aren't using a class tree instead of a jar file. But the
proxy code itself wasn't my main concern.

 > > I fail to see yet what reliability has to do with a hangup in a class
 > > loader as I believe that the timeout would be implemented by the client
 > > side of the RMI runtime and that any class downloading issues wouldn't
 > > interfere with the ability to throw the specific timeout exception

The particular issue I have in mind is when the results coming back
from the remote call involve third-party code (e.g., from a LUS
lookup, or a JavaSpaces read/take), which has to be downloaded on
the client side, and that causes a class loader running in the
client's calling thread to block for a long time. There is no
ready visibility into or control over such a hang.

 > > I believe that the server side doesn't need to be notified, in my
 > > opinion this case is similar to a connection failure while the return
 > > part of remote method invocation is in progress. If this happens too
 > > often to the like of the server (due to a too short timeout set by the
 > > client) it can find out and take its measurements. I don't envision
 > > support here by the RMI runtime.

That's a reasonable position, but it depends on how many different
scenarios you want this timeout to help. If you want to support cases
where the client makes a remote call, not knowing how long it will
take, but knowing that the answer won't be useful unless it comes
back within a certain time, and it turns out it will take a lot
longer to compute the answer, overall system performance may well
benefit from attempting to notify the server side so it can stop
computing.

 > > if synchronous calls
 > > are expected to take a long time (expected within the design space for
 > > that interface) I think the business interface should be designed for
 > > that

A plausible position, but the desirability of having two different
mechanisms for roughly the same purpose is something I'm still
wrestling with. One might argue that all remote calls should be
"expected" at some point to take a long time.

 > > [*] In general I would say that a time-out constraint shouldn't be seen
 > > as a 'hard' constraint but as a 'soft' constraint.

I'm not really concerned about being off by a little, but with being
off by a lot. I wouldn't want a proxy to claim to support a
constraint as a requirement (as opposed to a preference) and then
simply ignore the constraint or be completely unable to satisfy
the constraint in some of the very scenarios you expected it to
work for in the first place. Another way of musing about this is,
if a constraint should/can only ever be used as a preference, is it
a good constraint?
-- 
Mark

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________

 
************************************************
The information contained in, or attached to, this e-mail, may contain confidential information and is intended solely for the use of the individual or entity to whom they are addressed and may be subject to legal privilege.  If you have received this e-mail in error you should notify the sender immediately by reply e-mail, delete the message from your system and notify your system manager.  Please do not copy it for any purpose, or disclose its contents to any other person.  The views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of the company.  The recipient should check this e-mail and any attachments for the presence of viruses.  The company accepts no liability for any damage caused, directly or indirectly, by any virus transmitted in this email.
************************************************

--------------------------------------------------------------------------
Getting Started:     http://www.jini.org/wiki/Category:Getting_Started
Community Web Site:  http://jini.org
jini-users Archive:  http://archives.java.sun.com/archives/jini-users.html
Unsubscribing:       email "signoff JINI-USERS"  to [email protected]