Re: [Fresco-devel] Pinging stopped clients

On Sun, Dec 08, 2002 at 05:23:57PM -0500, Sean Rae wrote:
> > I noticed today that if you C-z the demo, it _doesn't_ get killed by
> > the server, and I don't understand why.  Tracing confirms that it has
> > stopped responding to pings.  (Perhaps I should say it stops
> > responding to "ping", not "pings"; a single ping is sent, and no
> > response comes back; apparently the ping loop blocks entirely,
> > apparently without timeout -- at least, if there is a timeout, it's
> > more than a few hours long.)
> [... clipped some words...]
> > More serious, though, is the fact that apparently, making a CORBA call
> > to a stopped process blocks indefinitely, with no timeout.  This is
> > kinda uncool, and I didn't expect omniORB to have this behaviour.  Any
> > ideas on why it does this?  Or is it the correct behaviour, and
> > someone can tell me why?  (And if it is, then how do you modify the
> > ping loop?)
> Maybe it's my knowledge of the coffee language speaking, but it could
> block until a timer sends an event to timeout... ?  Which might open up
> the loop for other events (like the network cable being disconnected,
> which could just stall all network connections until the server says "hey
> the network cable is back in").

Oh, sure, it could do all those things.  I'm confused because it
doesn't do any of them -- it doesn't even time out.  The remote call
just sits there waiting indefinitely for the client to start running
again.

> > (And a deeper question: how did we end up with killing off stopped
> > clients the _goal_ we wish to achieve by fixing these problems?)
> I have an idea... As a disclaimer, I know nothing of the code.  But would
> it not fix this to keep a time-of-last-comminication variable for each
> client?  Whenever any communication from the client is handled, update it.
> After the client is quiet for 30 seconds (or whatever) then ping the
> client.  That way, the worst case scenario is you stop the client and the
> server waits 30 seconds, pings it, maybe pings it again, then kills it
> (reporting this fact somewhere?).  This could lead to being able to grade
> the speed of clients.  For example, let's say you run a client across the
> world and it's slow.  If the server knows it's slow, it can up the time
> until it times out the connection (from 30 seconds to 60 seconds?).  The
> window holding the slow client could have a little swirley icon like web
> browsers have to show that it's "thinking" (as my dad says).  This might
> seem wierd to add a little bit of the web-world to the windowing system,
> but it would be nice to have for slow clients.  What do ya'll think?

This is essentially what we're doing now, though rather than try and
keep track of every communication (which is essentially impossible
with CORBA, because we have no way to know who's making any given call
-- this could be fixed via the use of intercepters/the security
service, but that's complicated), we just ping every second, and trust
the ORB to tell us when a client stops responding to these pings.

The somewhat obscure point I was making with the above comment,
though, is that all these methods will screw up with a client that is
stopped but not yet dead.  A stopped client cannot respond to pings;
therefore, a correct implementation of pinging will detect that they
are not responding to pings, and infer that they must be dead, and
kick them off the server.  So killing stopped clients is correct
behaviour for a ping-based solution.  It's not exactly an optimal
solution overall, though :-(.

> Has anyone tried to use error-finding tools on the fresco tree yet?  Like
> valgrind, electric fence, leaky, etc etc... ?  Ciao.

I know that some people have played with that, but I'm sure there are
still more things to find :-)

(One annoying thing is that since we push things so hard with
multithreading and dynamically loaded libraries and to some extent
CORBA, historically many of these tools haven't worked very well :-(.
But perhaps they work better now -- I know valgrind has been used
successfully, at least.)

-- Nathaniel

-- 
"Of course, the entire effort is to put oneself
 Outside the ordinary range
 Of what are called statistics."
  -- Stephan Spender

This email may be read aloud.