Re: [Fresco-devel] Pinging stopped clients

OK, I've spent too long thinking about this...hopefully my tiredness
doesn't show too much ;)

Nathaniel Smith wrote:

> I noticed today that if you C-z the demo, it _doesn't_ get killed by
> the server, and I don't understand why.  Tracing confirms that it has
> stopped responding to pings.  (Perhaps I should say it stops
> responding to "ping", not "pings"; a single ping is sent, and no
> response comes back; apparently the ping loop blocks entirely,
> apparently without timeout -- at least, if there is a timeout, it's
> more than a few hours long.)

Would you want an application to get killed if you just C-Z'ed it? Does
C-z'ing in vim kill it? That's not to say that *something* shouldn't
happen in the server if it can't contact a client, but that's a
different matter IMO.

> Somehow the server ignores this and continues to work normally (but of
> course freezes if you click "run" or "quit").
> Well, not quite normally; I tried starting a second demo, and it was
> unable to connect to the server.  If I start two demos and then C-z
> one of them, and then kill the other, the server doesn't notice that
> the other has died; if I then fg the stopped demo, the server then
> notices the death of the other immediately.

Ping-based detection of client death/unavailability lies within a mutex.
If you C-z one running app, it effectively suspends the ping thread for
*all* clients, which keeps the mutex locked. Restarting the stopped demo
causes the mutex to unlock, allowing the ping (and hence client-death)
thread to continue.

bug92 is related to this; sometimes Kits are not garbage-collected
properly, but since this process can occur from within destroy_context,
and since that function is called from within the same mutex...

> I suspect that the reason clients can't connect while the ping loop is
> blocked is a mutex somewhere (since during connect, the new client
> handle has to be added to the list of pingable hosts); this is
> probably a bug, since CORBA requests can take arbitrarily long to
> handle.  

AFAIK this is exactly the problem...new-client connect also occurs
within the same mutex above.

> The solution, I think, is to have the "accept loop"
> communicate with the ping loop via a sort of queue; the accept loop 1)
> acquires a lock, 2) puts things on this queue, 3) releases the lock;
> and the ping loop 1) acquires the lock, 2) pulls everything on the
> queue off and adds them to its private data structure, 3) releases the
> lock, 4) does the actual CORBA calls.  This way the lock isn't held
> over any remote calls.

How does this help? Assuming a loop within the ping thread, as before,
then each ping loop a) updates the queue b) does the pings. If any ping
hangs, then we have exactly the same problem as before, surely? Yes, new
client connections won't block, but the queue will just get bigger (and
elements remain unprocessed) if the ping loop (which updates the queue)
blocks/hangs in a CORBA call.

The only solution I can see is for the thread updating the list of
client connections to never do CORBA calls (ie. the looping block not
contain blocking functions). That is, ping calls must occur in a
separate thread. The thread which initiates the ping calls might now be
termed a 'connection manager' thread. Perhaps it calls some
update_client_status method, which creates a thread which pings each
client ref. After pinging each client, it updates the status for that
client to say that it has pinged it. The update_client_status method
sleeps for some timeout time, after which it checks that the number of
pinged clients has increased; if not then it kills (?!) the pinging
thread, marks the previous client ref as uncontactable and restarts the
ping thread starting at the next client ref. Once all pings are done,
update_client_status returns, with a status report for each client: the
connection-manager then determines whether uncontactable clients should
be ignored, removed, etc.

Ideally the update_client_status function is handled by some pluggable
method, such that we can use something brute-force like the above or use
some CORBA AMI if and when that becomes available (see below). But
perhaps that would be overdesign at this early stage.

Adding new connections only after every round of pinging clients limits
us to a small ping timeout, in order to allow clients to connect asap
(worst case: lots of clients time-out; total ping-time = many timeouts).
Instead we might have two client-lists in the server, one for the
'established' connections and one for the 'new' ones. That allows 'new'
connections to work independently of the ping thread. After each
successful update_client_status we move all the new connections onto the
established list, and start the next connection-check. At all times we
have a list of connections which we consider valid = 'new' +
'established'. One alternative to this would be to have more ping
threads, ie. each ping thread pings a distinct subset of all the
clients; in the upper limit of one thread per client, this reduces the
time between new clients being noticed to the (maximum) timeout
value...I think I prefer the first choice ;)

On a related topic, Chalky mentioned using this for other CORBA object
references, eg. Commands. What happens when a user clicks on an object
belonging to an 'inactive' (suspended/killed/network-dead) client?
Ideally we'd have some system whereby all calls to client-implemented
objects have some timeout - but that seems impractical without some orb
support?

So I can understand how we can solve the alive/dead problem in the
context of some ping mechanism, but wrt general calls into the client:
how could we approach this? AFAICT this is a *huge* problem: we cannot
build robust applications without solving this! Thoughts of nfs pop into
my head for some reason...

> More serious, though, is the fact that apparently, making a CORBA call
> to a stopped process blocks indefinitely, with no timeout.  This is
> kinda uncool, and I didn't expect omniORB to have this behaviour.  Any
> ideas on why it does this?  Or is it the correct behaviour, and
> someone can tell me why?  (And if it is, then how do you modify the
> ping loop?)

My guess why this occurs is that when a process is suspended it...stops
running ;) More importantly, since the orb part of that process is just
a library, not a separate process, the orb also stops running, and calls
into it also stop.

I believe that 'built-in' timeouts for CORBA calls are available, but
are either orb-specific or else require implementation of something like
the Asynchronous Messaging Interface (AMI) from the CORBA Messaging
spec. I've not really read about either; if noone comments soon, then
perhaps we should ask on the omniorb list about this. OTOH I heard that
TAO supports the latter ;)

> (And a deeper question: how did we end up with killing off stopped
> clients the _goal_ we wish to achieve by fixing these problems?)

What makes you think this is the aim? Just because stopped clients are
not handled correctly doesn't mean that we want to consider
stopped==dead :)

-- 
Neil