Re: Outrigger and blitz -- lingering take?

Hi Patrick:

I'm not that familiar with the innards of Outrigger or Blitz, but the
behaviour you're seeing actually makes sense when you take into account
the possibility of partial or intermittent network failure.

To understand why, let's assume that a blocking take is implemented
using a callback from the service to the proxy on the client side.
Further, let's say the client asked to wait a maximum of 60000
milliseconds, or 60 seconds.  That's puts a time bound on the whole
transaction.

OK, so the service finds a matching entry and attempts to deliver it
using a callback to the service proxy on the client side.  But the
client side process is unreachable for some reason (like maybe you
killed it, or maybe it failed).  What should the service do?

Here's the problem; the service doesn't know that the client process is
dead.  All the service knows is that it can't communicate with the
client-side proxy.  What if the service wasn't dead, but it was just
unreachable due to a temporary network problem?

(This isn't a theoretical situation - I've had factory floor systems see
short network outages when people sweep the floor near PCs and knock out
the network cable.  Not to mention the vagaries of wireless connections
and overeager or overworked network staff.  You don't really want a
catastrophic failure in these circumstances.  Things should recover when
the network problem is cleared.)

The service should retry delivery of the entry until the agreed-upon
time bound has been passed (in the case of more generalized event
delivery, delivery should be retried until the lease on the notification
request expires), since it is entirely possible that the client could
reappear.  In this case, that first matched entry is in your "lingering
take" state until the service can establish that delivery is
impossible.  At that point, perhaps the entry can be delivered to
another matching taker.  I'm not sure when that state would terminate.

We should also note that it sometimes takes the TCP/IP system a
significant amount of time to report a failure, so that's also a
possible explanation.

How can you handle the situation?  Well, you have to accept that there
is a possibility of network failure (possibly partial and/or
intermittent) in a distributed system.  The level of retry/fallback
logic will depend on the likelihood of failures and the level of
resilience required.  In an application with a human user and low
probability of network failure, I'd probably just have the client side
use a timeout value and report the failure condition to the user when
you don't get a response in the appropriate timeframe.  The human user
can then retry the operation according to her own needs and "retry
policy".  If I were processing credit card transactions, I'd want to do
something more resilient, perhaps write the in-process transactions to a
transaction log or something.


Hope that helps,

Greg.

On Fri, 2007-01-19 at 01:44, Logan, Patrick wrote:
> I am seeing what apears to be some sort of "lingering take" on a
> javaspace. This happens in Outrigger as well as Blitz.
>
> What I am observing? a worker process is blocked on a take with a null
> transaction. I kill the process.
>
> Using the Blitz dashboard I can see after the kill, 1 blocked take.
> (I'm not up on similar tools for Outrigger but all the other
> observations are the same when I use that.)
>
> Then I start another process running the same worker, and after doing
> some other spaces stuff, it soon blocks on a take using the same
> template.
>
> When another process writes an entry that matches that template, it
> appears this "lingering taker" gets the entry.
>
> I am not sure what the explanation is for this "lingering take".
> Running ps indicates the process is really gone.
>
> When I run this with a non-null transaction everything seems fine. I
> have no trouble using transactions to make sure cleanup happens, but I
> am a little surprise that the null transaction case with both space
> implementations do not get cleaned up when a process blocked on a take
> gets killed.
>
> Thanks
> -Patrick
>
>
> Patrick Logan
> Premium Innovation
> Liberty Mutual - Agency Markets IT West
> 650 NE Holladay Street
> PO Box 4555
> Portland, OR   97208-4555
>
> Office: 503.736.7537
> Cell: 503.476.5387
>
>
> =========================================================================== To unsubscribe, send email to [email protected] and include in the body of the message "signoff JAVASPACES-USERS". For general help, send email to [email protected] and include in the body of the message "help".
>
> To view past JAVASPACES-USERS postings, please see:
> http://archives.java.sun.com/archives/javaspaces-users.html
--
Greg Trasuk, President
StratusCom Manufacturing Systems Inc. - We use information technology to
solve business problems on your plant floor.
http://stratuscom.com

===========================================================================
To unsubscribe, send email to [email protected] and include in the body
of the message "signoff JAVASPACES-USERS".  For general help, send email to
[email protected] and include in the body of the message "help".

To view past JAVASPACES-USERS postings, please see:
http://archives.java.sun.com/archives/javaspaces-users.html