Good guy timers (was: Re: [friam] Timeouts considered harmful )

On 2/28/2014 11:43 AM, Kevin Reid wrote:
> On Feb 28, 2014, at 11:34, Kurt Thams <[email protected]> wrote:
>
>> Kevin (et al),
>>
>> can you reiterate your thoughts about timeouts considered harmful, as discussed in the hangout?
> Not _my_ thoughts, sorry. But timeouts were the subject of a substantial debate a few meetings ago.
>
> If I recall correctly, the thesis was that timeouts should only exist in the bottom and top layers of the system; anything in the middle can't possibly know what the right duration is.

I see that I occasionally miss things that I feel passionately about by 
not following cap-talk/friam more closely.

I agree with the above statement as far as it goes.  I was dealt severe 
pain by timeouts when debugging our capability/network OS (NLTSS -> 
http://en.wikipedia.org/wiki/NLTSS ), particularly timeouts at the top 
layer.  Perhaps you can imagine it.  Set a breakpoint somewhere in "the 
middle" as you note (e.g. you're looking for a problem in the 
file/memory server triggered by a request in the higher level directory 
or process servers), then take time to look at what is going on.  In the 
mean 'time' a high level timer goes off.  Now you complete your 
analysis, perhaps correct something along the way, and desire to 
complete the process to see if that fixes things.  Sadly due to the 
timer expiring things have changed, all you can see is a timeout 
acknowledgement and must essentially start all over.

To deal with this problem I developed a philosophy and code at the top 
(application) levels of our servers that I referred to as "good guy 
timers".   The reasoning was this:

The only reason timers (at least at the top level - there are some other 
issues at the bottom) are needed is to deal with resource issues.  If, 
for example, a partial request comes in or perhaps a full request, but 
then the reply isn't accepted, some resource is tied up - perhaps a 
thread, stack frame, whatever.  Get into enough such situations and you 
can have problems, so a timeout seems reasonable.

The reality of the situation in most cases, however, is that the 
resource generally doesn't need to be released because in fact there 
aren't so many partial requests or requests with blocked replies or the 
like.  So the way our "good guy timers" worked was that we would note 
the time at which a resource began to be tied up.  For us it was 
generally a thread which was synonymous with a 'spaghetti' stack frame.  
We did have a "timeout" - that is a time after which we would consider 
recycling such threads before their processing was complete.  However, 
we only did so if they were needed for incoming requests.  The basic 
idea was that we had a trigger value (minimum number of available 
threads) below which we would look for threads to recycle.  We wouldn't 
consider any threads that hadn't yet reached their "timeout" value (in 
that case if we ran out of threads we would block incoming requests) and 
we would recycle others on an oldest first basis.

The net result of this "good guy" timer philosophy/mechanism was that we 
almost never did "timeout" any requests.  Any such requests that got 
"stuck" just hung around as an odd sort of extension of our heap that 
could be recycled at any time.  Sometimes it was interesting to go 
through them to see what sorts of hangups were actually happening.  We 
could of course clear them all out with a quick blast of requests that 
had a slight delay in the middle of them.

This approach completely solved the practical problem that I had of 
debugging such network service requests - which were all that we had
in our network OS - so I considered it a terrific solution.  Of course 
there is an obvious difficulty with such an approach in the face of 
denial of service attacks, but then even a normal timeout mechanism 
faces these same difficulties.

I was kind of sad that I never got or took the opportunity to publish 
this thought/work.  I would like to know if others had similar thoughts 
and came to similar mechanisms - so I'm delighted to get an opportunity 
to mention it at least to friam/cap-talk.

--Jed