Re: Deadly embrace on queue semaphores in LiS 2.17.2 and LiS 2.17.U

Hi all...

Sorry that I have not had a chance to respond, I've been travelling
the last few days.

For me, in fact the qlock=2 style is better for my modules.  On
Solaris they are already QPAIR locked, but since this notion did not
exist before 2.17.U, I was forced to implement QPAIR "by hand" with
my own spin locks called from within the module itself.  What I
quickly learned was that this doesn't help you in the situation that
I outlined in my previous email.  I have not had a chance to finish
testing on 2.17.U yet with qlock=2, but I expect that it will make
things "all better".

One question that I had from my original email that never was really
addressed was what is the need to hold a queue's lock while calling
putnext or qreply()?  That is, once we leave a particular module, why
do we need to continue to hold that module's semaphore?  It seems to
me that there is really no reason to continue to hold the queue's
lock after the thread has passed control to the next module?

Can't we just do something like (sorry for the rough pseudocode):

QUNLOCK(q);
QLOCK(q->q_next);
call put routine of (q->q_next);

when we do a putnext?

As it is now we hold the queue that we are leaving's lock when we
call putnext() or qreply().  This would allow more parallelism since
you wouldn't lock the entire side of a stream as you pass a message
downstream and even more so in the case of ioctls() where you end up
having to hold evey lock in the stream by the time that the message
makes it back up to the stream head.

I haven't had  chance to try and answer this for myself as I've been
tied up with other projects but I'm planning on check it out sometime
this week.

Is there something fundamental that I'm missing here, Dave?


thanks-
dan

--- Dave Grothe <[email protected]> wrote:

> Klaus:
> 
> Too much to respond to in detail.
> 
> qlock=1 is the previous LiS style (2.16 and before).
> 
> qlock=2 may very well fit Dan's situation better.
> 
> The options are there precisely because different STREAMS drivers
> behave 
> differently and some can take advantage of the different locking 
> models.  qlock=0 is there to allow individual drivers to maximize
> knowledge 
> of special circumstances to implement locking strategies.  You can
> also get 
> some multi-cpu pipelining going with this option that is not
> possible with 
> others.
> 
> You can do some pretty innocent things in your put/service
> procedures that 
> lead to "might sleep" situations in the opinion of the 2.6 kernel. 
> Notice 
> that ioctls are performed on top of put/service procedures and are
> often 
> used for configuration, so memory allocation calls are common in 
> ioctls.  If you have ever had the experience of having to allocate 
> multi-megabytes of control block space for huge number of streams
> then you 
> know the impracticality of calling the memory allocator with 
> GFP_ATOMIC.  You get allocation failures long before reaching the
> limits of 
> physical memory.
> 
> If LiS enters a put/service procedure with a spin lock held then
> LiS has 
> pre-defeated any attempt on the part of the driver to deal with its
> own 
> execution situation.  That is why LiS does not do that.
> 
> You can call putnext() from interrupt context under qlock=0 if the
> put 
> procedure knows that this is happening and is careful in what it
> does.  But 
> if you trigger an arbitrary chain of putnext() calls from interrupt
> context 
> you are asking for trouble.  So somewhere along the line service
> procedures 
> must be involved -- at the stream head if not before.
> 
> Your queueing idea is interesting and LiS has a deferred message
> processing 
> mechanism that could be used to implement it, but I don't want to
> "break 
> the eggs" to do so at the moment.
> 
> Thanks for the comments,
> Dave
> 
> At 08:08 AM 9/14/2004, Golbach, Klaus wrote:
> 
> >Hallo Dave,
> >
> >Dan seems to be satisfied by Your answer. For me still things are
> not
> >clear.
> >
> >To come together I repeat, what I think to have understood about
> qlock.
> >LiS 2.17.* provides a new feature to serialize the put and service
> >functions according to the qlock numerical values:
> >
> >0 = All put and service functions may run in parallel. No
> >qlock-semaphore is grabbed, if put and service functions are
> called.
> >This type of working equals that in earlier LiS-Versions 2.16.18.
> >1 = The put and service functions on each queue side are
> serialized.
> >There is a semaphore for each read and write side (default)
> >2 = The put and service functions of a queue pair are serialized.
> There
> >is a semaphore for a queue pair.
> >3 = There is one semaphore for all queues of all drivers, which
> have
> >qlock set to 3.
> >
> >So I do not understand really Your sentence:" The qlock option is
> >intended to help out drivers whose internal model does not fit
> well with
> >LiS's use of the locking."
> >I thought this feature helps to overcome problems running a
> >streamsdriver in parallel.
> >There are some remarks to this point at the end.
> >
> >The described solution may cause a deadlock situation as described
> by
> >Dan. This deadlock situation may happen:
> >-if qlock=1 is chosen and
> >-if neighbour streamsdriver do a qreply or a putnext(on OTHER)
> back to
> >the calling driver and
> >-if on the other queue-side also a put or service routine is
> running,
> >which does a qreply.
> >
> >This deadlock is independent of the special implementation of Dan
> s
> >driver. I think it may happen to other drivers as well as to
> streams
> >head, which runs with qlock=1, if a neighbour driver behaves
> >accordingly.
> >
> >My conclusion is:
> >The offered qlock-feature does not work correct with qlock=1.
> (with
> >qlock=3 the described situation will not lead to such a
> >deadlock-situation  - and hopefully qlock=2.)
> >Or You assume restrictions about the behaviour of the
> streams-drivers:
> >"in the put routines it is not allowed to call back to the calling
> >driver. This has always to be done from service routines."
> >
> >By the way. There is another "implicit" restriction coming with
> qlock:
> >"It is forbidden to call putnext from interrupt-context, if qlock
> !=0"
> >putnext forces to grep the qlock-semaphore. If one does putnext
> from
> >interrupt-context, this will lead to unpredictable results, since
> there
> >is no thread, on which we can sleep.
> >
> >Idea for a solution for both problems:
> >from qreply or putnext do not call the put-routine, if a semaphore
> is to
> >be grabbed. Instead store the parameters in a queue, which is then
> >processed in runq-context. This may be not as simple, as it looks
> like
> >and will change the time-behaviour.
> >
> >For the moment I would suggest to disable the feature qlock=1. And
> all,
> >who use qlock!=0  do not call putnext from interrupt-context in
> their
> >whole streams-stack.
> >
> >
> >I understand Your sentence:" The qlock option is intended to help
> out
> >drivers whose internal model does not fit well with LiS's use of
> the
> >locking."  in the following way. You are afraid that if LiS does
> for
> >some reasons a lock around a put- or service-routine, this may
> lead to
> >unpredictable results, if in this routine some ddi or dki
> functions are
> >called, which are not allowed under lock-protection like:
> allocating
> >memory with GFP_USER , or doing Semaphore operation down or sleep
> in
> >some way...
> >
> >On my point of view in streams-context only ddi/dki calls should
> be
> >done, which are allowed in interrupt-context. Reasons for this
> >restriction:
> >1.) The streams-subsystem is implemented different in different
> OS. On
> >Unixware for example service-routines run completely under
> >interrupt-context. To be portable this restriction is in own
> interest.
> >2.) You never know from which context the put-routines are called.
> It
> >very often happens- that drivers at hardware level call streams
> >functions like putnext from interrupt-context, to process things
> very
> >fast and synchronous. So to work well with different drivers
> below,
> >streams-drivers should use this restriction on own interest.
> >3) ddi-dki-functions which are allowed in interrupt-context are
> also
> >allowed to be called under lock-protection.
> >
> >To have an easier life, You should guarantee LiS working well only
> if
> >this restriction is fulfilled. All exceptions on own risk, even
> when LiS
> >provides a thread-context for the service-routines.
> >
> >Best regards
> >     Klaus
>