Re: Deadly embrace on queue semaphores in LiS 2.17.2 and LiS 2.17.U

At 03:20 PM 9/14/2004, dan_gora wrote:


>Hi all...
>
>Sorry that I have not had a chance to respond, I've been travelling
>the last few days.
>
>For me, in fact the qlock=2 style is better for my modules.  On
>Solaris they are already QPAIR locked, but since this notion did not
>exist before 2.17.U, I was forced to implement QPAIR "by hand" with
>my own spin locks called from within the module itself.  What I
>quickly learned was that this doesn't help you in the situation that
>I outlined in my previous email.  I have not had a chance to finish
>testing on 2.17.U yet with qlock=2, but I expect that it will make
>things "all better".
>
>One question that I had from my original email that never was really
>addressed was what is the need to hold a queue's lock while calling
>putnext or qreply()?  That is, once we leave a particular module, why
>do we need to continue to hold that module's semaphore?  It seems to
>me that there is really no reason to continue to hold the queue's
>lock after the thread has passed control to the next module?
>
>Can't we just do something like (sorry for the rough pseudocode):
>
>QUNLOCK(q);
>QLOCK(q->q_next);
>call put routine of (q->q_next);
>
>when we do a putnext?

If the putnext() was always guaranteed to be the last thing that a module 
did before returning then this would be safe, in the general case.  But 
since we are going to return to the module and the module will continue 
execution we will have allowed a "nested" entry into the module if we let 
go the lock here.  The putnext() function would have to re-acquire the 
lock, but the effect might lead to interlaced execution of the 
module.  Streams fed by multiplexors would be especially vulnerable to this 
since multiple worker threads on the upper streams could be feeding a 
single lower stream and contending for the put procedure of the next lower 
queue.

If you really want to allow this kind of thing then use qlock=0 and code 
your own abstraction for putnext that implements the locking model that you 
want to use.



>As it is now we hold the queue that we are leaving's lock when we
>call putnext() or qreply().  This would allow more parallelism since
>you wouldn't lock the entire side of a stream as you pass a message
>downstream and even more so in the case of ioctls() where you end up
>having to hold evey lock in the stream by the time that the message
>makes it back up to the stream head.
>
>I haven't had  chance to try and answer this for myself as I've been
>tied up with other projects but I'm planning on check it out sometime
>this week.
>
>Is there something fundamental that I'm missing here, Dave?

It is just necessary to know that for optimal performance there is no "one 
size fits all" solution to this problem.  If you are satisfied with the 
performance characteristics you get with qlock=2 then you're 
done.  Otherwise qlock=0 and custom locking is for you.

-- Dave