Re: Minor bug in lis_process_rput() LiS 2.18 that probably doesn't effect anyone except me.

Thanks for the buglet fix.

I too was chasing latency problems over the summer.  I had (still have) at 
my disposal a couple of IBM x335 machines.  These are 3.2GHz and have two 
Xeon CPUs, which Linux views as four CPUs.

I was also seeing some 20ms delays which turned out to be measurement error 
in my test program.  I forget the details just now but the program was 
basically putting the time stamp in the messages at the wrong moment in 
time.  It kind of got out of phase with itself.

-- Now I remember -- I was using poll as you are and I was over-managing 
the "read bits" thinking that I knew when to set them and when to clear 
them.  That was what got the program out of phase with itself.  The fix was 
simply to always keep the "read bits" set so that a returned packet could 
arrive at any time and have its time stamp evaluated.

One of the things that I put into LiS to assist with this was the spin lock 
contention and semaphore contention code.  Check out "streams -L" and 
"streams -T" in conjunction with "streams -D0x28".

I also found that there was a highly significant difference between Red Hat 
9 and a stock 2.4 kernel of the same nominal version.  RH 9 has the 2.6 O1 
processor scheduler back-ported to it.  It makes a huge difference if you 
have a number of threads active.

I played with oprofile but found it not to be the correct tool for 
diagnosing latencies.  It only tells you where the CPU *is* executing, not 
where is *isn't* executing.  If CPU bottlenecks are your problem oprofile 
is a really cool tool.

Once I was able to get lock and semaphore contention to a minimum I was 
able to get some real pipelining going among the 4 CPUs.  This allowed me 
to tune the number of messages queued before waking up another LiS queue 
runner thread.  I found in my testing that 12-13 messages seemed to be 
about right, so I went for the Fibonacci 13.

Oh, another tip:  In an SMP system you are best off if you can run your 
driver with qlock=0, that is, no locking of your queue by LiS prior to 
driver entry.  This is what allows your put procedure to enqueue a message 
on one CPU at the same time that the same service procedure is operating on 
another one on another CPU.  This is what gets the pipeline going.

-- Dave

At 07:52 PM 10/28/2004, Dan Gora wrote:


>Hi Dave...
>
>I wanted to let you know that I found a minor bug in LiS 2.18 in
>lis_process_rput() in head/head.c:
>
>*** 3037,3043 ****
>       {
>         freemsg(mp) ;
>         lis_stream_error(shead, err, err) ;
>-       return;
>       }
>
>       CP(q,mp) ;
>--- 3037,3042 ----
>
>If the lis_lockq() right at the top of this routine fails it frees
>the message, but then goes on to process it normally.
>I had a quick look for similar occurances, but didn't see any.
>
>Granted I found this bug by doing something bad, which is to put back
>the code in lis_setqsched() which calls lis_run_queues() directly if
>can_call is set and !intr, even if we're in SMP mode.  I think
>that being on a SMP machine is what is allowing this signal to break
>lis_lockq() here in lis_process_rput().
>
>Why am I doing this, you ask?  I'm trying to improve our driver's SMP
>performance.  I'm seeing that it takes excessive periods of time,
>7 to 10ms sometimes, to get our driver's read service routine to run.
>  This (and a host of other small delays that you don't have on UP
>machines) is causing our SMP performance to be about 1/2 to 3/4 of
>what we get on a UP machine with lower throughput and higher
>latencies and higher CPU utilization.  I am pretty convinced that it
>is not just a problem with our driver because I don't get the same
>problems with Solaris where adding more CPUs increases the
>performance.  I also measured the various delays from a packet being
>putmsg()'d from user space, processed in the wput routine, wsrv
>routine, the packet comming back and having it's indication processed
>back in the ISR, copying it into a buffer in rsrv, then the time from
>putnext in the rsrv routine to getmsg() back in user space.  I found
>that 90% of the delay is in the period of time from after the ISR
>qenables the rsrv routine to when the rsrv routine processes the
>frame and from the time that the rsrv does putnext() to the stream
>head to when the application does getmsg().  I've seen delays as long
>as 10-20ms from the ISR processing the indication and qenabling the
>queue to when the ISR actually runs and delays from 20-70ms from when
>the rsrv putnext()'s the message to when the application actually
>reads it.
>
>The application forks a process which sits in a while loop calling
>putmsg() and poll() when putmsg() fails and a thread which sets in a
>while loop calling getmsg() and then poll() when getmsg() fails, so
>there is not really any excessive computation going on in user space.
>
>So I've been on a desperate hunt over the last week trying to find
>things to tweak to try and improve the SMP performance.  The bad news
>is that I've not found and one thing that will just "fix" it.
>It seems most the extra delay is just from extra scheduling overhead
>everywhere that is just a bit longer, but quickly adds up.
>
>About the only thing that I could find that made any real difference
>is putting the direct calling of lis_run_queues back into
>lis_setqsched().  This improves things a bit, but not a whole lot
>since our driver's read service routine is going to have to run from
>the queue runner thread 90% of the time any way since 90% of the time
>it is only started from a qenable from our ISR.  (yes, I realize that
>we can't call lis_run_queues directly in 2.6 kernels, but I'll worry
>about that once our drivers get there.  That's part of the reason
>that I don't want to release these changes quite yet to everybody...)
>
>Doing this ran me into the problem that you warned about in
>lis_setqsched where in previous versions you had:
>  if (num_cpus)
>       can_call = 0;
>
>and a comment about sigals flying around that will interrupt lis_down
>in the queue runners.  That's what led me to this bug when I couldn't
>unload and reload LiS due to the double freemsg() caused by this bug
>above.....
>
>So, in short, I found the bug doing something semi-illegal, but it's
>still a bug.....
>
>BTW: LiS 2.18 is much better than 2.17.2 which is better than
>2.16.18, but it still has significantly worse performance in SMP mode
>than in UP mode.
>
>If you have any ideas about what else might be a sticking point for
>SMP operation in here, I'd love to hear about it and take a shot at
>trying it.
>
>I'm sorry to hear that you're not the offical maintainer anymore.  I
>don't necessarily blame you as I'm sure that it's a lot of work, but
>we'll certainly miss you and appreciate all of the work that you've
>done on LiS.  We almost certainly would not have the Linux product
>line that we do now if not for you and LiS.
>
>thanks again...
>
>dan
>
>
>
>=====
>Dan Gora
>Software Engineer
>Adax, Inc.
>
>Tel: +55 12-3845-3572
>email: [email protected]