Minor bug in lis_process_rput() LiS 2.18 that probably doesn't effect anyone except me.

Hi Dave...

I wanted to let you know that I found a minor bug in LiS 2.18 in
lis_process_rput() in head/head.c:

*** 3037,3043 ****
      {
        freemsg(mp) ;
        lis_stream_error(shead, err, err) ;
-       return;
      }

      CP(q,mp) ;
--- 3037,3042 ----

If the lis_lockq() right at the top of this routine fails it frees
the message, but then goes on to process it normally.
I had a quick look for similar occurances, but didn't see any.

Granted I found this bug by doing something bad, which is to put back
the code in lis_setqsched() which calls lis_run_queues() directly if
can_call is set and !intr, even if we're in SMP mode.  I think
that being on a SMP machine is what is allowing this signal to break
lis_lockq() here in lis_process_rput().

Why am I doing this, you ask?  I'm trying to improve our driver's SMP
performance.  I'm seeing that it takes excessive periods of time,
7 to 10ms sometimes, to get our driver's read service routine to run.
 This (and a host of other small delays that you don't have on UP
machines) is causing our SMP performance to be about 1/2 to 3/4 of
what we get on a UP machine with lower throughput and higher
latencies and higher CPU utilization.  I am pretty convinced that it
is not just a problem with our driver because I don't get the same
problems with Solaris where adding more CPUs increases the
performance.  I also measured the various delays from a packet being
putmsg()'d from user space, processed in the wput routine, wsrv
routine, the packet comming back and having it's indication processed
back in the ISR, copying it into a buffer in rsrv, then the time from
putnext in the rsrv routine to getmsg() back in user space.  I found
that 90% of the delay is in the period of time from after the ISR
qenables the rsrv routine to when the rsrv routine processes the
frame and from the time that the rsrv does putnext() to the stream
head to when the application does getmsg().  I've seen delays as long
as 10-20ms from the ISR processing the indication and qenabling the
queue to when the ISR actually runs and delays from 20-70ms from when
the rsrv putnext()'s the message to when the application actually
reads it.

The application forks a process which sits in a while loop calling
putmsg() and poll() when putmsg() fails and a thread which sets in a
while loop calling getmsg() and then poll() when getmsg() fails, so
there is not really any excessive computation going on in user space.

So I've been on a desperate hunt over the last week trying to find
things to tweak to try and improve the SMP performance.  The bad news
is that I've not found and one thing that will just "fix" it.
It seems most the extra delay is just from extra scheduling overhead
everywhere that is just a bit longer, but quickly adds up.

About the only thing that I could find that made any real difference
is putting the direct calling of lis_run_queues back into
lis_setqsched().  This improves things a bit, but not a whole lot
since our driver's read service routine is going to have to run from
the queue runner thread 90% of the time any way since 90% of the time
it is only started from a qenable from our ISR.  (yes, I realize that
we can't call lis_run_queues directly in 2.6 kernels, but I'll worry
about that once our drivers get there.  That's part of the reason
that I don't want to release these changes quite yet to everybody...)

Doing this ran me into the problem that you warned about in
lis_setqsched where in previous versions you had:
 if (num_cpus)
      can_call = 0;

and a comment about sigals flying around that will interrupt lis_down
in the queue runners.  That's what led me to this bug when I couldn't
unload and reload LiS due to the double freemsg() caused by this bug
above.....

So, in short, I found the bug doing something semi-illegal, but it's
still a bug.....

BTW: LiS 2.18 is much better than 2.17.2 which is better than
2.16.18, but it still has significantly worse performance in SMP mode
than in UP mode.

If you have any ideas about what else might be a sticking point for
SMP operation in here, I'd love to hear about it and take a shot at
trying it.

I'm sorry to hear that you're not the offical maintainer anymore.  I
don't necessarily blame you as I'm sure that it's a lot of work, but
we'll certainly miss you and appreciate all of the work that you've
done on LiS.  We almost certainly would not have the Linux product
line that we do now if not for you and LiS.

thanks again...

dan



=====
Dan Gora
Software Engineer
Adax, Inc.

Tel: +55 12-3845-3572
email: [email protected]