Re: u3+ isos for testing

Hi,


On Tue, Oct 26, 2004 at 10:50:56AM +0200, Remco Barendse wrote:
> On Mon, 25 Oct 2004, Pasi Pirhonen wrote:
> 
> >
> >The ext3 problems i am seeing are not the drivers fault and i am pretty
> >sure that it's those nasty lock_kernel() and it's other locking friends
> >that are actually causing the random performance drop. It's still under
> >investigation/debugging.
> 
> Interesting!! I kept on thinking about the performance problem and this 
> morning I thought about this e-mail you wrote. If I understand correctly 
> the performance problem is not really filesystem type related.


No. It's definetly restricted to ext3 (even ext2 does perform well).
It's _not_ any way related to 3w-9xxx as i've seen mentios for same
kind of behaviour normal SCSI-hardware.

With XFS i have never seen any perfoemance degration what so ever. With
very same hardware and config.

> 
> When I was buying my raptors I looked on several sites for comments about 
> it. I remember seeing a comment somewhere that the Raptors *only support 
> 9 or 10 (don't know exactly) queued commands per drive.
> 

If you take a peek to drivers/scsi, you see that many SCSI HBAs are
restrciting the TCQ to 8 anyway.


> Could this be the root of the problem? If the 3Ware fails to detect how 
> many TCQ's a drive can handle and just assumes it's 25 or 64 for example 
> when in reality the drive supports only 10 I can imagine that weird stuff 
> happens when occasionally a lot of drive commands are queued.
> 
> Real scsi drives can handle much more queued commands maybe 3Ware did not 
> anticipate for drives only supporting a limited number.
> 
> That would make it a driver problem after all.
> 
> Just a n00b's thought and thanks for your great analysis, very usefull.
> 

No. There is nothing leading to even suspect it being 3ware problem.
Maybe in that matter that the 8-way RAID is fast enought to trigger the
situation easily (which really seems to be locking problem).

I actually initially wrote some more hooks in 3w-9xxx and monitored
those via procfs. Nothing gave me reson to suspect the driver working
wrong. Oprofile shows same direction ewven tho i don't undestan all
that i see with profiling (i'll evolve :)

I don't have facts now available, but i seem to remeber jbd spending
most of it's time at do_get_write_access or something which is bloking
anyway. ext3 top being some .text.lock.inode or similar.

Spcially jbd seem to use lock_kernel() several ocassion. I don't know
kernel internals, but from name and history, i'd assume that it's
really the monolithic lock_kernel (locks ALL CPUs maybe).

Maybe even true as i was running a long test with UP kernel (same setup
other way) and there was not a sinle I/O stall with some 250 loops of
tiobench. Normally one gets the first degration before 10 loop are
done. It's mostly triggered when one has 4 parallel (or more) threads
writing on 2-way machine. Sometimes all writing (1,2,4,8 threads) is
stalled for several rounds of tiobench. But tiobench just being here 'a
tool which easily triggers the problem'. It's not way absolute pattern
for uitable I/O. It's just too narrowed to be that kind of utility.

The 3ware is handlin queues up to that 255 (or 254 or whatever) just
fine. For sequential patterns that is just fine, but when you push
random reads between sequential writes, the linux elevator does put
that 'almoust top of the queue'. That doesn't much help the speed if
the 3ware controller has already 254 pending writes _ahead_ of that
elevator.

generally i've found that using TCQ over 8 isn't much helping the
performance of 3ware (for the patterns i have been testing). I've even
tested a low as 2 and that seem to be a little slower.

For responsiveness the TCQ=8 is much better than TCQ=254 bot above
mentioned reasons. Linux elvator (which is even on the fly tunable)
does know much better about ordering of the request than the dummy
driver which is communicating with the firmware of the controller.

Basically the 3w-9xxx is same driver than 3w-xxx. It's not much
recoded. The 'intelligent' being the firmware running on the
controller.

I know about NCQ (native command queueing), but i don't have any facts
for it and how much better that would be. Future will show as
some of theSATA-hardware is already NCQ capable.



-- 
Pasi Pirhonen - [email protected] - http://iki.fi/upi/