Re: WG: AW: AW: AW: WG: Re: read performance much slower than write

In article <512FA669CAABD711AFC000047581AD0932BC@FORD-PREFECT> you wrote:
> Hope you are well again!

Well is relative, but I am better. Just returned from a plane trip. The
pain of the journey erased the pain from my kidneys!

> I discovered that changing options on the client side had no real effect:
> options tried out additional to -b 4096:
> -n 1 -n 2 -n 3 ... to 8
> -m
> -a (both client and server)
> -e

OK.


> options on the server side additional to -b 4096:
> -a (also on client)
> -K
> -h 0 to 10000

> and finally -n

> The option -n made a huge difference! The read performance with raid5 using

This simply makes the server open the resource using direct i/o,
removing the serverside kernel/fs/block cache from the equation.

> 4 devices changed from 26MB/s up to 43MB/s. Unfortunately the write

Why read should be faster using direct i/o I find impossible to
understand! I don't think you would have been caching on read anyway,
or at least I mean that you would not have been reading cached data. So
one would expect that read was the same speed. Look-ahead would be
cut, but that's all. If you see a difference, then the kernel has a
problem with its caching algorithm.

I imagine a huge cache and a slow processor might be nasty. What was
the output from "free" like during slow reads? Or meminfo? ANy
noticable difference wrt fast reads under direct i/o?

> performance is almost cut into half the speed (from 65MB/S to 36MB/s which
> is not bad) with this option.

Well, that makes some sense. It is writing directly to disk without any
buffering. I would epect it to be a little slower.

You can compare speed with/without direct i/o on the server side using a
simple c program. I'll write it if you like.


> Another test was also interesting to me. When I used 2 different
> client-machines with the same setup using raid5 with 4 devices (2 partitions
> exported on each server side) I got always half the read speed compared to
> using 1 client-machine.
> I expected that using 2 clients with the same setup would result in doubling
> the performance (43MB/s twice), since the limit is not the network card.

Well, I'm not sure it is not the netork card. Many datums make it look
as though it is.

How can you use "2 clients"?

> test scenario:
> - 1 enbd-client using raid5 with 4 enbd-devices without the option -n on the
> 4 servers:
>   26MB/s measured with time dd 

> - 2 enbd-clients both using raid5 with 4 enbd-devices without the option -n
> on the 4 servers:
>   13MB/s measured with time dd on both clients

You need to measure cpu loads for this to make any sense. But I am
unsure of the experimental setup anyway.


> - 1 enbd-client using raid5 with 4 enbd-devices using the option -n on the 4
> servers:
>   43MB/s measured with time dd

> - 1 enbd-client using raid5 with 4 enbd-devices using the option -n on the 4
> servers:
>   27MB/s measured with time dd on first client
>   15MB/s measured with time dd on second client


> I do not understand the test results! The problem seems to be on the
> server-side of enbd. The servers only deliver the maximum data rate of
> 43MB/s. But if I reduce the servers used (from 4 to 3 servers) the speed
> increases which seems to be a problem on the client side.
> 1 backend=65MB/s
> 2 backends=79MB/s (65MB/s with raid1)
> 3 backends=51MB/s raid5
> 4 backends=43MB/s raid5


> I am more confused than ever before but maybe you can use the test results!

You need to measure much more at the same time for the data to be more
meaningful. In particular network and cpu loading values are needed.

For the moment (late night!) I have no particular insight.  That you see
an improvement using direct i/o rather implies that the servers kernel
block cache is filling and choking the network - but that's
incomprehenisble.  It should only happen on write, if at all.  On read,
it's silly.

But you would need to measure the network loading (timewise) to see such
an effect. You would see the network ceasing to pass packets for a
while, then resuming, then stopping, etc.

Maybe Arne can tell you how to make such measurements.


> best whishes for your health,

Thanks!

Peter