Re: WG: Re: read performance much slower than write

"Also sprach Hoffmann Christian:"
[Charset iso-8859-1 unsupported, filtering to ASCII...]
> Thanks, I did lots of testing with the following setups:
> 
> On local disk (3ware Escalade 9600 with 4 SATA 250GB disks raid 5) on one of
> the enbd-servers with reiserfs (same result on 3 other servers):
> Version  1.03
>      ------Sequential Output------ 
>      -Per Chr- --Block-- -Rewrite- 
> Size K/sec %CP K/sec %CP K/sec %CP 
>   4G 41786  99 79583  29 48658  17 

Well, if I understand you correctly, this is reiserfs over a local
disk. Notice that character write  saturates the cpu at 40MB/s, so
it would seem that a single character takes 50ns to copy, and say that
is about 20 clock cycles, then the machine is a 800MHz machine.

Block write is twice as fast, per byte, for only one third the loading,
so I presume the disk limits are being reached (or the cpu would not be
unsaturated).

> --Sequential Input- --Random-
> -Per Chr- --Block-- --Seeks--
> K/sec %CP K/sec %CP  /sec %CP
> 37161  88 163373  41 263.6  1

Character reading is the same speed as writing, and the cpu is again
saturated. That makes sense. The disk is idle and you are measuring
memory copy.

Block reading is four times as fast. The cpu is idle half the time, and
I presume the disk is limiting.

The seek speed is way too high.  Old unix lore says that seeks tend to
30/s, always.  You must be seeking over only a tiny fraction of the
disk.


>       ------Sequential Create------ 

I don't know what that is. Some file system measure on making small
files?

>       -Create-- --Read--- -Delete-- 
> files  /sec %CP  /sec %CP  /sec %CP 
>    16 26591  99 +++++ +++ 22029 100 
> 
> --------Random Create--------
> -Create-- --Read--- -Delete--
>  /sec %CP  /sec %CP  /sec %CP
> 25958  99 +++++ +++ 20655  99
> 
> 
> 
> On one enbd-device (enbd-client) not using raid (/dev/nde only this device
> is loaded) with reiserfs:


Well, I would have liked to see the raw figure, but it is OK since you
have a cmparison in the experiment above.

> Version  1.03       
>      ------Sequential Output------ 
>      -Per Chr- --Block-- -Rewrite- 
> Size K/sec %CP K/sec %CP K/sec %CP 
>   4G 34954  97 89405  51 20011   6 

Well, character write is again limited at close to 40MB/s by the CPU
(there is oberhead in enbd). This implies that the network is extremely
fast!

Block write is at 90MB/s, faster than with reiserfs on the server side.

That result is impossible, so you can stop right there - the
measurement is not meaningful. It implies that you are measuring
something that does not have physical significance, since the result is
physically impossible.


> --Sequential Input- --Random-
> -Per Chr- --Block-- --Seeks--
> K/sec %CP K/sec %CP  /sec %CP
> 26427  65 44505   7 184.8   0

Character read has dropped, and the cpu is idle, implying that the
transport is limiting. That makes sense, since we know that small
reads over a fast network are horrible - either the overhead is
immense per packet, or packets will be held up until they can be
bunched together.

Block read is at 44MB/s and the cpu is idle. It looks as though
transport is influencial.


> 
>       ------Sequential Create------ 
>       -Create-- --Read--- -Delete-- 
> files  /sec %CP  /sec %CP  /sec %CP 
>    16 24953  99 +++++ +++ 21020 100 
> 
> --------Random Create--------
> -Create-- --Read--- -Delete--
>  /sec %CP  /sec %CP  /sec %CP
> 24748  99 +++++ +++ 19699 100
> 
> 
> 
> On enbd-client using raid5 (/dev/nda /dev/ndb /dev/ndc /dev/ndd)with
> reiserfs:

Well, here you add raid5 to the above experiment, under the reiserfs.
It's not very meaningful until the same thing can be measured without
enbd.


> Version  1.03       
>      ------Sequential Output------ 
>      -Per Chr- --Block-- -Rewrite- 
> Size K/sec %CP K/sec %CP K/sec %CP 
>   4G 32725  96 70303  43 18766   9

It slows down block writing - that is to be expected, because each high
level write now forces two disk writes and one or two reads.  In the
right order.  CPU is low, so presumably it is the sequential nature of
the read read write write sequence that slows things down - there can be
no request amalgamation. I would need to see the enbd spectrum data to
be sure.

> --Sequential Input- --Random-
> -Per Chr- --Block-- --Seeks--
> K/sec %CP K/sec %CP  /sec %CP
> 12271  35 12439   3 650.4   2

Now reading is slowed also - but it's a raid effect, as we can see by
comparing the speeds without raid. Probably the block size does not
match the raid stripe size, or some such thing.


> 
>       ------Sequential Create------ 
>       -Create-- --Read--- -Delete-- 
> files  /sec %CP  /sec %CP  /sec %CP 
>    16 26404  99 +++++ +++ 22008 100 
> 
> --------Random Create--------
> -Create-- --Read--- -Delete--
>  /sec %CP  /sec %CP  /sec %CP
> 25597 100 +++++ +++ 20470  99
> 
> 
> I am using Gigabit Ethernet cards on all servers and client and a D-Link
> Gigabit switch.

Well, I would suggest you drop to 100BT and see if it makes a
difference.  Gigabit ethernet will probably use packet aggregation,
which is precisely what you don't want in a to-and-fro protocol.


> on the local disks the read operations per block are about twice as fast as
> writing. As soon as I start using enbd the performance for read per block is

No - the data above does not support that assessment. 

You would need to test using enbd as the transport versus some other
transport to bbe sure. To me it looked as though the performance drop
as due to using raid. It's likely that each read block spits into reads of
different stripes, with high overhead from pulling the whole stripe in
when you are reading only part of it (the block). Or some other related
effect.

Anyway, you could tell by swiching raid in or out, or rynning raid1
instead, or running raid5 with only one disk(or anything else
degenerate).

> about half of write. Using software raid cuts it down to 1/7.

Looks like it's raid!

> The performance per Chr is not breaking in that much.
> I also checked if I am using the same blocksize on server and client and I
> do.
> 
> I searched the logs and could not find anything there.

Well, what does nbdinfo say? Anyway, you want to be running your own read
and write tests, since the bonnie result was physically impossible at
times, therefore incorrect.

Can you please test the difference between

    enbd -> raid -> disk
and

    raid -> enbd -> disk

(no reiserfs!). Just read and write tests. Use time dd.

Peter