Re: AW: AW: WG: Re: read performance much slower than write

"Also sprach Hoffmann Christian:"
> Like you already guessed if I do not use merge_requests the performance
> drops signifficantly.

OK - that indicates that the problem is assciated with small requests
going across the Ge link, each of them being folloed by an ack in the
other direction.

At least it sounds plausible.

It might be possible to test that idea by changing the MTU on the link
... up to whatever is the maximum. I think Ge can take much more than
1500. You'll have to google for it.

> I am using kernel 2.6.8-2-smp (Debian-testing) patched with enbd-2.6.8.patch
> and fr1-2.17 patch on the client side on a P4 2.8GHZ computer.

Hmm. I think that in that case there IS a separate queue per device
... yes, there is:

    struct enbd_device *lo = &enbd_dev[i];
    struct gendisk *disk = alloc_disk(ENBD_MAXCONN);
    ...
    lo->disk = disk;
    disk->queue = blk_init_queue(do_enbd_request, &lo->lock);

So now I can't think of much to do about what is happening, if what I
think is happening is happening. Maybe we need a bit more data first,
and then some things will become clear.


> testing /dev/nda /dev/ndb /dev/ndc /dev/ndd without overlying raid:
> 
> Before the test:
> 
> eth0      Protokoll:Ethernet  Hardware Adresse 00:04:23:B0:DA:64
>           inet Adresse:192.168.100.180  Bcast:192.168.100.255
> Maske:255.255.255.0

All going through the same Ge channel might also be a factor. I don't
know. Try increasing the MTU. Anyone know what the max is for Ge? 9000?

>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:33059087 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:18616424 errors:0 dropped:0 overruns:0 carrier:0
>           Kollisionen:0 Sendewarteschlangenl_nge:1000
>           RX bytes:3682625881 (3.4 GiB)  TX bytes:1570968677 (1.4 GiB)

> 
> I started all 4 tests at the same time!!
> 
> time dd if=/dev/nda of=/dev/null bs=4K
> time dd if=/dev/ndb of=/dev/null bs=4K
> time dd if=/dev/ndc of=/dev/null bs=4K
> time dd if=/dev/ndd of=/dev/null bs=4K
> 
> /dev/nda
> 2002092+0 Datens_tze ein
> 2002092+0 Datens_tze aus
> 8200568832 bytes transferred in 309,167733 seconds (26524659 bytes/sec)
> 
> real    5m9.170s
> user    0m1.232s
> sys     0m15.475s
> 
> /dev/ndb
> 2002092+0 Datens_tze ein
> 2002092+0 Datens_tze aus
> 8200568832 bytes transferred in 312,078835 seconds (26277235 bytes/sec)

Uh, EACH of them did 26MB/s? That is 100MB/s.

> 
> real    5m12.082s
> user    0m1.227s
> sys     0m16.011s
> 
> /dev/ndc
> 2002092+0 Datens_tze ein
> 2002092+0 Datens_tze aus
> 8200568832 bytes transferred in 315,870283 seconds (25961824 bytes/sec)
> 
> real    5m15.873s
> user    0m1.183s
> sys     0m15.956s
> 
> /dev/ndd
> 2002092+0 Datens_tze ein
> 2002092+0 Datens_tze aus
> 8200568832 bytes transferred in 313,595331 seconds (26150162 bytes/sec)
> 
> real    5m13.599s
> user    0m1.237s
> sys     0m15.844s

Well, you got much more than with raid5 out of them, collectively. So
you have 4 disks, under raid5. Hmmm.

> During the test:
> 
> [a] Spectrum:   83%1    16%32

This one has fragmentation.

> [b] Spectrum:   99%32

This one is running at 128KB requests.

> [c] Spectrum:   99%32

Ditto.

> [d] Spectrum:   99%32

Ditto.

> And ifconfig after the test:
>           RX packets:56084952 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:29575748 errors:0 dropped:0 overruns:0 carrier:0
>           Kollisionen:0 Sendewarteschlangenl_nge:1000
>           RX bytes:3753057213 (3.4 GiB)  TX bytes:2372107819 (2.2 GiB)

Compared to before:
> 
>           RX packets:33059087 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:18616424 errors:0 dropped:0 overruns:0 carrier:0
>           Kollisionen:0 Sendewarteschlangenl_nge:1000
>           RX bytes:3682625881 (3.4 GiB)  TX bytes:1570968677 (1.4 GiB)

About 23M packets received, and 11M sent.

Almost no bytes were read! Something is wrong with these stats. Indeed,
the number received has gone down! Maybe I should assume rollover at
4GB? The sent bytes is 0.8GB, or abut 70B per packet, which I believe.
That would be about the right size for an ack.

If the received bytes is really about 4GB, then the average size per
packet would be about 160B. That's way too small. The rollover must
have happened several times.


> testing /dev/nda /dev/ndb /dev/ndc /dev/ndd using overlying raid:
> 
> Before the test:
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:90067710 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:45921220 errors:0 dropped:0 overruns:0 carrier:0
>           Kollisionen:0 Sendewarteschlangenl_nge:1000
>           RX bytes:2774836223 (2.5 GiB)  TX bytes:3586429308 (3.3 GiB)
> 
> time dd if=/dev/md0 of=/dev/null bs=4K
> 6006192+0 Datens_tze ein
> 6006192+0 Datens_tze aus
> 24601362432 bytes transferred in 912,996689 seconds (26945730 bytes/sec)

About 26MB/s. Well, that is the same as for one disk. It looks from
this as though something is serialising something.

Uh .. I need to see the data rate from a test on just a single dd on a
single enbd device. Was it 66MB/s? I seem to recall so from earlier
tests?

It begins to look as though the choke point is the single Ge device. It
might be that it is synchronous at a low level ... only one request at
a time goes through it. Enbd does async r/w on the net, but still, the
statistical behaviour might be near synchronous.


> [a] Queued:     +0R/0W curr (check 0R/0W) +62R/1W max

Quite low queue sizes.

> [a] Requested:  15.009M (6.90M) (8.10M) 15.00MR/0W      max 63
> [a] Despatched: 15.009M (6.90M) (8.10M) 15.00MR/0W      md5 0W (0 eq, 0 ne,
> 0 dn)
> [a] Errored:    40      (5)     (5)     10+30
> [a] Pending:    0       (0)     (0)     0R/0W+0R/0W
> [a] B/s now:    6.69M   (6.69MR+0W)
> [a] B/s ave:    3.95G   (3.95GR+0W)

Huge speed measured. Strange.

> [a] B/s max:    1.25G   (1.25GR+0W)
> [a] Spectrum:   81%1    17%32

Low packet size.

> [c] Requested:  4.7644M (2.37M) (2.39M) 4.764MR/0W      max 36
> [c] Despatched: 4.7644M (2.37M) (2.39M) 4.764MR/0W      md5 0W (0 eq, 0 ne,
> 0 dn)
> [c] Errored:    0       (0)     (0)     0+0
> [c] Pending:    0       (0)     (0)     0R/0W+0R/0W
> [c] B/s now:    6.07M   (6.07MR+0W)
> [c] B/s ave:    21.3M   (21.3MR+0W)

20MB/s.

> [c] B/s max:    1.72G   (1.72GR+0W)
> [c] Spectrum:   1%1     1%2     3%3     93%32

Large packets.

> [d] Despatched: 4.6537M (2.32M) (2.32M) 4.653MR/0W      md5 0W (0 eq, 0 ne,
> 0 dn)
> [d] Errored:    0       (0)     (0)     0+0
> [d] Pending:    6       (3)     (3)     6R/0W+0R/0W
> [d] B/s now:    6.30M   (6.30MR+0W)
> [d] B/s ave:    17.6M   (17.6MR+0W)

17MB/s.

> [d] B/s max:    1.64G   (1.64GR+0W)
> [d] Spectrum:   1%1     1%2     3%3     93%32

Large packets.

> 
> After the test:
> eth0      Protokoll:Ethernet  Hardware Adresse 00:04:23:B0:DA:64
>           inet Adresse:192.168.100.180  Bcast:192.168.100.255
> Maske:255.255.255.0
>           inet6 Adresse: fe80::204:23ff:feb0:da64/64
> G_ltigkeitsbereich:Verbindung
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:110831283 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:58565821 errors:0 dropped:0 overruns:0 carrier:0
>           Kollisionen:0 Sendewarteschlangenl_nge:1000
>           RX bytes:3235496495 (3.0 GiB)  TX bytes:527740801 (503.2 MiB)
>           Basisadresse:0xac00 Speicher:fc4e0000-fc500000
> 
> 
> There is no difference in weather using raid or not. The performance is
> about the same. Running all 4 enbd devices at once breakes down the
> performance.

No - it sent it up to 100MB/s. When you run raid5 over enbd, it also
looks to me as though the bandwidth is about 100MB/s. But the ifconfig
results seemed to rollver.  Can you extract the measured rates from
each of th enebd devices, and from the ethernet device, please?

> 
> >I think one could load several enbd drivers, each on a different major,
> >and use one each for each enbd device of yours. That would separate the
> >queues. One can load the driver with major=XX. The result might be
> >better aggregation.
> 
> I am sorry, but I do not know how to try that out. How can I load the same
> module and assign it to a spezial enbd-device?

?? I don't understand ... I merely suggested that you load  the module
several times with different major numbers defined during the module
load. Then make and use special device files that talk to those majors.

But anyway, it doesn't seem necessary. It begins to look as though the
bandwidth is constant at about 100MB/s.

> 
> The reason why I am so interested in performance is that I found an article
> in a german magazine about enbd where they use almost the same setup as I
> do. They have also benchmark tests of their setup and get a very high read
> performance about 79MB/s. I include a link of this article. Even though it
> is in german there is a table of the benchmarks included. (bonnie++)
> http://www.linux-magazin.de/Artikel/ausgabe/2004/11/enbd/enbd.html

OK.

> Do you think it may help to increase the blocksize enbd is using? I tried to

No. One cannot. 4K is the maximum.

> do that, but I guess it is compiled to the kernel within the patch, because
> if I change the -b option on server and client it won_t start anymore. In
> the logs it states that the blocksize does not match.

That will be true! It must match the requests that are received.

Peter