WG: AW: AW: AW: WG: Re: read performance much slower than write
Hoffmann Christian <[email protected]>
| Newsgroups | gmane.linux.enbd.general |
|---|---|
| Message-ID | <512FA669CAABD711AFC000047581AD0932BA@FORD-PREFECT> |
Hi Peter!
I already sent this mail yesterday but I am not shure if it came through. So
I send it again, sorry if it comes twice.
>Well, OK. It is possible that it needs to be set both sides (and maybe a
>direct cable used).
I raised it on both sides client and servers to mtu9000 and the switch
supports large packets! It is a managed gigabit switch and I found an option
to raise the mtu to 9000 as well. I did test with raised mtu (9000) and
regular mtu (1500). The mtu did not change the test results.
>I forget how many channels each enbd device as using .. can you remind
>me?
I am using 2 channels for each device with option -n 2 on client side.
>And with raid? Can one run raid5 with two disks? I suppose so.
No! Raid5 needs at least 3 devices to work.
>That's fine. But the nbdinfo counts you showed said that all four
>together are doing 100MB/s, which is why I want you to confirm that by
>measuring the network thruput. It asn't measurable from the data yu
>provided earlier because a rollover had occurred in the count.
I do not understand it. nbdinfo shows 100MB/s but nload on the client states
that not the full bandwith possible is used! I measured it several times to
be shure and get the same result every time. As soon as I use more than 2
devices of enbd, no matter raid or not the bandwith measured with nload on
the client-gigabit card drops under the possible max. 3 devices 450MBit/s
which is about the half bandwith, 4 devices 250MBit/s which is 1/4.
But maybe nload is not the right tool to measure...
Anyway I tested again with 1,2,3,4 devices without raid and 3,4 enbd devices
WITH raid5 using ifconfig before and after the test!
>That's fine. But the nbdinfo counts you showed said that all four
>together are doing 100MB/s, which is why I want you to confirm that by
>measuring the network thruput. It asn't measurable from the data yu
>provided earlier because a rollover had occurred in the count.
I hope I could avoid the rollover by using smaller test scenarios.
(count=256000) However in the 4th test a rollover occured, the rollover
happens at 4GB.
>But I really don't understand the readings, please can you be CLEAR
>about what you are measuring.
I try to be as clear as possible,
- Single enbd-device no raid:
time dd count=256000 if=/dev/nda of=/dev/null bs=4K
256000+0 Datensätze ein
256000+0 Datensätze aus
1048576000 bytes transferred in 16,932242 seconds (61927771 bytes/sec)
real 0m16.946s
user 0m0.121s
sys 0m1.248s
Before the test:
eth0 Protokoll:Ethernet Hardware Adresse 00:04:23:B0:DA:64
inet Adresse:192.168.100.180 Bcast:192.168.100.255
Maske:255.255.255.0
inet6 Adresse: fe80::204:23ff:feb0:da64/64
Gültigkeitsbereich:Verbindung
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:114497850 errors:0 dropped:0 overruns:0 frame:0
TX packets:65031834 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:1000
RX bytes:1113140830 (1.0 GiB) TX bytes:4014251066 (3.7 GiB)
Basisadresse:0xac00 Speicher:fc4e0000-fc500000
After the test:
eth0 Protokoll:Ethernet Hardware Adresse 00:04:23:B0:DA:64
inet Adresse:192.168.100.180 Bcast:192.168.100.255
Maske:255.255.255.0
inet6 Adresse: fe80::204:23ff:feb0:da64/64
Gültigkeitsbereich:Verbindung
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:115234082 errors:0 dropped:0 overruns:0 frame:0
TX packets:65407927 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:1000
RX bytes:2214005775 (2.0 GiB) TX bytes:4041606452 (3.7 GiB)
Basisadresse:0xac00 Speicher:fc4e0000-fc500000
- 2 enbd-devices no raid:
Device1
time dd count=256000 if=/dev/nda of=/dev/null bs=4K
256000+0 Datensätze ein
256000+0 Datensätze aus
1048576000 bytes transferred in 24,305642 seconds (43141259 bytes/sec)
real 0m24.317s
user 0m0.149s
sys 0m1.844s
Device2
time dd count=256000 if=/dev/ndb of=/dev/null bs=4K
256000+0 Datensätze ein
256000+0 Datensätze aus
1048576000 bytes transferred in 24,195133 seconds (43338303 bytes/sec)
real 0m24.198s
user 0m0.151s
sys 0m1.776s
Before the test:
eth0 Protokoll:Ethernet Hardware Adresse 00:04:23:B0:DA:64
inet Adresse:192.168.100.180 Bcast:192.168.100.255
Maske:255.255.255.0
inet6 Adresse: fe80::204:23ff:feb0:da64/64
Gültigkeitsbereich:Verbindung
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:116706753 errors:0 dropped:0 overruns:0 frame:0
TX packets:66112472 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:1000
RX bytes:120794351 (115.1 MiB) TX bytes:4092992851 (3.8 GiB)
Basisadresse:0xac00 Speicher:fc4e0000-fc500000
After the test:
eth0 Protokoll:Ethernet Hardware Adresse 00:04:23:B0:DA:64
inet Adresse:192.168.100.180 Bcast:192.168.100.255
Maske:255.255.255.0
inet6 Adresse: fe80::204:23ff:feb0:da64/64
Gültigkeitsbereich:Verbindung
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:118179158 errors:0 dropped:0 overruns:0 frame:0
TX packets:66817272 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:1000
RX bytes:2322516911 (2.1 GiB) TX bytes:4144382495 (3.8 GiB)
Basisadresse:0xac00 Speicher:fc4e0000-fc500000
- 3 enbd-devices no raid:
Device1
time dd count=256000 if=/dev/nda of=/dev/null bs=4K
256000+0 Datensätze ein
256000+0 Datensätze aus
1048576000 bytes transferred in 28,587029 seconds (36680132 bytes/sec)
real 0m28.589s
user 0m0.157s
sys 0m1.956s
Device2
time dd count=256000 if=/dev/ndb of=/dev/null bs=4K
256000+0 Datensätze ein
256000+0 Datensätze aus
1048576000 bytes transferred in 29,312626 seconds (35772162 bytes/sec)
real 0m29.316s
user 0m0.162s
sys 0m2.097s
Device3
time dd count=256000 if=/dev/ndc of=/dev/null bs=4K
256000+0 Datensätze ein
256000+0 Datensätze aus
1048576000 bytes transferred in 29,664052 seconds (35348374 bytes/sec)
real 0m29.668s
user 0m0.164s
sys 0m1.944s
Before the test:
eth0 Protokoll:Ethernet Hardware Adresse 00:04:23:B0:DA:64
inet Adresse:192.168.100.180 Bcast:192.168.100.255
Maske:255.255.255.0
inet6 Adresse: fe80::204:23ff:feb0:da64/64
Gültigkeitsbereich:Verbindung
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:108619601 errors:0 dropped:0 overruns:0 frame:0
TX packets:62219542 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:1000
RX bytes:917551063 (875.0 MiB) TX bytes:3808816367 (3.5 GiB)
Basisadresse:0xac00 Speicher:fc4e0000-fc500000
After the test:
eth0 Protokoll:Ethernet Hardware Adresse 00:04:23:B0:DA:64
inet Adresse:192.168.100.180 Bcast:192.168.100.255
Maske:255.255.255.0
inet6 Adresse: fe80::204:23ff:feb0:da64/64
Gültigkeitsbereich:Verbindung
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:110813626 errors:0 dropped:0 overruns:0 frame:0
TX packets:63250662 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:1000
RX bytes:4198357370 (3.9 GiB) TX bytes:3884052255 (3.6 GiB)
Basisadresse:0xac00 Speicher:fc4e0000-fc500000
- 4 enbd-devices no raid:
Device1
time dd count=256000 if=/dev/nda of=/dev/null bs=4K
256000+0 Datensätze ein
256000+0 Datensätze aus
1048576000 bytes transferred in 38,401943 seconds (27305285 bytes/sec)
real 0m38.404s
user 0m0.157s
sys 0m1.951s
Device2
time dd count=256000 if=/dev/ndb of=/dev/null bs=4K
256000+0 Datensätze ein
256000+0 Datensätze aus
1048576000 bytes transferred in 40,128271 seconds (26130605 bytes/sec)
real 0m40.131s
user 0m0.162s
sys 0m2.053s
Device3
time dd count=256000 if=/dev/ndc of=/dev/null bs=4K
256000+0 Datensätze ein
256000+0 Datensätze aus
1048576000 bytes transferred in 39,771431 seconds (26365056 bytes/sec)
real 0m39.774s
user 0m0.176s
sys 0m1.972s
Device4
time dd count=256000 if=/dev/ndd of=/dev/null bs=4K
256000+0 Datensätze ein
256000+0 Datensätze aus
1048576000 bytes transferred in 40,185540 seconds (26093366 bytes/sec)
real 0m40.189s
user 0m0.165s
sys 0m2.004s
Before the test:
eth0 Protokoll:Ethernet Hardware Adresse 00:04:23:B0:DA:64
inet Adresse:192.168.100.180 Bcast:192.168.100.255
Maske:255.255.255.0
inet6 Adresse: fe80::204:23ff:feb0:da64/64
Gültigkeitsbereich:Verbindung
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:111550631 errors:0 dropped:0 overruns:0 frame:0
TX packets:63627435 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:1000
RX bytes:1004348854 (957.8 MiB) TX bytes:3911496438 (3.6 GiB)
Basisadresse:0xac00 Speicher:fc4e0000-fc500000
After the test:
eth0 Protokoll:Ethernet Hardware Adresse 00:04:23:B0:DA:64
inet Adresse:192.168.100.180 Bcast:192.168.100.255
Maske:255.255.255.0
inet6 Adresse: fe80::204:23ff:feb0:da64/64
Gültigkeitsbereich:Verbindung
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:114495395 errors:0 dropped:0 overruns:0 frame:0
TX packets:65029872 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:1000
RX bytes:1112821920 (1.0 GiB) TX bytes:4013987396 (3.7 GiB)
Basisadresse:0xac00 Speicher:fc4e0000-fc500000
- 3 enbd-devices WITH raid:
md0 consists of 3 enbd-devices (/dev/nda,/dev/ndb,/dev/ndc) using raid5:
time dd count=256000 if=/dev/md0 of=/dev/null bs=4K
256000+0 Datensätze ein
256000+0 Datensätze aus
1048576000 bytes transferred in 210,897904 seconds (4971960 bytes/sec)
real 3m30.899s
user 0m0.152s
sys 0m2.888s
eth0 Protokoll:Ethernet Hardware Adresse 00:04:23:B0:DA:64
inet Adresse:192.168.100.180 Bcast:192.168.100.255
Maske:255.255.255.0
inet6 Adresse: fe80::204:23ff:feb0:da64/64
Gültigkeitsbereich:Verbindung
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:133212081 errors:0 dropped:0 overruns:0 frame:0
TX packets:74509681 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:1000
RX bytes:2522782504 (2.3 GiB) TX bytes:1194551511 (1.1 GiB)
Basisadresse:0xac00 Speicher:fc4e0000-fc500000
eth0 Protokoll:Ethernet Hardware Adresse 00:04:23:B0:DA:64
inet Adresse:192.168.100.180 Bcast:192.168.100.255
Maske:255.255.255.0
inet6 Adresse: fe80::204:23ff:feb0:da64/64
Gültigkeitsbereich:Verbindung
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:133963706 errors:0 dropped:0 overruns:0 frame:0
TX packets:74933964 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:1000
RX bytes:3625612953 (3.3 GiB) TX bytes:1227052810 (1.1 GiB)
Basisadresse:0xac00 Speicher:fc4e0000-fc500000
- 4 enbd-devices WITH raid:
md0 consists of 4 enbd-devices (/dev/nda,/dev/ndb,/dev/ndc,/dev/ndd) using
raid5:
time dd count=256000 if=/dev/md0 of=/dev/null bs=4K
256000+0 Datensätze ein
256000+0 Datensätze aus
1048576000 bytes transferred in 36,374947 seconds (28826874 bytes/sec)
real 0m36.376s
user 0m0.158s
sys 0m3.128s
Before the test:
eth0 Protokoll:Ethernet Hardware Adresse 00:04:23:B0:DA:64
inet Adresse:192.168.100.180 Bcast:192.168.100.255
Maske:255.255.255.0
inet6 Adresse: fe80::204:23ff:feb0:da64/64
Gültigkeitsbereich:Verbindung
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:118180316 errors:0 dropped:0 overruns:0 frame:0
TX packets:66818156 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:1000
RX bytes:2322928883 (2.1 GiB) TX bytes:4144498195 (3.8 GiB)
Basisadresse:0xac00 Speicher:fc4e0000-fc500000
After the test:
eth0 Protokoll:Ethernet Hardware Adresse 00:04:23:B0:DA:64
inet Adresse:192.168.100.180 Bcast:192.168.100.255
Maske:255.255.255.0
inet6 Adresse: fe80::204:23ff:feb0:da64/64
Gültigkeitsbereich:Verbindung
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:119058734 errors:0 dropped:0 overruns:0 frame:0
TX packets:67347312 errors:0 dropped:0 overruns:0 carrier:0
Kollisionen:0 Sendewarteschlangenlänge:1000
RX bytes:3440286568 (3.2 GiB) TX bytes:4195646915 (3.9 GiB)
Basisadresse:0xac00 Speicher:fc4e0000-fc500000
Hope the tests are clear now,
Thanks,
Christian