AW: AW: Re: WG: AW: AW: AW: WG: Re: read performance much slow er than write

Hi Peter!

>These results are not likely - reading the resource ought to put all of
>it into the kernel cache. That has not happened. What happens when you
>just dd the resource into /dev/null? Does the cache grow any larger?

I always tested dd into /dev/null. The syntax was "time dd count=512000
if=/dev/md0 of=/dev/null bs=4K":
I am back now and first tried to do the memtests as requested. I have to
leave again today and will be away again till thursday. hope I can go on
with testing next week:

Here are the memtest-results for all 4 servers during testing and
/proc/nbdinfo form the client:

all 4 servers without option -n, client with merge_requests=512:

- server1 meminfo:
MemTotal:      1036524 kB
MemFree:        789112 kB
Buffers:        153572 kB
Cached:          38928 kB
SwapCached:          0 kB
Active:          62080 kB
Inactive:       135724 kB
HighTotal:      131008 kB
HighFree:        83552 kB
LowTotal:       905516 kB
LowFree:        705560 kB
SwapTotal:     1999992 kB
SwapFree:      1999992 kB
Dirty:              28 kB
Writeback:           0 kB
Mapped:          10360 kB
Slab:            42568 kB
Committed_AS:    19856 kB
PageTables:        324 kB
VmallocTotal:   114680 kB
VmallocUsed:      3516 kB
VmallocChunk:   110692 kB

- server2 meminfo:
MemTotal:      1036524 kB
MemFree:        790576 kB
Buffers:        169704 kB
Cached:          19540 kB
SwapCached:          0 kB
Active:          58116 kB
Inactive:       136452 kB
HighTotal:      131008 kB
HighFree:       102928 kB
LowTotal:       905516 kB
LowFree:        687648 kB
SwapTotal:     1999992 kB
SwapFree:      1999992 kB
Dirty:              32 kB
Writeback:           0 kB

- server3 meminfo:
MemTotal:      1036524 kB
MemFree:        785580 kB
Buffers:        173936 kB
Cached:          19992 kB
SwapCached:          0 kB
Active:          57580 kB
Inactive:       141672 kB
HighTotal:      131008 kB
HighFree:       102508 kB
LowTotal:       905516 kB
LowFree:        683072 kB
SwapTotal:     1999992 kB
SwapFree:      1999992 kB
Dirty:              32 kB
Writeback:           0 kB
Mapped:          10380 kB
Slab:            44704 kB
Committed_AS:    19892 kB
PageTables:        324 kB
VmallocTotal:   114680 kB
VmallocUsed:      3516 kB
VmallocChunk:   110692 kB

- server4 meminfo:
MemTotal:      1036524 kB
MemFree:        785668 kB
Buffers:        177772 kB
Cached:          15532 kB
SwapCached:          0 kB
Active:          53792 kB
Inactive:       144836 kB
HighTotal:      131008 kB
HighFree:       106904 kB
LowTotal:       905516 kB
LowFree:        678764 kB
SwapTotal:     1999992 kB
SwapFree:      1999992 kB
Dirty:              32 kB
Writeback:           0 kB
Mapped:          10380 kB
Slab:            45128 kB
Committed_AS:    19892 kB
PageTables:        324 kB
VmallocTotal:   114680 kB
VmallocUsed:      3560 kB
VmallocChunk:   110668 kB
Mapped:          10380 kB
Slab:            44268 kB
Committed_AS:    19892 kB
PageTables:        324 kB
VmallocTotal:   114680 kB
VmallocUsed:      3560 kB
VmallocChunk:   110668 kB

Here is the output of /proc/nbdinfo on client side:
Device a:       Open
[a] State:      verify, rw, merge requests, enabled, validated, show_errs,
last error 0, lives 7, bp 0
[a] Queued:     +1R/0W curr (check 1R/0W) +7R/1W max
[a] Buffersize: 262144  (sectors=512, blocks=64)
[a] Blocksize:  4096    (log=12)
[a] Size:       8008400KB
[a] Blocks:     2002100
[a] Sockets:    2       (*)     (+)
[a] Requested:  520.04K (259K)  (260K)  520.0KR/0W      max 16
[a] Despatched: 520.04K (259K)  (260K)  520.0KR/0W      md5 0W (0 eq, 0 ne,
0 dn)
[a] Errored:    0       (0)     (0)     0+0
[a] Pending:    6       (3)     (3)     6R/0W+0R/0W
[a] B/s now:    5.66M   (5.66MR+0W)
[a] B/s ave:    33.6M   (33.6MR+0W)
[a] B/s max:    2.09G   (2.09GR+0W)
[a] Spectrum:   23%1    20%2    55%3
[a] Kthreads:   0       (0 waiting/0 running/1 max)
[a] Cthreads:   0       (-)     (-)
[a] Cpids:      0       (11561) (11562)
Device b:       Open
[b] State:      verify, rw, merge requests, enabled, validated, show_errs,
last error 0, lives 6, bp 0
[b] Queued:     +4R/0W curr (check 4R/0W) +7R/1W max
[b] Buffersize: 262144  (sectors=512, blocks=64)
[b] Blocksize:  4096    (log=12)
[b] Size:       8008400KB
[b] Blocks:     2002100
[b] Sockets:    2       (*)     (+)
[b] Requested:  520.04K (257K)  (262K)  520.0KR/0W      max 16
[b] Despatched: 520.03K (257K)  (262K)  520.0KR/0W      md5 0W (0 eq, 0 ne,
0 dn)
[b] Errored:    0       (0)     (0)     0+0
[b] Pending:    6       (3)     (3)     6R/0W+0R/0W
[b] B/s now:    5.57M   (5.57MR+0W)
[b] B/s ave:    33.8M   (33.8MR+0W)
[b] B/s max:    676M    (676MR+0W)
[b] Spectrum:   23%1    20%2    56%3
[b] Kthreads:   0       (0 waiting/0 running/1 max)
[b] Cthreads:   0       (-)     (-)
[b] Cpids:      0       (11559) (11560)
Device c:       Open
[c] State:      verify, rw, merge requests, enabled, validated, show_errs,
last error 0, lives 6, bp 0
[c] Queued:     +0R/0W curr (check 0R/0W) +7R/1W max
[c] Buffersize: 262144  (sectors=512, blocks=64)
[c] Blocksize:  4096    (log=12)
[c] Size:       8008400KB
[c] Blocks:     2002100
[c] Sockets:    2       (+)     (*)
[c] Requested:  520.04K (260K)  (260K)  520.0KR/0W      max 16
[c] Despatched: 520.04K (260K)  (260K)  520.0KR/0W      md5 0W (0 eq, 0 ne,
0 dn)
[c] Errored:    0       (0)     (0)     0+0
[c] Pending:    0       (0)     (0)     0R/0W+0R/0W
[c] B/s now:    5.65M   (5.65MR+0W)
[c] B/s ave:    33.7M   (33.7MR+0W)
[c] B/s max:    1.04G   (1.04GR+0W)
[c] Spectrum:   22%1    20%2    56%3
[c] Kthreads:   0       (0 waiting/0 running/1 max)
[c] Cthreads:   2       (+)     (+)
[c] Cpids:      2       (11563) (11564)
Device d:       Open
[d] State:      verify, rw, merge requests, enabled, validated, show_errs,
last error 0, lives 7, bp 0
[d] Queued:     +1R/0W curr (check 1R/0W) +7R/1W max
[d] Buffersize: 262144  (sectors=512, blocks=64)
[d] Blocksize:  4096    (log=12)
[d] Size:       8008400KB
[d] Blocks:     2002100
[d] Sockets:    2       (*)     (+)
[d] Requested:  1.7293M (886K)  (883K)  1.729MR/0W      max 32
[d] Despatched: 1.7293M (886K)  (883K)  1.729MR/0W      md5 0W (0 eq, 0 ne,
0 dn)
[d] Errored:    0       (0)     (0)     0+0
[d] Pending:    6       (3)     (3)     6R/0W+0R/0W
[d] B/s now:    5.65M   (5.65MR+0W)
[d] B/s ave:    114M    (114MR+0W)
[d] B/s max:    2.95G   (2.95GR+0W)
[d] Spectrum:   19%1    17%2    48%3    14%32
[d] Kthreads:   0       (0 waiting/0 running/1 max)
[d] Cthreads:   0       (-)     (-)
[d] Cpids:      0       (11565) (11566)
Device e-p:     Closed


all 4 servers with option -n, client with merge_requests=512:

- server1 meminfo:
MemTotal:      1036524 kB
MemFree:        901632 kB
Buffers:         52084 kB
Cached:          38948 kB
SwapCached:          0 kB
Active:          62244 kB
Inactive:        34108 kB
HighTotal:      131008 kB
HighFree:        83552 kB
LowTotal:       905516 kB
LowFree:        818080 kB
SwapTotal:     1999992 kB
SwapFree:      1999992 kB
Dirty:              28 kB
Writeback:           0 kB
Mapped:          10380 kB
Slab:            31536 kB
Committed_AS:    19856 kB
PageTables:        324 kB
VmallocTotal:   114680 kB
VmallocUsed:      3516 kB
VmallocChunk:   110692 kB

- server2 meminfo:
MemTotal:      1036524 kB
MemFree:        921280 kB
Buffers:         51792 kB
Cached:          19560 kB
SwapCached:          0 kB
Active:          58240 kB
Inactive:        18452 kB
HighTotal:      131008 kB
HighFree:       102872 kB
LowTotal:       905516 kB
LowFree:        818408 kB
SwapTotal:     1999992 kB
SwapFree:      1999992 kB
Dirty:              32 kB
Writeback:           0 kB
Mapped:          10400 kB
Slab:            31500 kB
Committed_AS:    19892 kB
PageTables:        324 kB
VmallocTotal:   114680 kB
VmallocUsed:      3560 kB
VmallocChunk:   110668 kB

- server3 meminfo:
MemTotal:      1036524 kB
MemFree:        922364 kB
Buffers:         50440 kB
Cached:          20012 kB
SwapCached:          0 kB
Active:          57680 kB
Inactive:        18112 kB
HighTotal:      131008 kB
HighFree:       102452 kB
LowTotal:       905516 kB
LowFree:        819912 kB
SwapTotal:     1999992 kB
SwapFree:      1999992 kB
Dirty:              32 kB
Writeback:           0 kB
Mapped:          10400 kB
Slab:            31316 kB
Committed_AS:    19892 kB
PageTables:        324 kB
VmallocTotal:   114680 kB
VmallocUsed:      3516 kB
VmallocChunk:   110692 kB

- server4 meminfo:
MemTotal:      1036524 kB
MemFree:        927508 kB
Buffers:         49760 kB
Cached:          15544 kB
SwapCached:          0 kB
Active:          53920 kB
Inactive:        16724 kB
HighTotal:      131008 kB
HighFree:       106848 kB
LowTotal:       905516 kB
LowFree:        820660 kB
SwapTotal:     1999992 kB
SwapFree:      1999992 kB
Dirty:              32 kB
Writeback:           0 kB
Mapped:          10400 kB
Slab:            31260 kB
Committed_AS:    19892 kB
PageTables:        324 kB
VmallocTotal:   114680 kB
VmallocUsed:      3560 kB
VmallocChunk:   110668 kB


Here is the output of /proc/nbdinfo on client side:
Device a:       Open
[a] State:      verify, rw, merge requests, enabled, validated, show_errs,
last error 0, lives 0, bp 0
[a] Queued:     +0R/0W curr (check 0R/0W) +14R/1W max
[a] Buffersize: 262144  (sectors=512, blocks=64)
[a] Blocksize:  4096    (log=12)
[a] Size:       8008400KB
[a] Blocks:     2002100
[a] Sockets:    2       (*)     (+)
[a] Requested:  52.198K (26.0K) (26.1K) 52.19KR/0W      max 3
[a] Despatched: 52.198K (26.0K) (26.1K) 52.19KR/0W      md5 0W (0 eq, 0 ne,
0 dn)
[a] Errored:    68      (0)     (0)     0+68
[a] Pending:    0       (0)     (0)     0R/0W+0R/0W
[a] B/s now:    10.5M   (10.5MR+0W)
[a] B/s ave:    2.89M   (2.89MR+0W)
[a] B/s max:    10.6M   (10.6MR+0W)
[a] Spectrum:   23%1    21%2    55%3
[a] Kthreads:   0       (0 waiting/0 running/1 max)
[a] Cthreads:   2       (+)     (+)
[a] Cpids:      2       (2442)  (2443)
Device b:       Open
[b] State:      verify, rw, merge requests, enabled, validated, show_errs,
last error 0, lives 0, bp 0
[b] Queued:     +0R/0W curr (check 0R/0W) +14R/1W max
[b] Buffersize: 262144  (sectors=512, blocks=64)
[b] Blocksize:  4096    (log=12)
[b] Size:       8008400KB
[b] Blocks:     2002100
[b] Sockets:    2       (+)     (*)
[b] Requested:  52.202K (25.9K) (26.2K) 52.20KR/0W      max 3
[b] Despatched: 52.198K (25.9K) (26.2K) 52.19KR/0W      md5 0W (0 eq, 0 ne,
0 dn)
[b] Errored:    68      (0)     (0)     0+68
[b] Pending:    4       (3)     (1)     4R/0W+0R/0W
[b] B/s now:    10.5M   (10.5MR+0W)
[b] B/s ave:    3.01M   (3.01MR+0W)
[b] B/s max:    10.6M   (10.6MR+0W)
[b] Spectrum:   25%1    20%2    54%3
[b] Kthreads:   1       (0 waiting/1 running/1 max)
[b] Cthreads:   0       (-)     (-)
[b] Cpids:      0       (2440)  (2441)
Device c:       Open
[c] State:      verify, rw, merge requests, enabled, validated, show_errs,
last error 0, lives 0, bp 0
[c] Queued:     +2R/0W curr (check 2R/0W) +14R/1W max
[c] Buffersize: 262144  (sectors=512, blocks=64)
[c] Blocksize:  4096    (log=12)
[c] Size:       8008400KB
[c] Blocks:     2002100
[c] Sockets:    2       (+)     (*)
[c] Requested:  52.205K (26.1K) (26.0K) 52.20KR/0W      max 3
[c] Despatched: 52.197K (26.1K) (26.0K) 52.19KR/0W      md5 0W (0 eq, 0 ne,
0 dn)
[c] Errored:    68      (0)     (0)     0+68
[c] Pending:    4       (3)     (1)     4R/0W+0R/0W
[c] B/s now:    10.5M   (10.5MR+0W)
[c] B/s ave:    3.11M   (3.11MR+0W)
[c] B/s max:    10.6M   (10.6MR+0W)
[c] Spectrum:   24%1    20%2    54%3
[c] Kthreads:   0       (0 waiting/0 running/1 max)
[c] Cthreads:   0       (-)     (-)
[c] Cpids:      0       (2446)  (2447)
Device d:       Open
[d] State:      verify, rw, merge requests, enabled, validated, show_errs,
last error 0, lives 0, bp 0
[d] Queued:     +0R/0W curr (check 0R/0W) +14R/1W max
[d] Buffersize: 262144  (sectors=512, blocks=64)
[d] Blocksize:  4096    (log=12)
[d] Size:       8008400KB
[d] Blocks:     2002100
[d] Sockets:    2       (+)     (*)
[d] Requested:  52.197K (26.1K) (26.0K) 52.19KR/0W      max 3
[d] Despatched: 52.197K (26.1K) (26.0K) 52.19KR/0W      md5 0W (0 eq, 0 ne,
0 dn)
[d] Errored:    68      (0)     (0)     0+68
[d] Pending:    0       (0)     (0)     0R/0W+0R/0W
[d] B/s now:    10.5M   (10.5MR+0W)
[d] B/s ave:    2.56M   (2.56MR+0W)
[d] B/s max:    10.5M   (10.5MR+0W)
[d] Spectrum:   23%1    20%2    55%3
[d] Kthreads:   0       (0 waiting/0 running/1 max)
[d] Cthreads:   2       (+)     (+)
[d] Cpids:      2       (2444)  (2445)
Device e-p:     Closed


One question about the test forcing the use of aligned_read().
Do I have to recompile the kernel to use aligned_read() even without the
option -n? Or is there another way to test this?

greetings,


Christian