Re: Disk I/O slowness

--Apple-Mail=_56672FB2-C589-4597-89B5-E92719B33E2F
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8


> On Jul 31, 2026, at 12:40=E2=80=AFAM, Tom Dial <[email protected]> =
wrote:
>=20
>=20
> On 7/30/26 14:49, Casey Deccio wrote:
>>> On Jul 30, 2026, at 1:04=E2=80=AFPM, Alexander V. Makartsev =
<[email protected]> wrote:
>>>=20
>>> On 7/30/26 18:17, Casey Deccio wrote:
>>>>> On Jul 29, 2026, at 5:28=E2=80=AFPM, Andy =
Smith<[email protected]> wrote:
>>>>>=20
>>>>> ...
>>>>> If it does occur when /home is mounted, then it is the disks in =
the RAID
>>>>> setup or the RAID setup itself. Does each individual disk perform
>>>>> anomalously?
>>>>>=20
>>>>> "fio" is a good tool,to carry out performance testing of storage
>>>>> devices. Have a look what the random 4k read and write I/O is like =
in
>>>>> terms of IOPS, and the sequential disk transfer rate in MB/s for =
reads
>>>>> or writes. You can try that on the array and then break the array =
and
>>>>> try it on each individual disk. Obviously this is destructive to =
the
>>>>> data.
>>>>>=20
>>>>> If the performance is as expected on a per-disk basis but terrible =
only
>>>>> with RAID, it may be time to post to the linux-raid mailing list =
as this
>>>>> is unlikely to be a Debian-specific issue. I would then also be =
checking
>>>>> if it happens in other RAID configurations like RAID-1.
>>>>>=20
>>>>> 150 IOPS of random reads (not coming from RAM) is reasonable for a
>>>>> single 7.2kRPM SATA disk. You could expect 2-3x this from a RAID-5 =
of
>>>>> three identical disks.
>>>>>=20
>>>>> Write IOPS is a bit more difficult to guess and will depend on =
whether
>>>>> you have disabled the write cache in the drives themselves. Due to =
the
>>>>> RAID-5 write penalty (read-modify-write) you may expect only =
50-70% of a
>>>>> single disk's IOPS for random writes.
>>>> Thanks for that very helpful information.  I've run fio on the =
still-assembled array, and here's what I get for read/write:
>>>>=20
>>>>   read: IOPS=3D237
>>>>   write: IOPS=3D79
>>>>=20
>>>> I'm not sure exactly what to make of that...
>>> I might add insult to the injury, but are you sure you need RAID-5 =
if it consists of just 3 drives?
>>> Usually around 30% per drive's capacity will be sacrificed to hold =
error correction data for the array,
>>> so maybe it will be much better to use 2 drives in RAID-1 (Mirror) =
configuration, if you need decent write performance.
>>> Third drive could be used as spare device for the array, or a =
separate storage without redundancy.
>> Great ideas.  I originally did RAID5 because I wasn't sure how many =
drives I wanted in the array, and I wondered if I might grow the array =
at some point.  But I'll consider RAID 1 if I settle on three drives.
>=20
> A bit of analysis suggests this is a low performance server, mainly if =
not entirely because of the disks. The server board surely is not the =
bottleneck. As an earlier post suggested, the sync times noted are quite =
as should be expected.
>=20
> The bottleneck almost certainly is the 6GBit/Sec consumer disks. The =
6GBits/Sec transfer speed is only attainable on the cables from the =
system board to the individual disks. It implies a maximum transfer rate =
around 600 MBytes/Sec. but in sustained transfers, the disks cannot =
write that fast. AI query suggests 100-180 MB/Sec (depending on what =
part of the disk is being accessed) for CMR disks and 10-30 MB/Sec(!) =
for SMR disks. Raid 5 with 3 disks will reduce those to around 50% -75%, =
around 8-24 MB/sec  so the expected average sustained write speed is =
unlikely to be more than 100MB/Sec. even with CMR disks. That is about, =
or slightly above, what is available from a Gigabit Ethernet transfer; =
for SMR disks, expect no more than around 25 MB/Sec. Large writes will =
take a while.
>=20
> The pattern of sync times reported is consistent with that, suggesting =
that rsync makes the data available for writing somewhat faster than it =
can be written to the disk surfaces. Note that this doesn't apply to =
reading; the RAID will read and deliver data roughly twice as fast as a =
single disk and continue to do so on sustained reads as long as the data =
read is consumed at least as fast. Once the bulk data is transferred, =
performance may be satisfactory even with the low-spec drives.
>=20
> The system with the Megaraid card would appear to perform better due =
to its internal cache: it will report write completion as soon as the =
data is in its cache (thus unblocking sync) and handle transfer to the =
disks internally. AI suggestion for the new server: set Write-Back =
rather than Write-Through for PCH Cache settings if that is not already =
done; that will allow the Intel driver to treat main memory as cache. It =
will not, however, make the data transfer any more quickly to the disks =
in the array.
>=20
> AI was used here, but not extensively.

Thanks for the assessment.  And also thanks to all the other respondents =
for their contributions.

I can't really change the hardware I have, but this is helpful for =
figuring out how to move forward, both with my current system and for =
future system design.

Casey=

--Apple-Mail=_56672FB2-C589-4597-89B5-E92719B33E2F
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

<html aria-label=3D"message body"><head><meta http-equiv=3D"content-type" =
content=3D"text/html; charset=3Dutf-8"></head><body =
style=3D"overflow-wrap: break-word; -webkit-nbsp-mode: space; =
line-break: after-white-space;"><br><div><blockquote type=3D"cite"><div>On=
 Jul 31, 2026, at 12:40=E2=80=AFAM, Tom Dial &lt;[email protected]&gt; =
wrote:</div><div><br style=3D"caret-color: rgb(0, 0, 0); font-family: =
Helvetica; font-size: 12px; font-style: normal; font-variant-caps: =
normal; font-weight: 400; letter-spacing: normal; orphans: 2; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration-line: none; text-decoration-thickness: auto; =
text-decoration-style: solid;"><br style=3D"caret-color: rgb(0, 0, 0); =
font-family: Helvetica; font-size: 12px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
orphans: 2; text-align: start; text-indent: 0px; text-transform: none; =
white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration-line: none; =
text-decoration-thickness: auto; text-decoration-style: solid;"><span =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
12px; font-style: normal; font-variant-caps: normal; font-weight: 400; =
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration: none; float: none; =
display: inline !important;">On 7/30/26 14:49, Casey Deccio =
wrote:</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: =
Helvetica; font-size: 12px; font-style: normal; font-variant-caps: =
normal; font-weight: 400; letter-spacing: normal; orphans: 2; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration-line: none; text-decoration-thickness: auto; =
text-decoration-style: solid;"><blockquote type=3D"cite" =
style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
orphans: 2; text-align: start; text-indent: 0px; text-transform: none; =
white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration-line: none; =
text-decoration-thickness: auto; text-decoration-style: =
solid;"><blockquote type=3D"cite">On Jul 30, 2026, at 1:04=E2=80=AFPM, =
Alexander V. Makartsev &lt;[email protected]&gt; wrote:<br><br>On =
7/30/26 18:17, Casey Deccio wrote:<br><blockquote =
type=3D"cite"><blockquote type=3D"cite">On Jul 29, 2026, at 5:28=E2=80=AFP=
M, Andy Smith&lt;[email protected]&gt; wrote:<br><br>...<br>If it does =
occur when /home is mounted, then it is the disks in the RAID<br>setup =
or the RAID setup itself. Does each individual disk =
perform<br>anomalously?<br><br>"fio" is a good tool,to carry out =
performance testing of storage<br>devices. Have a look what the random =
4k read and write I/O is like in<br>terms of IOPS, and the sequential =
disk transfer rate in MB/s for reads<br>or writes. You can try that on =
the array and then break the array and<br>try it on each individual =
disk. Obviously this is destructive to the<br>data.<br><br>If the =
performance is as expected on a per-disk basis but terrible only<br>with =
RAID, it may be time to post to the linux-raid mailing list as =
this<br>is unlikely to be a Debian-specific issue. I would then also be =
checking<br>if it happens in other RAID configurations like =
RAID-1.<br><br>150 IOPS of random reads (not coming from RAM) is =
reasonable for a<br>single 7.2kRPM SATA disk. You could expect 2-3x this =
from a RAID-5 of<br>three identical disks.<br><br>Write IOPS is a bit =
more difficult to guess and will depend on whether<br>you have disabled =
the write cache in the drives themselves. Due to the<br>RAID-5 write =
penalty (read-modify-write) you may expect only 50-70% of a<br>single =
disk's IOPS for random writes.<br></blockquote>Thanks for that very =
helpful information. &nbsp;I've run fio on the still-assembled array, =
and here's what I get for read/write:<br><br>&nbsp;&nbsp;read: =
IOPS=3D237<br>&nbsp;&nbsp;write: IOPS=3D79<br><br>I'm not sure exactly =
what to make of that...<br></blockquote>I might add insult to the =
injury, but are you sure you need RAID-5 if it consists of just 3 =
drives?<br>Usually around 30% per drive's capacity will be sacrificed to =
hold error correction data for the array,<br>so maybe it will be much =
better to use 2 drives in RAID-1 (Mirror) configuration, if you need =
decent write performance.<br>Third drive could be used as spare device =
for the array, or a separate storage without =
redundancy.<br></blockquote>Great ideas. &nbsp;I originally did RAID5 =
because I wasn't sure how many drives I wanted in the array, and I =
wondered if I might grow the array at some point. &nbsp;But I'll =
consider RAID 1 if I settle on three drives.<br></blockquote><br =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
12px; font-style: normal; font-variant-caps: normal; font-weight: 400; =
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration-line: none; =
text-decoration-thickness: auto; text-decoration-style: solid;"><span =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
12px; font-style: normal; font-variant-caps: normal; font-weight: 400; =
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration: none; float: none; =
display: inline !important;">A bit of analysis suggests this is a low =
performance server, mainly if not entirely because of the disks. The =
server board surely is not the bottleneck. As an earlier post suggested, =
the sync times noted are quite as should be expected.</span><br =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
12px; font-style: normal; font-variant-caps: normal; font-weight: 400; =
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration-line: none; =
text-decoration-thickness: auto; text-decoration-style: solid;"><br =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
12px; font-style: normal; font-variant-caps: normal; font-weight: 400; =
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration-line: none; =
text-decoration-thickness: auto; text-decoration-style: solid;"><span =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
12px; font-style: normal; font-variant-caps: normal; font-weight: 400; =
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration: none; float: none; =
display: inline !important;">The bottleneck almost certainly is the =
6GBit/Sec consumer disks. The 6GBits/Sec transfer speed is only =
attainable on the cables from the system board to the individual disks. =
It implies a maximum transfer rate around 600 MBytes/Sec. but in =
sustained transfers, the disks cannot write that fast. AI query suggests =
100-180 MB/Sec (depending on what part of the disk is being accessed) =
for CMR disks and 10-30 MB/Sec(!) for SMR disks. Raid 5 with 3 disks =
will reduce those to around 50% -75%, around 8-24 MB/sec &nbsp;so the =
expected average sustained write speed is unlikely to be more than =
100MB/Sec. even with CMR disks. That is about, or slightly above, what =
is available from a Gigabit Ethernet transfer; for SMR disks, expect no =
more than around 25 MB/Sec. Large writes will take a while.</span><br =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
12px; font-style: normal; font-variant-caps: normal; font-weight: 400; =
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration-line: none; =
text-decoration-thickness: auto; text-decoration-style: solid;"><br =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
12px; font-style: normal; font-variant-caps: normal; font-weight: 400; =
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration-line: none; =
text-decoration-thickness: auto; text-decoration-style: solid;"><span =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
12px; font-style: normal; font-variant-caps: normal; font-weight: 400; =
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration: none; float: none; =
display: inline !important;">The pattern of sync times reported is =
consistent with that, suggesting that rsync makes the data available for =
writing somewhat faster than it can be written to the disk surfaces. =
Note that this doesn't apply to reading; the RAID will read and deliver =
data roughly twice as fast as a single disk and continue to do so on =
sustained reads as long as the data read is consumed at least as fast. =
Once the bulk data is transferred, performance may be satisfactory even =
with the low-spec drives.</span><br style=3D"caret-color: rgb(0, 0, 0); =
font-family: Helvetica; font-size: 12px; font-style: normal; =
font-variant-caps: normal; font-weight: 400; letter-spacing: normal; =
orphans: 2; text-align: start; text-indent: 0px; text-transform: none; =
white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration-line: none; =
text-decoration-thickness: auto; text-decoration-style: solid;"><br =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
12px; font-style: normal; font-variant-caps: normal; font-weight: 400; =
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration-line: none; =
text-decoration-thickness: auto; text-decoration-style: solid;"><span =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
12px; font-style: normal; font-variant-caps: normal; font-weight: 400; =
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration: none; float: none; =
display: inline !important;">The system with the Megaraid card would =
appear to perform better due to its internal cache: it will report write =
completion as soon as the data is in its cache (thus unblocking sync) =
and handle transfer to the disks internally. AI suggestion for the new =
server: set Write-Back rather than Write-Through for PCH Cache settings =
if that is not already done; that will allow the Intel driver to treat =
main memory as cache. It will not, however, make the data transfer any =
more quickly to the disks in the array.</span><br style=3D"caret-color: =
rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: =
normal; font-variant-caps: normal; font-weight: 400; letter-spacing: =
normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: =
none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration-line: none; =
text-decoration-thickness: auto; text-decoration-style: solid;"><br =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
12px; font-style: normal; font-variant-caps: normal; font-weight: 400; =
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration-line: none; =
text-decoration-thickness: auto; text-decoration-style: solid;"><span =
style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: =
12px; font-style: normal; font-variant-caps: normal; font-weight: 400; =
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; =
text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; text-decoration: none; float: none; =
display: inline !important;">AI was used here, but not =
extensively.</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: =
Helvetica; font-size: 12px; font-style: normal; font-variant-caps: =
normal; font-weight: 400; letter-spacing: normal; orphans: 2; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration-line: none; text-decoration-thickness: auto; =
text-decoration-style: solid;"></div></blockquote><div><br></div>Thanks =
for the assessment. &nbsp;And also thanks to all the other respondents =
for their contributions.</div><div><br></div><div>I can't really change =
the hardware I have, but this is helpful for figuring out how to move =
forward, both with my current system and for future system =
design.</div><div><br></div><div>Casey</div></body></html>=

--Apple-Mail=_56672FB2-C589-4597-89B5-E92719B33E2F--