Disk I/O slowness
Casey Deccio <[email protected]> Wed, 29 Jul 2026 16:38:55 -0600
| Newsgroups | gmane.linux.debian.user |
|---|---|
| Message-ID | <[email protected]> |
--Apple-Mail=_2A7EFF8E-4443-4458-BBEF-FA0E92FE6138
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=us-ascii
Hi all,
I'm experiencing some unexpected disk I/O latency on a newly built =
server. Here are some of the details:
- Supermicro motherboard with Intel PCH RAID [1]
- Debian 13.6 installed
- / is mounted on /dev/nvme0n1p2, which is an NVMe SSD, formatted with =
ext4
- /home is mounted on /dev/md125, which is a RAID 5 consisting of three =
disks, formatted with ext4
$ sudo mdadm -D /dev/md125
/dev/md125:
Container : /dev/md/imsm1, member 0
Raid Level : raid5
Array Size : 9279152128 (8.64 TiB 9.50 TB)
Used Dev Size : 4639576064 (4.32 TiB 4.75 TB)
Raid Devices : 3
Total Devices : 3
State : active=20
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-asymmetric
Chunk Size : 64K
Consistency Policy : resync
UUID : 2fbd736b:9b750e2a:e723b80c:df2f2c33
Number Major Minor RaidDevice State
2 8 0 0 active sync /dev/sda
1 8 16 1 active sync /dev/sdb
0 8 32 2 active sync /dev/sdc
I'm getting some weird latency issues with fsync on the RAID disk (/home =
on /dev/md125):
(Caveat: I'm doing a file transfer that is writing to /home, so I might =
expect some latency, but nothing like this.)
$ for i in {1..5}; do time sync; done 2>&1 | grep real
real 0m21.647s
real 0m11.770s
real 0m5.234s
real 0m3.023s
real 0m0.462s
Even when I turn off the file transfer, I still get something like this:
$ for i in {1..10}; do time sync; done 2>&1 | grep real
real 0m0.147s
real 0m0.077s
real 0m0.003s
real 0m0.004s
real 0m0.003s
I've tried turning on and off smart monitoring with smartctl to see if =
that would help, and it didn't seem to help.
I'm not really sure what else I might try, and I'm reaching out to the =
community to see if anyone has any suggestions for identifying and =
fixing the problem.
Thanks,
Casey
[1] https://www.supermicro.com/manuals/other/Intel_PCH_RAID_Config.pdf
--Apple-Mail=_2A7EFF8E-4443-4458-BBEF-FA0E92FE6138
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=us-ascii
<html aria-label=3D"message body"><head><meta http-equiv=3D"content-type" =
content=3D"text/html; charset=3Dus-ascii"></head><body =
style=3D"overflow-wrap: break-word; -webkit-nbsp-mode: space; =
line-break: after-white-space;">Hi all,<div><br></div><div>I'm =
experiencing some unexpected disk I/O latency on a newly built server. =
Here are some of the details:</div><div><br></div><div>- =
Supermicro motherboard with Intel PCH RAID [1]</div><div>- Debian 13.6 =
installed</div><div>- / is mounted on /dev/nvme0n1p2, which is an NVMe =
SSD, formatted with ext4</div><div>- /home is mounted on /dev/md125, =
which is a RAID 5 consisting of three disks, formatted with =
ext4</div><div><br></div><div><div>$ sudo mdadm -D =
/dev/md125</div><div>/dev/md125:</div><div> =
Container : /dev/md/imsm1, member 0</div><div> =
Raid Level : raid5</div><div> Array =
Size : 9279152128 (8.64 TiB 9.50 TB)</div><div> Used =
Dev Size : 4639576064 (4.32 TiB 4.75 TB)</div><div> =
Raid Devices : 3</div><div> Total Devices : =
3</div><div><br></div><div> =
State : active </div><div> Active Devices : =
3</div><div> Working Devices : 3</div><div> =
Failed Devices : 0</div><div> Spare Devices : =
0</div><div><br></div><div> =
Layout : left-asymmetric</div><div> Chunk =
Size : 64K</div><div><br></div><div>Consistency Policy : =
resync</div><div><br></div><div><br></div><div> =
UUID : =
2fbd736b:9b750e2a:e723b80c:df2f2c33</div><div> Number =
Major Minor RaidDevice State</div><div> =
2 8 =
0 0 active sync =
/dev/sda</div><div> 1 =
8 16 1 =
active sync /dev/sdb</div><div> =
0 8 32 =
2 active sync =
/dev/sdc</div></div><div><br></div><div><br></div><div>I'm getting some =
weird latency issues with fsync on the RAID disk (/home on =
/dev/md125):</div><div>(Caveat: I'm doing a file transfer that is =
writing to /home, so I might expect some latency, but nothing like =
this.)</div><div><br></div><div>$ for i in {1..5}; do time sync; done =
2>&1 | grep real</div><div>real<span class=3D"Apple-tab-span" =
style=3D"white-space:pre"> </span>0m21.647s</div><div>real<span =
class=3D"Apple-tab-span" style=3D"white-space:pre"> =
</span>0m11.770s</div><div>real<span class=3D"Apple-tab-span" =
style=3D"white-space:pre"> </span>0m5.234s</div><div>real<span =
class=3D"Apple-tab-span" style=3D"white-space:pre"> =
</span>0m3.023s</div><div>real<span class=3D"Apple-tab-span" =
style=3D"white-space:pre"> =
</span>0m0.462s</div><div><br></div><div>Even when I turn off the file =
transfer, I still get something like =
this:</div><div><br></div><div><div>$ for i in {1..10}; do time sync; =
done 2>&1 | grep real</div><div>real<span class=3D"Apple-tab-span" =
style=3D"white-space:pre"> </span>0m0.147s</div><div>real<span =
class=3D"Apple-tab-span" style=3D"white-space:pre"> =
</span>0m0.077s</div><div>real<span class=3D"Apple-tab-span" =
style=3D"white-space:pre"> </span>0m0.003s</div><div>real<span =
class=3D"Apple-tab-span" style=3D"white-space:pre"> =
</span>0m0.004s</div><div>real<span class=3D"Apple-tab-span" =
style=3D"white-space:pre"> =
</span>0m0.003s</div></div><div><br></div><div>I've tried turning on and =
off smart monitoring with smartctl to see if that would help, and it =
didn't seem to help.</div><div><br></div><div>I'm not really sure what =
else I might try, and I'm reaching out to the community to see if anyone =
has any suggestions for identifying and fixing the =
problem.</div><div><br></div><div>Thanks,</div><div>Casey</div><div><br></=
div><div>[1] <a =
href=3D"https://www.supermicro.com/manuals/other/Intel_PCH_RAID_Config.pdf=
">https://www.supermicro.com/manuals/other/Intel_PCH_RAID_Config.pdf</a><b=
r></div></body></html>=
--Apple-Mail=_2A7EFF8E-4443-4458-BBEF-FA0E92FE6138--