Re: Disk I/O slowness

On 7/30/26 14:49, Casey Deccio wrote:
> 
>> On Jul 30, 2026, at 1:04 PM, Alexander V. Makartsev <[email protected]> wrote:
>>
>> On 7/30/26 18:17, Casey Deccio wrote:
>>>> On Jul 29, 2026, at 5:28 PM, Andy Smith<[email protected]> wrote:
>>>>
>>>> ...
>>>> If it does occur when /home is mounted, then it is the disks in the RAID
>>>> setup or the RAID setup itself. Does each individual disk perform
>>>> anomalously?
>>>>
>>>> "fio" is a good tool,to carry out performance testing of storage
>>>> devices. Have a look what the random 4k read and write I/O is like in
>>>> terms of IOPS, and the sequential disk transfer rate in MB/s for reads
>>>> or writes. You can try that on the array and then break the array and
>>>> try it on each individual disk. Obviously this is destructive to the
>>>> data.
>>>>
>>>> If the performance is as expected on a per-disk basis but terrible only
>>>> with RAID, it may be time to post to the linux-raid mailing list as this
>>>> is unlikely to be a Debian-specific issue. I would then also be checking
>>>> if it happens in other RAID configurations like RAID-1.
>>>>
>>>> 150 IOPS of random reads (not coming from RAM) is reasonable for a
>>>> single 7.2kRPM SATA disk. You could expect 2-3x this from a RAID-5 of
>>>> three identical disks.
>>>>
>>>> Write IOPS is a bit more difficult to guess and will depend on whether
>>>> you have disabled the write cache in the drives themselves. Due to the
>>>> RAID-5 write penalty (read-modify-write) you may expect only 50-70% of a
>>>> single disk's IOPS for random writes.
>>> Thanks for that very helpful information.  I've run fio on the still-assembled array, and here's what I get for read/write:
>>>
>>>    read: IOPS=237
>>>    write: IOPS=79
>>>
>>> I'm not sure exactly what to make of that...
>> I might add insult to the injury, but are you sure you need RAID-5 if it consists of just 3 drives?
>> Usually around 30% per drive's capacity will be sacrificed to hold error correction data for the array,
>> so maybe it will be much better to use 2 drives in RAID-1 (Mirror) configuration, if you need decent write performance.
>> Third drive could be used as spare device for the array, or a separate storage without redundancy.
> 
> Great ideas.  I originally did RAID5 because I wasn't sure how many drives I wanted in the array, and I wondered if I might grow the array at some point.  But I'll consider RAID 1 if I settle on three drives.
> 

A bit of analysis suggests this is a low performance server, mainly if not entirely because of the disks. The server board surely is not the bottleneck. As an earlier post suggested, the sync times noted are quite as should be expected.

The bottleneck almost certainly is the 6GBit/Sec consumer disks. The 6GBits/Sec transfer speed is only attainable on the cables from the system board to the individual disks. It implies a maximum transfer rate around 600 MBytes/Sec. but in sustained transfers, the disks cannot write that fast. AI query suggests 100-180 MB/Sec (depending on what part of the disk is being accessed) for CMR disks and 10-30 MB/Sec(!) for SMR disks. Raid 5 with 3 disks will reduce those to around 50% -75%, around 8-24 MB/sec  so the expected average sustained write speed is unlikely to be more than 100MB/Sec. even with CMR disks. That is about, or slightly above, what is available from a Gigabit Ethernet transfer; for SMR disks, expect no more than around 25 MB/Sec. Large writes will take a while.

The pattern of sync times reported is consistent with that, suggesting that rsync makes the data available for writing somewhat faster than it can be written to the disk surfaces. Note that this doesn't apply to reading; the RAID will read and deliver data roughly twice as fast as a single disk and continue to do so on sustained reads as long as the data read is consumed at least as fast. Once the bulk data is transferred, performance may be satisfactory even with the low-spec drives.

The system with the Megaraid card would appear to perform better due to its internal cache: it will report write completion as soon as the data is in its cache (thus unblocking sync) and handle transfer to the disks internally. AI suggestion for the new server: set Write-Back rather than Write-Through for PCH Cache settings if that is not already done; that will allow the Intel driver to treat main memory as cache. It will not, however, make the data transfer any more quickly to the disks in the array.

AI was used here, but not extensively.

Regards,
Tom Dial
> Thanks,
> Casey