Re: Subject: RFC: Read repair for md RAID1 after mirror read failures

On 7/17/26 8:11 PM, Roger Heflin wrote:
> The drives are said to do refreshes when it finds a weak sector when
> doing a long media test.
> 
> The manufactures do not seem to document any disk internals, they
> never have.    All manufactures claim their magic firmware/processes
> give them a competitive advantage and so don't disclose what goes on.
>   Pretty much all of the hardware is handled this way, with poor
> external documentation.
> 
> Smart seems to only disclose the common settings/errors, and don't
> seem to allow anything except setting changes, and to run defined
> tests (short, long and similar).
> 
> I have been told by a WD engineer that their internal tools have the
> ability to read out a number of other parameters that aren't exposed
> to SMART, and may also be able to do other things.  Those tools are
> not externally available and the when I had access to one of those
> tools (for a few hours so I could get it into a high security site
> with the WD engineer) I had to agree to not keep it.
> 
> I watched one of the tools be used on some disks by a WD engineer that
> went onsite (but I don't remember if I saw any non-SMART like data on
> the screen--it was 20 years ago) to check for errors and determine if
> WD would RMA/replace (without media return) around 1000 disks.
> 
> I have been reading smart data for as long as smart has been around,
> and outside of maybe at most 10 parameters, most of the rest of the
> parms are rather useless because the manufacturer does not supply a
> decoder ring to make any sense of what said parameter is really
> measuring.
> 
> And whatever the manufactures document would be what was documented
> before the disk firmware was completely finished, and may or may not
> have really worked in the given firmware.   I have ran into disks that
> would refuse to relocate most bad sectors (even though older disk by
> the same company did this just fine), and a number of other weird disk
> actions indicating that the firmware does not always even implement
> the features that they are supposed to correctly in all disks.
> 
> Basically if the smart tools team actually knows anything about the
> disk internals, they are almost certainly under an NDA and cannot tell
> you (likely this is not the case), and if they aren't under an NDA
> they were not told anything.
> 
> And contrary to what other experts claim (that disks regularly will
> return bad data--ie 1 in 1e15 sectors), I have never seen the disk
> itself return bad data(I have seen raid controllers deliver corrupted
> data many times from both hardware failures and defective raid
> controller firmware), if the sector is read successfully it is the
> right data simply because the disks ECC correction uses so many bits
> that the chance of a corrupted sector being corrupted and still
> passing ECC is so low as to not be worried about.

Roger,

Thanks for sharing this. I like your observation:

"...if the sector is read successfully it is the right data..."

It made me look at the problem from an information-theory perspective.

To me there are actually two cascaded communication channels.

The first is the time channel: from the moment the data is written to 
the magnetic medium until the drive firmware has reconstructed the 
sector using its ECC. If the ECC decoder succeeds, then, as you point 
out, the probability that the returned sector is still incorrect is 
extraordinarily small. In other words, the information has survived 
storage in time.

Only after that does the second channel begin: transport of that sector 
through the SATA/SAS link, controller, DMA, memory, RAID layer and 
finally to the filesystem. This is the space channel. Corruption here is 
a different class of error. As you mention, RAID controllers or other 
hardware can introduce corruption after the disk has already 
reconstructed the correct sector.

Viewed this way, the disk ECC and a filesystem checksum (such as Btrfs 
or ZFS) are not competing mechanisms—they protect two different 
communication channels.

Disk ECC protects the temporal storage channel.
Btrfs checksums provide end-to-end protection over the transport (space) 
channel.

During my recent investigation into a Btrfs device removal with hundreds 
of ATA UNC errors, this distinction became quite useful. The Linux 
storage stack (MD RAID1 in particular) managed to recover the sector 
from the disk, and once the drive delivered it successfully, the Btrfs 
checksum verified that the data had also survived the remainder of the 
path through the system.

I thought you might appreciate this way of looking at it.