Re: Is it possible to set the amount of CRC32C hashes per a data sector in integritysetup?

On 9/11/23 00:05, Arno Wagner wrote:
> On Sun, Sep 10, 2023 at 18:53:04 CEST, Michael Kjörling wrote:
>> On 10 Sep 2023 18:12 +0300, from [email protected] (Горбешко Богдан):
>>> AFAIU, the primary goal of dm-integrity is checking if the data are not
>>> corrupted and throwing read errors otherwise. Though it still supports
>>> checksums besides of hashes. Can they be used for recovering the data
>>> on-the-fly?
>>
>> _My understanding_ (which may be wrong) is that this is not what
>> dm-integrity is intended for; nor does it look like it supports it.
>   
> It is not. What this needs is things like RAID 1/5/6 (which can
> be done in files) and some specialized tools.
> 
> Incidentally, "checksums" do never support recovery from
> errors. What you need for that is error correcting codes.

Yes, this fully applies to dm-integrity (read-write integrity protected device).


That said, dm-verity (that is read-only target and uses Merkle tree
protection for the whole device) can be combined with forward
error correction codes (FEC, here Reed-Salomon) and can fix
a lot of corrupted data (and veritysetup fully supports FEC options).

(If used with smartphones, it apparently extends lifetime by
continuously repairing failing flash storage that internal correction
no longer handles - but I have never seen real data that it actually works
for real phones in the wild...
But I think that was motivation for Google to implement FEC for dm-verity.).


I had an idea how to combine FEC with dm-integrity, but it never
happened and probably never happen, as nobody will pay the time
for development. (And obviously it will be very slow.)

> Transparently, RAID is probably the easiest option in Linux
> and the one the works on sector-level. Given that modern drives
> (HDD and SSDs) already do extensive error-correction, you probably
> need to deal with whole-sector loss (or worse) to get any
> additional effect.

Reed-Solomon (and some new algorithms) are very good in repairing such
situations. (It is all about erasure coding.)

But I think many people just use RAID with dm-integrity now (despite it is slow).

Milan