Re: [clug] Is a weekly RAID scrub too much?

On 26/02/17 23:11, Eyal Lebedinsky wrote:
> On 26/02/17 22:16, Paul Wayper wrote:
>> On 26/02/17 17:32, Eyal Lebedinsky wrote:
>>> On 26/02/17 17:15, Paul Wayper wrote:
>>>> On 24/02/17 22:35, Eyal Lebedinsky wrote:
>> [...]
>>>>> Is the scrub doing more harm than good by shortening the service 
>>>>> life of the
>>>>> disk?
>>>>
>>>> So, what do you mean by "weekly RAID scrub"?
>>>>
>>>> I've never heard of anyone doing this on a RAID array.
>>>>
>>>> I've used 'scrub' on a disk or files to overwrite them with random 
>>>> data to
>>>> avoid anyone using a scanning electron microscope on them to view 
>>>> your data
>>>> again.  With disk geometries, cylinder tolerances, and encoding 
>>>> mechanisms
>>>> these days, even overwriting with zeros will render the data 
>>>> inaccessible to
>>>> almost all attackers.
>>>
>>> I use 'shred' for this.
>>>
>>>> But I've never heard of 'RAID scrubbing'.  What are you doing, Eyal?
>>>
>>> RAID scrub refers to, in linux software RAID, a 'check' request. It 
>>> means that
>>> the full RAID is read and verified. For RAID6 it means P and Q are 
>>> verified.
>>
>> I looked up the ever-helpful Arch Linux pages on RAID and LVM:
>>
>> https://wiki.archlinux.org/index.php/Software_RAID_and_LVM#Scrubbing
>>
>> That took me to Wikipedia's page on "Data Scrubbing":
>>
>> https://en.wikipedia.org/wiki/Data_scrubbing
>>
>> Which contained lots of hyperbolic, panic-inducing language like:
>>
>> """Data integrity is a high-priority concern in writing, reading, 
>> storage,
>> transmission, or processing of the computer data in computer 
>> operating systems
>> and in computer storage and data transmission systems. However, only 
>> a few of
>> the currently existing and used file systems provide sufficient 
>> protection
>> against data corruption."""
>>
>> To me this is totally overblown.  In all the many years I've been 
>> looking at
>> files, I don't think I've ever seen a file corrupted on a healthy 
>> disk.  I've
>
> I had the scrub discover problems a few times. I do not know the 
> reason for those
> failures, there was no disk error logged and I was not aware of any 
> event that
> could cause it.
>
> However, these problems pointed to files that were read happily but 
> the data
> delivered is likely to be bad.
>
> [aside]
> We do not have a "safe read" option for raid6, which could check data 
> as it is
> read and correct it if necessary (and possible). Furthermore, while 
> raid6 can
> do a 'repair', which is like a 'check' but on detecting an error it 
> recalculates
> the syndromes, this repair is not so good. raid6 has enough data to 
> detect
> *which* sector is bad and rewrite it rather than assume the data is 
> good and
> the syndromes are bad. Note: if a disk returns an actual i/o error 
> then the
> kernel does repair the failed sectors on the fly.
>
> In short, when a 'check' error is detected I identify the affected 
> files and
> deal with their content, restoring from backup if necessary.
> [/aside]
>
>> seen one IBM XT with dodgy memory that started corrupting everything 
>> that was
>> written to disk - and funnily enough running full disk scans made the 
>> problem
>> worse.  In the case of RAID 5 and 6 you've already got full 
>> redundancy should
>> one piece go missing - you can literally lose an entire disk and keep 
>> going in
>> degraded mode, so why do you care if a block goes bad?
>>
>> My favourite bit was:
>>
>> """Due to the high integration density of contemporary computer 
>> memory chips,
>> the individual memory cell structures became small enough to be 
>> vulnerable to
>> cosmic rays and/or alpha particle emission. The errors caused by these
>> phenomena are called soft errors. This can be a problem for DRAM- and
>> SRAM-based memories."""
>>
>> So the solution to the possibility of one of the sectors going 
>> silently bad -
>> despite all the systems in the drive itself to detect and correct 
>> these errors
>> - is to read it into memory which can also be faulty?  Tell me again 
>> how this
>> is going to help? :-)
>>
>> Joking aside, I think maybe a scrub once a year might be a 
>> possibility if
>> you're dealing with hardware you don't entirely trust.  But I'd keep 
>> it to a
>> minimum because, as you observe, drives do wear out with overuse, and 
>> full
>> disk scans will do that.
>
> I doubt a yearly scrub will ever come out clean :-(
> But I probably need to reduce the frequency to monthly.
>
> Question: How often other people do a scrub?
>
>> It's actually one of the common failure modes in RAID arrays - you have
>> several near-failing drives but they're fine with the normal 
>> workload.  One
>> drive fails and you replace it, but now the RAID rebuild has to scan 
>> every
>> single disk, so all the other drives near fail point are pushed over 
>> the limit.
>
> True, but while seeing a second disk fail during a rebuild is scary, 
> it is not
> fatal. In my experience disks fails slowly, a few bad sectors at a 
> time, and I
> can deal with a few bad sectors while replacing a failing disk.
>
> [aside]
> I did not yet have a catastrophic failure in the 12 years of running 
> software
> raid, but the last (current) set of disks is the worst. These are WD 
> blacks and
> out of 7 disks six were replaced (some more than once) and the seventh 
> had
> "events" suggesting it is next. This is after 3.5 years in the planned 
> 5 years
> life of this array.
>
> BTW, these disks do not list a workload limit, and I assume they are 
> designed
> for continuous use with a high load.
I have not used Blacks, but you might not have them in suitable 
application.  WD list Blacks in the personal storage category and not in 
Business storage.  The spec does put a limit of 300k load/unload cycles.

I recognise that this is an aside, but perhaps Red, that were available 
3.5 yr ago or the newer Gold, would be a more appropriate choice for the 
impending replacements.

> [/aside]
>
>> But by "common" here I mean "this almost never happens".  I've heard
>> anecdotes, and I'm hoping some people on the list can share hilarious 
>> stories
>> about drive and array failures :-)
>>
>> So, anyone running BTRFS?  Can you grep your logs for "csum failed 
>> ino" and
>> tell us all how often you see that error?  BTRFS stores checksums on 
>> metadata
>> and data (XFS stores checksums only on metadata), and verifies the 
>> checksum on
>> read and re-reads data if it's not correct.  If we could get some 
>> kind of
>> statistical analysis of the number of reads and writes performed vs 
>> the number
>> of actual checksum failures, we'd get an idea of how likely they are 
>> in the
>> real world.
>>

Perhaps not quickly.  I do run BTRFS, although on a systemd system and 
might need a bit of reading to figure out what the equivalent grep 
phrase would be.
>> Have fun,
>>
>> Paul
>
> cheers
>
Rod

-- 
linux mailing list
linux-w/[email protected]
https://lists.samba.org/mailman/listinfo/linux