Re: SMART statistics for 40, 000 disk drives, from Backblaze

> > https://www.backblaze.com/blog/hard-drive-smart-stats/ 

The above blog post suggests that Backblaze removes an entire drive
from service whenever its "Reported Uncorrectable Errors" goes nonzero.

This seems like overkill to me.  I have had many drives(*) produce
uncorrectable errors in a single sector or small number of nearby
sectors.  When I overwrote those sectors, the drive generally did not
go on to fail more and more; the drive remapped them and went back to
100% functioning.  Only a small percentage of drives go on to rapid
failure into a totally unusable state.

(*) I don't have a farm of tens of thousands of drives, I just run
dozens of drives at a time.  Still, I've been doing this at companies
and at home since the 1970s.

I believe the Internet Archive (with about 20 petabytes of storage)
also treats uncorrectable sector errors by initially overwriting them
and continuing to use the drive.

I think it'd be useful to automate the recovery of initial drive
failures in a replicated environment.  For example, if you have RAID,
online backups, or dual copies of items, then when a sector or group
of sectors fails, they can be overwritten with the correct data, and
thus remapped by the drive.  This *should* be an automatic process,
particularly at scale when you have many thousands of drives, but at
the moment it isn't.

It isn't automatic for many reasons:

  *  Natural reluctance to have automated software write to a drive
     that might be failing.  Humans can look at more context and be
     more cautious to avoid compound failure.  Even reading a failing drive
     can cause more problems; see "GNU ddrescue" for a tool designed to
     copy as much data off such a drive as possible before failure.
  *  Sometimes sector errors are caused by environmental factors like
     bad power or too much heat; writing to such drives is probably
     a bad idea.
  *  Error reporting is via kernel dmesg, requiring human "screen scraping",
     rather than via an API.
  *  Reports are in inconsistent units (sectors from sector 0, or
     sectors from first sector of filesystem; and how big is a sector
     anyway, some use 4096 bytes but fake a 512 byte interface).
  *  Reports from S.M.A.R.T. on different drives may be inconsistent with
     kernel reports, and often only report the first erroneous sector found.
  *  Exactly where any redundant copies of the data are stored is
     widely variable and may require local customization.  
  *  Recovery requires coordination at the kernel, daemon, and application
     level, and may require human intervention.

Does Linux software RAID, or ZFS, already do this "write over a bad
sector" recovery?  I know there are really kludgy, slow ways to do it,
e.g. when a drive has one bad sector, an operator removes it from the
RAID pool and adds it back, so the whole thing will get rewritten
("resilvered").  But does any existing software do the silver-bullet
style rewrite that just rewrites the failing sector or sectors, in
seconds, and eliminates the overhead, downtime, risk of compound
failure, etc?

I was thinking of defining a publish/subscribe API for this kind of
disk sector failure information.  Then smartd would become a publisher
for this information, reporting a failed sector to the API whenever a
short or long scan reports a failing sector.  Smartd could also become
a subscriber and schedule a whole disk scan, or a selective self test
of that area of the drive, at some time soon after the API reports
that error recovery has completed for a particular sector.

A recovery daemon would subscribe, to listen for sector failure
reports, coordinate with software RAID to find replicated copies of
the data, and do a relatively safe read-fail-write-read sequence on
the drive and sector where the failure is reported.

There are some security issues that would need addressing around an
API that provokes a daemon to do writes to arbitrary disk sectors,
too.  :-)  That's part of why the daemon would read any such sector,
and only write to it if the read fails.

Eventually, kernel error reports would also be published into this
API, allowing immediate response rather than only after a nightly
or weekly S.M.A.R.T. scan.

Even more eventually, this would be integrated with filesystems so
that they can report what *file* is affected by the failure of a
sector (currently a very manual and error-prone process using
filesystem debugging tools).  This would begin to allow automated
recovery of some filesystem metadata, and of "free space" areas inside
filesystems.  With more coordination, even filesystem contents can be
recovered and rewritten.  If the damaged file is part of the system
files, as opposed to user data, it probably exists on a readily
available OS install image, or in an online package repository.
Online backups (on another drive, or in the cloud) provide another
automatable source of recovery data.  And many applications replicate
data storage at an application level (such as the Internet Archive's
item manager).

Does anybody else on the list think it would be a good idea to get
started on this?

	John

------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk