[SPDK] Re: Bad Sectors / expected 'error' responses & timing

To add on to these responses, I'd recommend this article to come up to speed with what's generally happening inside of a modern SSD: https://spdk.io/doc/ssd_internals.html. They're very complex devices. Note that modern HDDs, especially those leveraging SMR, now do similar things too.

The most important take-away from the above is that the "logical blocks" that a user performs I/O to really are just logical constructs - they don't correspond to a specific physical location on the drive. For example, if the drive contains NAND, it may address the NAND internally as a number that represents the NAND channel, the NAND die, the particular erase block, and the offset into it, all combined into a single integer. The mapping of the logical blocks that the user sees to the NAND blocks (usually called pages) is *dynamic* and held in a logical to physical lookup table on the device. Most of my discussion will focus on NAND, but a lot of these same things happen for other types of media too.

This means that software, such as the filesystem, no longer needs to track the locations of bad blocks after they've been "corrected" by a rewrite to the logical block. Once the drive detects a bad block, it will automatically remove it from the pool of available NAND behind the scenes. Further, since the device is attempting to spread the writes over all of the physical NAND, once a single NAND block fails it's very likely the rest of the drive is going to follow shortly after (since every NAND block has about the same wear level). NAND does tend to fail in larger units (erase-blocks), but outside of manufacturing defects, the failures tend to happen only when the entire drive is going to die anyway. You can usually get an indication of how many bad blocks have internally been removed by viewing the SMART data for the drive.

Every NVMe drive may be different, but on many enterprise-class drives writes operations cannot fail unless the drive itself fails as a unit. They'll simply keep retrying the write in different physical locations until it works, making the write take longer.
Separately, almost all NAND drives already store some amount of parity and checksum data for each data block which is verified on a read. Data that corrupts at rest on the drive will be caught with a statistical certainty typically published on the drive's spec sheet. The value is usually called the "uncorrectable bit error rate", or UBER. Reads that are determined to have junk data will return a failure code according to the NVMe specification. If you need a lower error rate, you can add additional checksums either via end to end data protection or by writing them to another place on the drive (or buy a different device with a lower UBER).

I hope that helps. Cool project. I also briefly worked on small satellites in college!

Thanks,
Ben


> -----Original Message-----
> From: Harris, James R <james.r.harris(a)intel.com>
> Sent: Monday, June 22, 2020 9:28 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Re: Bad Sectors / expected 'error' responses & timing
> 
> Hi Alvar,
> 
> You may also want to refer to the NVM Express specification (nvmexpress.org)
> to read about the different kinds of status code that can be returned for read
> and write commands.  The SPDK APIs return these NVMe status codes to the
> caller via the "struct spdk_nvme_cpl *" parameter that is passed to the
> completion callback function.  Your code can then interpret these status codes
> to look for and handle unrecoverable read errors.
> 
> Timing impacts are difficult to define. Much of it can be dependent on the SSD
> and its firmware. For example, before returning an unrecovered read error, the
> SSD may try a bunch of recovery mechanisms internally before returning the
> read error to the host.
> 
> Regards,
> 
> Jim
> 
> 
> On 6/21/20, 9:28 PM, "松本周平" <shuheimatsumoto(a)gmail.com> wrote:
> 
>     Hi Alvar,
> 
>     Thanks for sharing your exciting project and your interest in SPDK.
>     About detecting bad sectors, how about end to end data protection (e.g. T10
>     DIF) ?
> 
>     Please take a look at the section 8.3 of NVMe specification.
>     https://nvmexpress.org/wp-content/uploads/NVM-Express-1_4a-2020.03.09-
> Ratified.pdf
>     Please also take a look at
>     https://en.wikipedia.org/wiki/Data_Integrity_Field
> 
>     SPDK NVMe API supports this feature.
> 
>     I think end to end data protection may satisfy your requirement but I'm not
>     certain yet.
> 
>     Others may be able to provide better information.
> 
>     Thanks,
>     Shuhei
> 
> 
>     On Sat, Jun 20, 2020 at 3:08 AM <alvarso(a)mit.edu> wrote:
> 
>     > Hello SPDK team,
>     >
>     > First, thank you for the really cool work you are doing!  I am working on
>     > a small satellite mission at MIT, which will use a 6-channel SDR (SW
>     > defined radio). My task is to ensure that the 6-channel SDR data is saved
>     > reliably to an SSD. I am working on the processor (PS) side. Another
>     > colleague is working on the FPGA PL (programmable logic) side. The FPGA
>     > will provide DMA (still under development).
>     >
>     > My general idea to try to achieve zero-copy performance is:
>     >
>     >  ADC -> PL Queue
>     >    PS Determines SDRAM temporary storage
>     >  DMA from PL Quque -> SDRAM  (likely using libiio)
>     >   PS Determines when a block (or other storage unit, TBD) is ready to go
>     > to SSD
>     >    -> because our data is always the same size/format, I believe we can
>     > use an analytic/deterministic equation to determine the storage location
>     >  DMA from SDRAM -> SSD (likely using SPDK)
>     >
>     > when its time to 'process' the data (which has to be at a later time due
>     > to power limits of the satellite):
>     >
>     >  PS determines (analytic equation) data to be processed
>     >  DMA from SSD -> SDRAM (w SPDK)
>     >  PS informs PL of data vailable
>     >  PL processes data via DMA
>     >  PL informs new 'processed data queue' ready
>     >  PS prepares location for processed data
>     >  DMA from SDRAM -> SSD (w SPDK)
>     >
>     > The mission PI (principal investigator) has one main worry of our
>     > approach: he is concerned that SSD's can end up with 'bad sectors', like
>     > older drives, but that its usually a big chunk of space that goes bad. We
>     > are not concerned about single-event-upsets (when just one individual piece
>     > of data gets damaged), but rather when a large section that can result in
>     > us loosing too much data.
>     >
>     > I understand that the idea of keeping track of 'bad sectors' in 'hard
>     > drives' is usually the task of a file-system. However, for our purposes a
>     > file system is appearing to be too much overhead and we have not found
> one
>     > that would help us with a 'zero copy' setup. But we need to be able to know
>     > when there are data errors (read data is garbage) and not slow down if
>     > there is a bad write request (if a bad write slows down the system, then we
>     > loose 'new' data that should have been saved).
>     >
>     > I read the documentation as much as possible, and did a good amount of
>     > online searching for the 'expected' response from SPDK when the SSD has
>     > errors. But I could not find any information on that. I would greatly
>     > appreciate if anyone in the team can guide me in the right direction (maybe
>     > its pointing to some standard that SPDK adheres to [NVMe & PCIe] but even
>     > that I was not sure how SPDK returns such errors and the expected timing of
>     > them).
>     >
>     > Hopefully this was clear and its in the scope of this list; if its not,
>     > please ask me to clarify or I greatly appreciate if you point me in the
>     > right direction.
>     >
>     > Thank you!
>     >
>     > Alvar
>     >
>     > PS, Summary:
>     > - Trying to do zero-copy 6-channel data saving from FPGA to SSD (PCIe
> NVME)
>     > - If I don't want a full file-system, how can I handle 'bad sector' type
>     > errors in the SSD?
>     > - Is there any spec of expectation on the timing impacts when an error
>     > occurs?
>     > _______________________________________________
>     > SPDK mailing list -- spdk(a)lists.01.org
>     > To unsubscribe send an email to spdk-leave(a)lists.01.org
>     >
>     _______________________________________________
>     SPDK mailing list -- spdk(a)lists.01.org
>     To unsubscribe send an email to spdk-leave(a)lists.01.org
> 
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org