Re: [svlug-announce] SVLUG Jan. 17th meeting: Louis Imershein on Inline Dedupe & Compression for Linux Block Storage

On Tue, Jan 16, 2018 at 10:20:47AM -0500, Steve Litt wrote:
> On Tue, 16 Jan 2018 04:16:30 -0800
> Rick Moen <rick-IyCrq+X4Fdq2oZ/[email protected]> wrote:
> 
> > WHEN:
> > 
> > Wednesday, January 17th, 2018
> > 7 - 8:30pm
> > 
> > 
> > 
> > MAIN PRESENTATION
> > 
> > TOPIC: Inline Dedupe & Compression for Linux Block Storage
> > 
> > PRESENTED BY: Louis Imershein,
> > Principal Product Manager for Data Reduction Technologies, Red Hat
> > 
> > TOPIC SUMMARY:
> > 
> > Data deduplication and compression can increase the capacity of
> > storage, allowing users make more efficient use of existing storage,
> > and to make new fast storage more affordable. 
> 
> [snip]
> 
> > The Virtual Data Optimizer
> > (https://access.redhat.com/ecosystem/software/1484263) dm-vdo) is a
> > new open source project from Red Hat that provides inline data
> > reduction for primary storage to the Linux block-storage stack.
> 
> Eliminating redundancy (duplication) can obviously increase storage
> density, but I've found many situations where the existance of some
> redundancy helped recover lost information when some little thing went
> bad.
> 
> I can't attend the meeting because it's kind of a long drive, but I
> hope that any facilities to make the newly deduplicated data less
> brittle will be discussed. For instance, I'm wondering if some kind of
> parity scheme is added to error-correct, but at a lower disk space cost
> than a simple filesystem?

The in-RAM costs are significantly lower than ZFS dedupe:

http://permabit.com/products-overview/albireo-virtual-data-optimizer-vdo/

so I would guess that it really skimps on error-correction
compared to ZFS.

On the other hand, I bet that the associated inline compression
offers similar performance advantages as ZFS. Reading and
writing less data is usually a win. (Obvious exception for
pre-compressed data or specialized data like audio and video).

-dsr-