Re: [GSOC] Introduction to HAMMER2 compression feature

On Wed, Jun 12, 2013 at 8:13 AM, Daniel Flores <[email protected]> wrote:

> Hello everyone,
>
> My name is Daniel Flores and my proposal called “HAMMER2 compression
> feature” was accepted for this year’s Google Summer of Code. I already
> posted the draft of my proposal [1] in this kernel list, so I will not
> repeat much of it, but instead I want to focus on some design decisions
> that I’ve already made. Since I’m an inexperienced developer at this point,
> I’d be happy to receive your suggestions and criticism.
>
> The feature itself consists in that it attempts to compress a HAMMER2
> block, which is of 64KB in size. The result should be a block of 32KB,
> 16KB, 8KB, etc. in size.
>
> Currently I’m searching for the algorithms that are the most appropriate
> for a file system. Particularly I’m searching for algorithms that are very
> fast; don’t require a lot of memory and processing power and offer fairly
> good compression rate (but not necessarily the best compression rate out of
> all). Right now I have two candidates – DEFLATE [2] and LZO [3]. Both of
> them have some available implementations which I intend to use, as Alex
> suggested in his review of my proposal.
>
> DEFLATE seems to be a good choice, because it works with small amounts of
> data and has a sliding window of 32KB – just nice for a 64KB block. It is
> based on another algorithm – LZ77, which is successfully used in
> compression feature for NTFS, so hopefully DEFLATE would be good as well.
>
> LZO seems to be a good choice, because, similarly, it works on small
> amounts of data, it is as fast as DEFLATE and was specifically designed to
> have a very fast decompression speed.
>

You might want to consider LZ4 [1].  It was just recently added to ZFS
[2].  It's multi-threaded-aware, extremely fast for both compression and
decompression, and gives very good compression ratios.  It also includes a
short-circuit design to prevent wasting time trying to compress already
compressed blocks or highly incompressible blocks.  It's also OSS and
several reference implementations are available in various languages.

[1] http://code.google.com/p/lz4/
[2] http://wiki.illumos.org/display/illumos/LZ4+Compression

-- 
Freddie Cash
[email protected]