Re: Compression method recommendations?

On Thu, Sep 09 2021, J. Roeleveld via Dar-support wrote:
> 1) Large collection of text-files (maildirs)
> 2) Large collection of binary files (videos/music)
> 3) Combination of both
>
> What I'm mostly looking for is a good trade-off between speed 
> and compression
> level, but I don't have the time to try every single option on 
> my datasets to
> see which performs best.

Hi Joost,

I have become a fan of zstd over the past year or two.  It 
outperforms bzip2 in terms of compression, at speeds that are more 
like gzip.  You can also achieve speeds more like lz4 (very, very 
fast) while still having compression ratios that outperform gzip. 
In short, it is quite incredible.  I have been using it on a 
number of my ZFS filesystems now, and is my go-to default 
compression tool in place of gzip and bzip2 in quite a few 
instances.

Compared to things like xz/lzma, it has FAR better performance 
with similar or better compression ratios.  Compared to lzo, it 
has a better compression ratio with similar performance.

The sweet spot for ratio vs. performance for me often seems to 
lurk somewhere between -3 and -8, but I've tried -18 for some 
things that could really use it.

Also, zstd is parallelizable.  I'm not sure if dar's support for 
it is, but I will often use -T0 to let it use whatever CPU cores 
it likes.

Basically the only times I go for something else now are when I 
need something more compatible on the receiving end.

- John