Re: [GSOC] HAMMER2 compression feature week1 report

On Sun, Jul 28, 2013 at 7:27 PM, Radio młodych bandytów <
[email protected]> wrote:

> Thanks for the data. I took a quick glance now, later will do some more.
> It appears that the requirement to halve input size is just too much for
> lz4 to notably compress even English text...I suspect that few workloads
> will benefit from LZ4 compression then. There's a need for something
> stronger...
>

I think, the compression will be beneficial for log files, source code
files and some types of uncompressed files. I don't think that changing the
algorithm will significantly change how things are, because we have certain
limitations, like we compress only one block at a time and we don't
reference previous blocks at this moment, so the algorithms don't perform
at their full potential.

But I'll try other algorithms if I'll have enough time.


> Anyway, my results with LZ4 r97 are slightly different, it managed to
> halve 1 block of book1. Which version do you use?
>

I think I use r97 as well. Are you sure that your compression conditions
are the same as mine? I only try to compress one 64KB block at a time and I
don't use the information from previous blocks or anything else.


> Overall, I have a feeling that a stronger LZ77 would be a better fit.
> Something like https://code.google.com/p/data-shrinker/ (Warning: the
> code is demo quality), Shrinker is the strongest-of-fast.
>

OK, this looks promising. When I was choosing an algorithm for the
compression feature, one of the candidates was DEFLATE, which is based on
LZ77. So, I'll definitely consider this data-shrinker as an alternative to
LZ4.

Thank you.


Daniel