JFFS2: improving compression

Dear All,

David asked me to post this letter to this list, too.

The goal of our project (called BBC, http://www.inf.u-szeged.hu/jffs2 ) 
is to improve the compression performance of JFFS2. There was an e-mail 
conversation between me and David how to insert BBC into the official 
JFFS2, and this is my last letter - with a patch.

Regards,

Ferenc

P.S.: David: I'll send the seperated patches tomorrow.

-------- Original Message --------
Subject: 	BBC patch
Date: 	Wed, 21 Apr 2004 00:04:34 +0200
From: 	Ferenc Havasi <[email protected]>
To: 	David Woodhouse <[email protected]>

Hi,

I rewrote the source of BBC for JFFS2.

This is the version I called "first step" in my last letter. I does not 
support model file yet, and this is the extended version of the sources 
you sended me some weeks ago. The patch for the latest CVS version is 
attached.

It supports 3 compression mode:
- none (no compression)
- priority compression (it tries the compressors
   in order of their priorities, and chooses the
   first successful - such the original does)
- size (it tries all the compressors and chooses
   the smallest result)

Compressors:
- zlib
- rtime
- dynrubin and rubinmips
- lzo (faster than zlib but has worse compression ratio)
- lzari (it has better compression ratio than zlib but
   much slower)

The two new ones are disabled for default.

The compressor interface support is in compr.h and compr.c,
which are used both in mkfs and kernel. (and compr.h are
used by compressor modules also, to make easier to alloc
memory, print error msgs...)

The mkfs.jffs2 has 5 new switches:
   -m, --compression-mode   Select compression mode (default: priortiry)
   -L, --list-compressors   Show the list of the avaiable compressors
   -x, --disable-compressor Disable a compressor
   -X, --enable-compressor  Enable a compressor
   -t, --test-compression   Call decompress and compare with the 
original (for test)

By default compressor lzo and lzari are disabled, and the compression 
mode is priority so if someone doesn't use the new swithes, the 
generated JFFS2 image will be exactly the same than using the mkfs 
without this patch.

In the kernel the compression support, the compressors plugins and
the default compression mode can be selected by "make menuconfig"/...

I tested both in kernel 2.6.x and mkfs and it seems that it works.

Using 2.4.x kernels it compiles and works as bad as the CVS version 
without BBC patch (so it doesn't work on my system with mtdram device, 
but it is true using the non-patched CVS version, too).

A short summary of the modified files:

fs:
   Config.in           config for compressor(s)
   Kconfig             config for compressor(s)
fs/jffs2:
   Makefile.common     support new kconfig
   compr.c
   compr.h
   compr_*.c           implement compression interface
   gc.c                jffs2_compress return value (preparing for models)
   nodelist.h          some function header to compr.h
   os-linux.h          compressor config now comes from kconfig
   read.c              jffs2_decompress parameter (preparing for models)
   super-v24.c         compressor_init/deinit
   super.c             compressor_init/deinit
   write.c             jffs2_compress return value (preparing for models)
include/linux:
   jffs2.h             new JFFS2_COMPR_XXX values
patches:
   patchin.sh          2.6.x kernel config patch
util:
   Makefile            new compr_*.c, compr.c, compr.h symliks...
   compr_zlib.c        implement compression interface
   mkfs.jffs2.1        document new switches of mkfs.jffs2
   mkfs.jffs2.c        compr_init, new switches, call compr.c...

There is only one #define that should be fixed in compr.h:
the value of JFFS2_PAGE_SIZE should be the maximal value of *cdatalen.
It is clear in mkfs.jffs2.c (and that is OK) but in the kernel I don't
where it is exatly comes, so I definied it to 4096 directly. It is used 
by the size compression mode and the lzo modul. (so the default config not)

If you like this patch and it is OK for you we'll continue the work with 
2-3 steps:

Step A
- some optional proc support to make is possible to switch
   compression mode, read some statistics, enable/disable compressors
- some additional compression mode based on estimator functions

Step B
- imlement model file support, and add our compressor: ARMLIB.
   Before it we should fix the technical details how to store the
   model files.

Step C
- documentation, howtos...

Bye,
Ferenc