Re: metadata compression

Joshua Bahnsen <[email protected]> writes:

> I am keeping track of 16 RHEL channels, using createrepo with the standard
> gzip I am totaling 1.4 GB of metadata.

 How many arches is that for?

> Compressing those same XML documents
> with LZMA yields a total of 140 MB. That's 10x savings overall, I think
> that's worth a look.

 Well, again, it'd depend on what it did _for the .sqlite_ files. As
shipping the .xml files to the client machines is suboptimal in many
ways.

 Doing some quick tests:

 CentOS-5
 ---------
 primary.xml          = 5.3M
 primary.xml.gz       = 888K
 primary.xml.bz2      = 584K
 primary.xml.lz       = 540K

...so I'm not sure how you get 10x. Although for the .sqlite data it
seems to do a little better:

 Fedora-rawhide
 --------------
 primary.sqlite       = 37M
 primary.sqlite.gz    = 12M
 primary.sqlite.bz2   = 8.5M
 primary.sqlite.lz    = 6.8M

 filelists.sqlite     = 66M
 filelists.sqlite.bz  = 15M
 filelists.sqlite.bz2 = 13M
 filelists.sqlite.lz  = 11M

 other.sqlite         = 19M
 other.sqlite.gz      = 6.5M
 other.sqlite.bz2     = 4.6M
 other.sqlite.lz      = 2.8M

...which implies somewhere in the 25-35% savings range, but I doubt
that's enough (on it's own) given the CPU/code requirements.

-- 
James Antill -- [email protected]