Re: Compressing CPAN Testers test results for long-term archival storage
[email protected] (Doug Bell) Thu, 8 May 2025 18:04:08 +0200
| Newsgroups | perl.cpan.testers.discuss,perl.cpan.testers.discuss |
|---|---|
| Message-ID | <[email protected]> |
--Apple-Mail=_B8A22F75-B7DC-4208-85C9-A773AE9E546D
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
Yeah, a looong long time ago I was hoping Zstd compression + =
dictionaries would solve the problem. I had, though, I think, designed =
some overly-complex systems for doing it, and therefore never got around =
to setting it up.
I did some tests w/ squashfs and got some good results as well. This =
option appeals to me for its transparency: The Zstd + dictionary =
approach means special tools for looking at the data, but squashfs would =
work w/ a standard CLI toolkit. Those results are below.
I'm collecting (heh) up a design spec for this =
<https://github.com/orgs/cpan-testers/discussions/24> in the CPAN =
Testers Discussions under a new Proposal category. And then once we =
isolate this problem, the rest of the problems seem almost trivial ;)
# The count of all reports
cpantesters@cpantesters4:~$ find reports-dir/_meta/timestamp -type f | =
xargs cat | wc -l
44987
# The total size on-disk (I'm assuming w/ extra tail blocks)
cpantesters@cpantesters4:~$ du -sh reports-dir/
614M reports-dir/
# LZ4 squashfs
Exportable Squashfs 4.0 filesystem, lz4 compressed, data block size =
131072
compressed data, compressed metadata, compressed fragments, =
compressed xattrs
duplicates are removed
Filesystem size 127003.50 Kbytes (124.03 Mbytes)
30.41% of uncompressed filesystem size (417572.73 Kbytes)
Inode table size 889805 bytes (868.95 Kbytes)
39.78% of uncompressed inode table size (2237004 bytes)
Directory table size 823072 bytes (803.78 Kbytes)
36.71% of uncompressed directory table size (2242366 bytes)
# XZ squashfs (best compression)
Exportable Squashfs 4.0 filesystem, xz compressed, data block size =
131072
compressed data, compressed metadata, compressed fragments, =
compressed xattrs
duplicates are removed
Filesystem size 92831.63 Kbytes (90.66 Mbytes)
22.23% of uncompressed filesystem size (417572.73 Kbytes)
Inode table size 479692 bytes (468.45 Kbytes)
21.44% of uncompressed inode table size (2237004 bytes)
Directory table size 493816 bytes (482.24 Kbytes)
22.02% of uncompressed directory table size (2242366 bytes)
# LZO squashfs
Exportable Squashfs 4.0 filesystem, lzo compressed, data block size =
131072
compressed data, compressed metadata, compressed fragments, =
compressed xattrs
duplicates are removed
Filesystem size 119522.29 Kbytes (116.72 Mbytes)
28.62% of uncompressed filesystem size (417572.73 Kbytes)
Inode table size 827963 bytes (808.56 Kbytes)
37.01% of uncompressed inode table size (2237004 bytes)
Directory table size 743654 bytes (726.22 Kbytes)
33.16% of uncompressed directory table size (2242366 bytes)
# Gzip squashfs
Exportable Squashfs 4.0 filesystem, gzip compressed, data block size =
131072
compressed data, compressed metadata, compressed fragments, =
compressed xattrs
duplicates are removed
Filesystem size 111798.37 Kbytes (109.18 Mbytes)
26.77% of uncompressed filesystem size (417572.73 Kbytes)
Inode table size 627493 bytes (612.79 Kbytes)
28.05% of uncompressed inode table size (2237004 bytes)
Directory table size 581621 bytes (567.99 Kbytes)
25.94% of uncompressed directory table size=20
# Ztsd squashfs (needed to move to a Debian 12 box to get this)
Exportable Squashfs 4.0 filesystem, zstd compressed
Filesystem size 100603.81 Kbytes (98.25 Mbytes)
24.09% of uncompressed filesystem size (417572.73 Kbytes)
Inode table size 537209 bytes (524.62 Kbytes)
24.01% of uncompressed inode table size (2237004 bytes)
Directory table size 490852 bytes (479.35 Kbytes)
21.89% of uncompressed directory table size (2242366 bytes)
Doug Bell
[email protected]
> On May 5, 2025, at 6:23=E2=80=AFPM, Scott Baker <[email protected]> =
wrote:
>=20
> CPAN Testers:
>=20
> As part of my research into Magpie we came up against a disk space =
hurdle. Currently CPT is ingesting ~25,000 tests per day. After =
capturing a sampling of about 40,000 tests I was able to determine that =
the average test is 9,129 bytes of text. If we store uncompressed text =
that's 223MB per day (81GB per year). Clearly that's not very =
sustainable so we need to look at compression.
>=20
> gzip -9 =3D 3198 bytes
> zstd -12 =3D 3124 bytes
> brotli -9 =3D 2699 bytes
> Brotli is the clear winner for compressing smallish chunks of text. =
Not surprising as that was one of the primary goals when it was =
designed. Compressing with Brotli gets us down to 66MB per day (24GB per =
year) which is more reasonable for sure.
>=20
> Doing some research I came across Zstandard dictionaries =
<x-msg://27/Zstandard%20dictionaries>. Zstandard dictionaries fit our =
use case perfectly: compressing many small but very similar (json, xml, =
etc.) files. I dumped the last 50,000 text test results from CPT and =
created a custom 128KB dictionary file. Using that CPT tuned dictionary =
I was able to get the average size on disk of a test result down to 1087 =
bytes (27MB per day or 10GB per year).
>=20
> As we move forward with reworking the DB side of CPT we should =
definitely consider Zstandard dictionaries. They are well tested, =
relatively easy to use, and well supported =
<https://metacpan.org/pod/Compress::Stream::Zstd::CompressionDictionary> =
by Perl and other tools.
>=20
> High speed database-grade cloud storage is not cheap. Whatever we can =
do to decrease the amount of raw storage we need the better. Lower =
storage usage means faster replication and quicker backups. Have you =
ever tried backing up 1TB of data in the cloud? Spoiler alert: it's not =
easy.
>=20
> -- Scottchiefbaker
>=20
> P.S. For bonus points what if we re-worked what we store? Do we need =
to store "Thank you for uploading your work to CPAN..." Do we need to =
store the opening boiler plate paragraph?
>=20
>=20
>> From: metabase:user:314402c4-2aae-11df-837a-5e0a49663a4f
>> Subject: NA Random-Simple-0.24 5.10.1 FreeBSD
>> Date: 2025-03-31T17:20:02Z
>>=20
>> This distribution has been tested as part of the CPAN Testers
>> project, supporting the Perl programming language. See
>> http://wiki.cpantesters.org/ for more information or email
>> questions to [email protected] =
<mailto:[email protected]>
> P.P.S. Raw numbers for reference:
>=20
>=20
>> perlmagpie> SELECT avg(octet_length(txt_zstd)), count(guid), grade =
FROM test_results INNER JOIN test USING (GUID) GROUP BY grade ORDER BY 1 =
asc LIMIT 30;=20
>> +-----------------------+-------+---------+=20
>> | avg | count | grade |=20
>> |-----------------------+-------+---------|=20
>> | 837.1807610993657505 | 1892 | NA |=20
>> | 862.9752690411719781 | 72015 | PASS |=20
>> | 1286.9555979297194225 | 3671 | UNKNOWN |=20
>> | 1515.2728811352688452 | 15362 | FAIL |=20
>> +-----------------------+-------+---------+=20
>> SELECT 4=20
>> Time: 0.223s
>>=20
>=20
>=20
--Apple-Mail=_B8A22F75-B7DC-4208-85C9-A773AE9E546D
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"content-type" content=3D"text/html; =
charset=3Dutf-8"></head><body style=3D"overflow-wrap: break-word; =
-webkit-nbsp-mode: space; line-break: after-white-space;">Yeah, a looong =
long time ago I was hoping Zstd compression + dictionaries would solve =
the problem. I had, though, I think, designed some overly-complex =
systems for doing it, and therefore never got around to setting it =
up.<div><br></div><div>I did some tests w/ squashfs and got some good =
results as well. This option appeals to me for its transparency: The =
Zstd + dictionary approach means special tools for looking at the data, =
but squashfs would work w/ a standard CLI toolkit. Those results are =
below.</div><div><br></div><div>I'm <a =
href=3D"https://github.com/orgs/cpan-testers/discussions/24">collecting =
(heh) up a design spec for this</a> in the CPAN Testers Discussions =
under a new Proposal category. And then once we isolate this problem, =
the rest of the problems seem almost trivial =
;)</div><div><br></div><blockquote style=3D"margin: 0 0 0 40px; border: =
none; padding: 0px;"><div><div><br></div><div># The count of all =
reports</div><div>cpantesters@cpantesters4:~$ find =
reports-dir/_meta/timestamp -type f | xargs cat | wc =
-l</div><div>44987</div><div><br></div><div># The total size on-disk =
(I'm assuming w/ extra tail =
blocks)</div><div>cpantesters@cpantesters4:~$ du -sh =
reports-dir/</div><div>614M =
reports-dir/</div><div><br></div><div># LZ4 =
squashfs</div><div>Exportable Squashfs 4.0 filesystem, lz4 compressed, =
data block size 131072</div><div> compressed =
data, compressed metadata, compressed fragments, compressed =
xattrs</div><div> duplicates are =
removed</div><div>Filesystem size 127003.50 Kbytes (124.03 =
Mbytes)</div><div> 30.41% of uncompressed =
filesystem size (417572.73 Kbytes)</div><div>Inode table size 889805 =
bytes (868.95 Kbytes)</div><div> 39.78% of =
uncompressed inode table size (2237004 bytes)</div><div>Directory table =
size 823072 bytes (803.78 Kbytes)</div><div> =
36.71% of uncompressed directory table size (2242366 =
bytes)</div><div><br></div><div># XZ squashfs (best =
compression)</div><div>Exportable Squashfs 4.0 filesystem, xz =
compressed, data block size 131072</div><div> =
compressed data, compressed metadata, compressed fragments, compressed =
xattrs</div><div> duplicates are =
removed</div><div>Filesystem size 92831.63 Kbytes (90.66 =
Mbytes)</div><div> 22.23% of uncompressed =
filesystem size (417572.73 Kbytes)</div><div>Inode table size 479692 =
bytes (468.45 Kbytes)</div><div> 21.44% of =
uncompressed inode table size (2237004 bytes)</div><div>Directory table =
size 493816 bytes (482.24 Kbytes)</div><div> =
22.02% of uncompressed directory table size (2242366 =
bytes)</div><div><br></div><div># LZO squashfs</div><div>Exportable =
Squashfs 4.0 filesystem, lzo compressed, data block size =
131072</div><div> compressed data, compressed =
metadata, compressed fragments, compressed xattrs</div><div> =
duplicates are removed</div><div>Filesystem size =
119522.29 Kbytes (116.72 Mbytes)</div><div> =
28.62% of uncompressed filesystem size (417572.73 =
Kbytes)</div><div>Inode table size 827963 bytes (808.56 =
Kbytes)</div><div> 37.01% of uncompressed =
inode table size (2237004 bytes)</div><div>Directory table size 743654 =
bytes (726.22 Kbytes)</div><div> 33.16% of =
uncompressed directory table size (2242366 =
bytes)</div><div><br></div><div># Gzip squashfs</div><div>Exportable =
Squashfs 4.0 filesystem, gzip compressed, data block size =
131072</div><div> compressed data, compressed =
metadata, compressed fragments, compressed xattrs</div><div> =
duplicates are removed</div><div>Filesystem size =
111798.37 Kbytes (109.18 Mbytes)</div><div> =
26.77% of uncompressed filesystem size (417572.73 =
Kbytes)</div><div>Inode table size 627493 bytes (612.79 =
Kbytes)</div><div> 28.05% of uncompressed =
inode table size (2237004 bytes)</div><div>Directory table size 581621 =
bytes (567.99 Kbytes)</div><div> 25.94% of =
uncompressed directory table size </div><div><br></div><div># Ztsd =
squashfs (needed to move to a Debian 12 box to get =
this)</div><div>Exportable Squashfs 4.0 filesystem, zstd =
compressed</div><div>Filesystem size 100603.81 Kbytes (98.25 =
Mbytes)</div><div> 24.09% of uncompressed =
filesystem size (417572.73 Kbytes)</div><div>Inode table size 537209 =
bytes (524.62 Kbytes)</div><div> 24.01% of =
uncompressed inode table size (2237004 bytes)</div><div>Directory table =
size 490852 bytes (479.35 Kbytes)</div><div> =
21.89% of uncompressed directory table size (2242366 =
bytes)</div></div><div><br></div></blockquote><div><div><br></div><div><br=
id=3D"lineBreakAtBeginningOfMessage"><div>
<div>Doug Bell</div><div>[email protected]</div><div><br></div><br =
class=3D"Apple-interchange-newline">
</div>
<div><br><blockquote type=3D"cite"><div>On May 5, 2025, at 6:23=E2=80=AFPM=
, Scott Baker <[email protected]> wrote:</div><br =
class=3D"Apple-interchange-newline"><div>
=20
<meta http-equiv=3D"content-type" content=3D"text/html; =
charset=3DUTF-8">
=20
<div><p>CPAN Testers:</p><p>As part of my research into Magpie we came =
up against a disk
space hurdle. Currently CPT is ingesting ~25,000 tests per day.
After capturing a sampling of about 40,000 tests I was able to
determine that the average test is 9,129 bytes of text. If we
store uncompressed text that's 223MB per day (81GB per year).
Clearly that's not very sustainable so we need to look at
compression.</p>
<ul>
<li>gzip -9 =3D 3198 bytes</li>
<li>zstd -12 =3D 3124 bytes</li>
<li>brotli -9 =3D 2699 bytes</li>
</ul><p>Brotli is the clear winner for compressing smallish chunks =
of
text. Not surprising as that was one of the primary goals when it
was designed. Compressing with Brotli gets us down to 66MB per day
(24GB per year) which is more reasonable for sure.</p><p>Doing =
some research I came across <a =
href=3D"x-msg://27/Zstandard%20dictionaries" =
class=3D"moz-txt-link-freetext">Zstandard
dictionaries</a>. Zstandard dictionaries fit our use case
perfectly: compressing many small but very similar (json, xml,
etc.) files. I dumped the last 50,000 text test results from CPT
and created a custom 128KB dictionary file. Using that <b>CPT
tuned</b> dictionary I was able to get the average size on disk
of a test result down to 1087 bytes (27MB per day or 10GB per
year).<br>
</p><p>As we move forward with reworking the DB side of CPT we =
should
definitely consider Zstandard dictionaries. They are well tested,
relatively easy to use, and <a =
href=3D"https://metacpan.org/pod/Compress::Stream::Zstd::CompressionDictio=
nary">well
supported</a> by Perl and other tools.<br>
</p><p>High speed database-grade cloud storage is not cheap. =
Whatever we
can do to decrease the amount of raw storage we need the better.
Lower storage usage means faster replication and quicker backups.
Have you ever tried backing up 1TB of data in the cloud? Spoiler
alert: it's not easy.</p><p>-- Scottchiefbaker</p><p>P.S. For =
bonus points what if we re-worked what we store? Do we
need to store "Thank you for uploading your work to CPAN..." Do we
need to store the opening boiler plate paragraph? <br>
</p><div>
<br class=3D"webkit-block-placeholder"></div><blockquote =
type=3D"cite">
<pre class=3D"border bg-light p-2 font-scp">From: =
metabase:user:314402c4-2aae-11df-837a-5e0a49663a4f
Subject: NA Random-Simple-0.24 5.10.1 FreeBSD
Date: 2025-03-31T17:20:02Z
This distribution has been tested as part of the CPAN Testers
project, supporting the Perl programming language. See
<a class=3D"moz-txt-link-freetext" =
href=3D"http://wiki.cpantesters.org/">http://wiki.cpantesters.org/</a> =
for more information or email
questions to <a class=3D"moz-txt-link-abbreviated" =
href=3D"mailto:[email protected]">[email protected]=
g</a>
</pre>
</blockquote>
P.P.S. Raw numbers for reference:<div><br =
class=3D"webkit-block-placeholder"></div><div>
<br class=3D"webkit-block-placeholder"></div><blockquote =
type=3D"cite"><font face=3D"monospace">perlmagpie>
SELECT avg(octet_length(txt_zstd)), count(guid), grade FROM
test_results INNER JOIN test USING (GUID) GROUP BY grade ORDER
BY 1 asc LIMIT 30;
<br>
+-----------------------+-------+---------+
<br>
| avg =
&n=
bsp; | count | grade |
<br>
|-----------------------+-------+---------|
<br>
| 837.1807610993657505 | 1892 | NA =
|
<br>
| 862.9752690411719781 | 72015 | PASS =
|
<br>
| 1286.9555979297194225 | 3671 | UNKNOWN |
<br>
| 1515.2728811352688452 | 15362 | FAIL |
<br>
+-----------------------+-------+---------+
<br>
SELECT 4
<br>
Time: 0.223s<br>
<span style=3D"background-color: rgb(255, 255, 255);">
</span><br>
</font></blockquote><div><br =
class=3D"webkit-block-placeholder"></div><p><br>
</p>
</div>
</div></blockquote></div><br></div></div></body></html>=
--Apple-Mail=_B8A22F75-B7DC-4208-85C9-A773AE9E546D--