Re: Compressing CPAN Testers test results for long-term archival storage
[email protected] (Doug Bell) Fri, 9 May 2025 12:50:54 +0200
| Newsgroups | perl.cpan.testers.discuss |
|---|---|
| Message-ID | <[email protected]> |
--Apple-Mail=_EB170F5E-E6C2-4288-A6CB-42D9B7972D45
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
Yeah: Here's one of the articles I read about using filesystem tricks to =
make a "read-write" squashfs: =
https://www.baeldung.com/linux/squashfs-filesystem-mount . TL;DR: An =
"overlay" filesystem is used to combine a writable system on top with =
read-only system(s) underneath so the writes happen to the top one and =
reads fall back to the bottom one(s).
/mnt/reports-all - Combined overlay
/mnt/reports-new - Writable btrfs
/mnt/reports-2025-W32 - SquashFS for a single week
/mnt/reports-2025-01 - SquashFS for a single month
/mnt/reports/2024 - SquashFS for a whole year
Once a week passes, I can make a new weekly /mnt/reports-2025-W33 with =
all the reports in /mnt/reports-new. Once I have enough of those, I can =
use them to make a new monthly /mnt/reports-2025-07, and etc... for the =
yearly archive.
My hope is that each of these archives are not too unwieldy to transport =
between systems for backups and replication and etc... And I imagine the =
more layers there are the more time (and CPU) it takes to find the =
report. So there may be some adjustments to the exact layout of the =
whole thing.
Doug Bell
[email protected]
> On May 8, 2025, at 6:15=E2=80=AFPM, Scott Baker <[email protected]> =
wrote:
>=20
> I don't know a ton about SquashFS but some reading on Wikipedia says =
it's a read-only filesystem. How would CPT use SquashFS? Storing report =
data?
>=20
> -- Scottchiefbaker
>=20
> On 5/8/25 9:04 AM, Doug Bell wrote:
>> Yeah, a looong long time ago I was hoping Zstd compression + =
dictionaries would solve the problem. I had, though, I think, designed =
some overly-complex systems for doing it, and therefore never got around =
to setting it up.
>>=20
>> I did some tests w/ squashfs and got some good results as well. This =
option appeals to me for its transparency: The Zstd + dictionary =
approach means special tools for looking at the data, but squashfs would =
work w/ a standard CLI toolkit. Those results are below.
>>=20
>> I'm collecting (heh) up a design spec for this =
<https://github.com/orgs/cpan-testers/discussions/24> in the CPAN =
Testers Discussions under a new Proposal category. And then once we =
isolate this problem, the rest of the problems seem almost trivial ;)
>>=20
>>=20
>> # The count of all reports
>> cpantesters@cpantesters4:~$ find reports-dir/_meta/timestamp -type f =
| xargs cat | wc -l
>> 44987
>>=20
>> # The total size on-disk (I'm assuming w/ extra tail blocks)
>> cpantesters@cpantesters4:~$ du -sh reports-dir/
>> 614M reports-dir/
>>=20
>> # LZ4 squashfs
>> Exportable Squashfs 4.0 filesystem, lz4 compressed, data block size =
131072
>> compressed data, compressed metadata, compressed fragments, =
compressed xattrs
>> duplicates are removed
>> Filesystem size 127003.50 Kbytes (124.03 Mbytes)
>> 30.41% of uncompressed filesystem size (417572.73 Kbytes)
>> Inode table size 889805 bytes (868.95 Kbytes)
>> 39.78% of uncompressed inode table size (2237004 bytes)
>> Directory table size 823072 bytes (803.78 Kbytes)
>> 36.71% of uncompressed directory table size (2242366 bytes)
>>=20
>> # XZ squashfs (best compression)
>> Exportable Squashfs 4.0 filesystem, xz compressed, data block size =
131072
>> compressed data, compressed metadata, compressed fragments, =
compressed xattrs
>> duplicates are removed
>> Filesystem size 92831.63 Kbytes (90.66 Mbytes)
>> 22.23% of uncompressed filesystem size (417572.73 Kbytes)
>> Inode table size 479692 bytes (468.45 Kbytes)
>> 21.44% of uncompressed inode table size (2237004 bytes)
>> Directory table size 493816 bytes (482.24 Kbytes)
>> 22.02% of uncompressed directory table size (2242366 bytes)
>>=20
>> # LZO squashfs
>> Exportable Squashfs 4.0 filesystem, lzo compressed, data block size =
131072
>> compressed data, compressed metadata, compressed fragments, =
compressed xattrs
>> duplicates are removed
>> Filesystem size 119522.29 Kbytes (116.72 Mbytes)
>> 28.62% of uncompressed filesystem size (417572.73 Kbytes)
>> Inode table size 827963 bytes (808.56 Kbytes)
>> 37.01% of uncompressed inode table size (2237004 bytes)
>> Directory table size 743654 bytes (726.22 Kbytes)
>> 33.16% of uncompressed directory table size (2242366 bytes)
>>=20
>> # Gzip squashfs
>> Exportable Squashfs 4.0 filesystem, gzip compressed, data block size =
131072
>> compressed data, compressed metadata, compressed fragments, =
compressed xattrs
>> duplicates are removed
>> Filesystem size 111798.37 Kbytes (109.18 Mbytes)
>> 26.77% of uncompressed filesystem size (417572.73 Kbytes)
>> Inode table size 627493 bytes (612.79 Kbytes)
>> 28.05% of uncompressed inode table size (2237004 bytes)
>> Directory table size 581621 bytes (567.99 Kbytes)
>> 25.94% of uncompressed directory table size=20
>>=20
>> # Ztsd squashfs (needed to move to a Debian 12 box to get this)
>> Exportable Squashfs 4.0 filesystem, zstd compressed
>> Filesystem size 100603.81 Kbytes (98.25 Mbytes)
>> 24.09% of uncompressed filesystem size (417572.73 Kbytes)
>> Inode table size 537209 bytes (524.62 Kbytes)
>> 24.01% of uncompressed inode table size (2237004 bytes)
>> Directory table size 490852 bytes (479.35 Kbytes)
>> 21.89% of uncompressed directory table size (2242366 bytes)
>>=20
>>=20
>>=20
>> Doug Bell
>> [email protected] <mailto:[email protected]>
>>=20
>>=20
>>=20
>>> On May 5, 2025, at 6:23=E2=80=AFPM, Scott Baker <[email protected]> =
<mailto:[email protected]> wrote:
>>>=20
>>> CPAN Testers:
>>>=20
>>> As part of my research into Magpie we came up against a disk space =
hurdle. Currently CPT is ingesting ~25,000 tests per day. After =
capturing a sampling of about 40,000 tests I was able to determine that =
the average test is 9,129 bytes of text. If we store uncompressed text =
that's 223MB per day (81GB per year). Clearly that's not very =
sustainable so we need to look at compression.
>>>=20
>>> gzip -9 =3D 3198 bytes
>>> zstd -12 =3D 3124 bytes
>>> brotli -9 =3D 2699 bytes
>>> Brotli is the clear winner for compressing smallish chunks of text. =
Not surprising as that was one of the primary goals when it was =
designed. Compressing with Brotli gets us down to 66MB per day (24GB per =
year) which is more reasonable for sure.
>>>=20
>>> Doing some research I came across Zstandard dictionaries =
<x-msg://27/Zstandard%20dictionaries>. Zstandard dictionaries fit our =
use case perfectly: compressing many small but very similar (json, xml, =
etc.) files. I dumped the last 50,000 text test results from CPT and =
created a custom 128KB dictionary file. Using that CPT tuned dictionary =
I was able to get the average size on disk of a test result down to 1087 =
bytes (27MB per day or 10GB per year).
>>>=20
>>> As we move forward with reworking the DB side of CPT we should =
definitely consider Zstandard dictionaries. They are well tested, =
relatively easy to use, and well supported =
<https://metacpan.org/pod/Compress::Stream::Zstd::CompressionDictionary> =
by Perl and other tools.
>>>=20
>>> High speed database-grade cloud storage is not cheap. Whatever we =
can do to decrease the amount of raw storage we need the better. Lower =
storage usage means faster replication and quicker backups. Have you =
ever tried backing up 1TB of data in the cloud? Spoiler alert: it's not =
easy.
>>>=20
>>> -- Scottchiefbaker
>>>=20
>>> P.S. For bonus points what if we re-worked what we store? Do we need =
to store "Thank you for uploading your work to CPAN..." Do we need to =
store the opening boiler plate paragraph?
>>>=20
>>>=20
>>>> From: metabase:user:314402c4-2aae-11df-837a-5e0a49663a4f
>>>> Subject: NA Random-Simple-0.24 5.10.1 FreeBSD
>>>> Date: 2025-03-31T17:20:02Z
>>>>=20
>>>> This distribution has been tested as part of the CPAN Testers
>>>> project, supporting the Perl programming language. See
>>>> http://wiki.cpantesters.org/ for more information or email
>>>> questions to [email protected] =
<mailto:[email protected]>
>>> P.P.S. Raw numbers for reference:
>>>=20
>>>=20
>>>> perlmagpie> SELECT avg(octet_length(txt_zstd)), count(guid), grade =
FROM test_results INNER JOIN test USING (GUID) GROUP BY grade ORDER BY 1 =
asc LIMIT 30;=20
>>>> +-----------------------+-------+---------+=20
>>>> | avg | count | grade |=20
>>>> |-----------------------+-------+---------|=20
>>>> | 837.1807610993657505 | 1892 | NA |=20
>>>> | 862.9752690411719781 | 72015 | PASS |=20
>>>> | 1286.9555979297194225 | 3671 | UNKNOWN |=20
>>>> | 1515.2728811352688452 | 15362 | FAIL |=20
>>>> +-----------------------+-------+---------+=20
>>>> SELECT 4=20
>>>> Time: 0.223s
>>>>=20
>>>=20
>>>=20
>>=20
--Apple-Mail=_EB170F5E-E6C2-4288-A6CB-42D9B7972D45
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"content-type" content=3D"text/html; =
charset=3Dutf-8"></head><body style=3D"overflow-wrap: break-word; =
-webkit-nbsp-mode: space; line-break: after-white-space;">Yeah: Here's =
one of the articles I read about using filesystem tricks to make a =
"read-write" squashfs: <a =
href=3D"https://www.baeldung.com/linux/squashfs-filesystem-mount">https://=
www.baeldung.com/linux/squashfs-filesystem-mount</a> . TL;DR: An =
"overlay" filesystem is used to combine a writable system on top with =
read-only system(s) underneath so the writes happen to the top one and =
reads fall back to the bottom one(s).<div><br></div><div> =
/mnt/reports-all - Combined overlay</div><div> =
/mnt/reports-new - Writable btrfs</div><div> =
/mnt/reports-2025-W32 - SquashFS for a single week</div><div> =
/mnt/reports-2025-01 - SquashFS for a single =
month</div><div> /mnt/reports/2024 - SquashFS for a whole =
year</div><div><br></div><div>Once a week passes, I can make a new =
weekly /mnt/reports-2025-W33 with all the reports in /mnt/reports-new. =
Once I have enough of those, I can use them to make a new monthly =
/mnt/reports-2025-07, and etc... for the yearly =
archive.</div><div><br></div><div>My hope is that each of these archives =
are not too unwieldy to transport between systems for backups and =
replication and etc... And I imagine the more layers there are the more =
time (and CPU) it takes to find the report. So there may be some =
adjustments to the exact layout of the whole thing.</div><div><br =
id=3D"lineBreakAtBeginningOfMessage"><div>
<div>Doug Bell</div><div>[email protected]</div><div><br></div><br =
class=3D"Apple-interchange-newline">
</div>
<div><br><blockquote type=3D"cite"><div>On May 8, 2025, at 6:15=E2=80=AFPM=
, Scott Baker <[email protected]> wrote:</div><br =
class=3D"Apple-interchange-newline"><div>
=20
<meta http-equiv=3D"Content-Type" content=3D"text/html; =
charset=3DUTF-8">
=20
<div><p>I don't know a ton about SquashFS but some reading on =
Wikipedia
says it's a read-only filesystem. How would CPT use SquashFS?
Storing report data?<br>
</p><p>-- Scottchiefbaker<br>
</p>
<div class=3D"moz-cite-prefix">On 5/8/25 9:04 AM, Doug Bell =
wrote:<br>
</div>
<blockquote type=3D"cite" =
cite=3D"mid:[email protected]">
<meta http-equiv=3D"content-type" content=3D"text/html; =
charset=3DUTF-8">
Yeah, a looong long time ago I was hoping Zstd compression +
dictionaries would solve the problem. I had, though, I think,
designed some overly-complex systems for doing it, and therefore
never got around to setting it up.
<div><br>
</div>
<div>I did some tests w/ squashfs and got some good results as
well. This option appeals to me for its transparency: The Zstd +
dictionary approach means special tools for looking at the data,
but squashfs would work w/ a standard CLI toolkit. Those results
are below.</div>
<div><br>
</div>
<div>I'm <a =
href=3D"https://github.com/orgs/cpan-testers/discussions/24" =
moz-do-not-send=3D"true">collecting (heh) up a design spec for
this</a> in the CPAN Testers Discussions under a new =
Proposal
category. And then once we isolate this problem, the rest of the
problems seem almost trivial ;)</div>
<div><br>
</div>
<blockquote style=3D"margin: 0 0 0 40px; border: none; padding: =
0px;">
<div>
<div><br>
</div>
<div># The count of all reports</div>
<div>cpantesters@cpantesters4:~$ find
reports-dir/_meta/timestamp -type f | xargs cat | wc =
-l</div>
<div>44987</div>
<div><br>
</div>
<div># The total size on-disk (I'm assuming w/ extra tail
blocks)</div>
<div>cpantesters@cpantesters4:~$ du -sh reports-dir/</div>
<div>614M reports-dir/</div>
<div><br>
</div>
<div># LZ4 squashfs</div>
<div>Exportable Squashfs 4.0 filesystem, lz4 compressed, data
block size 131072</div>
<div> compressed data, compressed =
metadata, compressed
fragments, compressed xattrs</div>
<div> duplicates are removed</div>
<div>Filesystem size 127003.50 Kbytes (124.03 Mbytes)</div>
<div> 30.41% of uncompressed =
filesystem size (417572.73
Kbytes)</div>
<div>Inode table size 889805 bytes (868.95 Kbytes)</div>
<div> 39.78% of uncompressed inode =
table size (2237004
bytes)</div>
<div>Directory table size 823072 bytes (803.78 Kbytes)</div>
<div> 36.71% of uncompressed =
directory table size
(2242366 bytes)</div>
<div><br>
</div>
<div># XZ squashfs (best compression)</div>
<div>Exportable Squashfs 4.0 filesystem, xz compressed, data
block size 131072</div>
<div> compressed data, compressed =
metadata, compressed
fragments, compressed xattrs</div>
<div> duplicates are removed</div>
<div>Filesystem size 92831.63 Kbytes (90.66 Mbytes)</div>
<div> 22.23% of uncompressed =
filesystem size (417572.73
Kbytes)</div>
<div>Inode table size 479692 bytes (468.45 Kbytes)</div>
<div> 21.44% of uncompressed inode =
table size (2237004
bytes)</div>
<div>Directory table size 493816 bytes (482.24 Kbytes)</div>
<div> 22.02% of uncompressed =
directory table size
(2242366 bytes)</div>
<div><br>
</div>
<div># LZO squashfs</div>
<div>Exportable Squashfs 4.0 filesystem, lzo compressed, data
block size 131072</div>
<div> compressed data, compressed =
metadata, compressed
fragments, compressed xattrs</div>
<div> duplicates are removed</div>
<div>Filesystem size 119522.29 Kbytes (116.72 Mbytes)</div>
<div> 28.62% of uncompressed =
filesystem size (417572.73
Kbytes)</div>
<div>Inode table size 827963 bytes (808.56 Kbytes)</div>
<div> 37.01% of uncompressed inode =
table size (2237004
bytes)</div>
<div>Directory table size 743654 bytes (726.22 Kbytes)</div>
<div> 33.16% of uncompressed =
directory table size
(2242366 bytes)</div>
<div><br>
</div>
<div># Gzip squashfs</div>
<div>Exportable Squashfs 4.0 filesystem, gzip compressed, data
block size 131072</div>
<div> compressed data, compressed =
metadata, compressed
fragments, compressed xattrs</div>
<div> duplicates are removed</div>
<div>Filesystem size 111798.37 Kbytes (109.18 Mbytes)</div>
<div> 26.77% of uncompressed =
filesystem size (417572.73
Kbytes)</div>
<div>Inode table size 627493 bytes (612.79 Kbytes)</div>
<div> 28.05% of uncompressed inode =
table size (2237004
bytes)</div>
<div>Directory table size 581621 bytes (567.99 Kbytes)</div>
<div> 25.94% of uncompressed =
directory table size </div>
<div><br>
</div>
<div># Ztsd squashfs (needed to move to a Debian 12 box to get
this)</div>
<div>Exportable Squashfs 4.0 filesystem, zstd compressed</div>
<div>Filesystem size 100603.81 Kbytes (98.25 Mbytes)</div>
<div> 24.09% of uncompressed =
filesystem size (417572.73
Kbytes)</div>
<div>Inode table size 537209 bytes (524.62 Kbytes)</div>
<div> 24.01% of uncompressed inode =
table size (2237004
bytes)</div>
<div>Directory table size 490852 bytes (479.35 Kbytes)</div>
<div> 21.89% of uncompressed =
directory table size
(2242366 bytes)</div>
</div>
<div><br>
</div>
</blockquote>
<div>
<div><br>
</div>
<div><br id=3D"lineBreakAtBeginningOfMessage">
<div>
<div>Doug Bell</div>
<div><a class=3D"moz-txt-link-abbreviated" =
href=3D"mailto:[email protected]">[email protected]</a></div>
<div><br>
</div>
<br class=3D"Apple-interchange-newline">
</div>
<div><br>
<blockquote type=3D"cite">
<div>On May 5, 2025, at 6:23=E2=80=AFPM, Scott Baker
<a class=3D"moz-txt-link-rfc2396E" =
href=3D"mailto:[email protected]"><[email protected]></a> =
wrote:</div>
<br class=3D"Apple-interchange-newline">
<div>
<meta http-equiv=3D"content-type" content=3D"text/html; =
charset=3DUTF-8">
<div><p>CPAN Testers:</p><p>As part of my research into =
Magpie we came up
against a disk space hurdle. Currently CPT is
ingesting ~25,000 tests per day. After capturing a
sampling of about 40,000 tests I was able to
determine that the average test is 9,129 bytes of
text. If we store uncompressed text that's 223MB per
day (81GB per year). Clearly that's not very
sustainable so we need to look at compression.</p>
<ul>
<li>gzip -9 =3D 3198 bytes</li>
<li>zstd -12 =3D 3124 bytes</li>
<li>brotli -9 =3D 2699 bytes</li>
</ul><p>Brotli is the clear winner for compressing =
smallish
chunks of text. Not surprising as that was one of
the primary goals when it was designed. Compressing
with Brotli gets us down to 66MB per day (24GB per
year) which is more reasonable for sure.</p><p>Doing =
some research I came across <a =
href=3D"x-msg://27/Zstandard%20dictionaries" =
class=3D"moz-txt-link-freetext" moz-do-not-send=3D"true">Zstandard =
dictionaries</a>.
Zstandard dictionaries fit our use case perfectly:
compressing many small but very similar (json, xml,
etc.) files. I dumped the last 50,000 text test
results from CPT and created a custom 128KB
dictionary file. Using that <b>CPT tuned</b>
dictionary I was able to get the average size on
disk of a test result down to 1087 bytes (27MB per
day or 10GB per year).<br>
</p><p>As we move forward with reworking the DB side =
of
CPT we should definitely consider Zstandard
dictionaries. They are well tested, relatively easy
to use, and <a =
href=3D"https://metacpan.org/pod/Compress::Stream::Zstd::CompressionDictio=
nary" moz-do-not-send=3D"true">well supported</a> by Perl
and other tools.<br>
</p><p>High speed database-grade cloud storage is not
cheap. Whatever we can do to decrease the amount of
raw storage we need the better. Lower storage usage
means faster replication and quicker backups. Have
you ever tried backing up 1TB of data in the cloud?
Spoiler alert: it's not easy.</p><p>-- =
Scottchiefbaker</p><p>P.S. For bonus points what if we re-worked what we
store? Do we need to store "Thank you for uploading
your work to CPAN..." Do we need to store the
opening boiler plate paragraph? <br>
</p>
<div> <br class=3D"webkit-block-placeholder">
</div>
<blockquote type=3D"cite">
<pre class=3D"border bg-light p-2 font-scp">From: =
metabase:user:314402c4-2aae-11df-837a-5e0a49663a4f
Subject: NA Random-Simple-0.24 5.10.1 FreeBSD
Date: 2025-03-31T17:20:02Z
This distribution has been tested as part of the CPAN Testers
project, supporting the Perl programming language. See
<a class=3D"moz-txt-link-freetext" href=3D"http://wiki.cpantesters.org/" =
moz-do-not-send=3D"true">http://wiki.cpantesters.org/</a> for more =
information or email
questions to <a class=3D"moz-txt-link-abbreviated moz-txt-link-freetext" =
href=3D"mailto:[email protected]" =
moz-do-not-send=3D"true">[email protected]</a>
</pre>
</blockquote>
P.P.S. Raw numbers for reference:
<div><br class=3D"webkit-block-placeholder">
</div>
<div> <br class=3D"webkit-block-placeholder">
</div>
<blockquote type=3D"cite"><font =
face=3D"monospace">perlmagpie>
SELECT avg(octet_length(txt_zstd)), count(guid),
grade FROM test_results INNER JOIN test USING
(GUID) GROUP BY grade ORDER BY 1 asc LIMIT 30; =
<br>
+-----------------------+-------+---------+ <br>
| avg =
&n=
bsp; | count | grade | <br>
|-----------------------+-------+---------| <br>
| 837.1807610993657505 | 1892 | NA =
| <br>
| 862.9752690411719781 | 72015 | PASS =
| <br>
| 1286.9555979297194225 | 3671 | UNKNOWN | =
<br>
| 1515.2728811352688452 | 15362 | FAIL =
| <br>
+-----------------------+-------+---------+ <br>
SELECT 4 <br>
Time: 0.223s<br>
<span style=3D"background-color: rgb(255, 255, =
255);"> </span><br>
</font></blockquote>
<div><br class=3D"webkit-block-placeholder">
</div><p><br>
</p>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
</div></blockquote></div><br></div></body></html>=
--Apple-Mail=_EB170F5E-E6C2-4288-A6CB-42D9B7972D45--