Re: Compressing CPAN Testers test results for long-term archival storage
[email protected] (Doug Bell) Wed, 14 May 2025 08:57:11 -0500
| Newsgroups | perl.cpan.testers.discuss |
|---|---|
| Message-ID | <[email protected]> |
--Apple-Mail=_98E322A5-8045-431A-A5DD-73FF3928796D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On May 13, 2025, at 4:38=E2=80=AFPM, David Cantrell = <[email protected]> wrote: >=20 > On 05/05/2025 17:23, Scott Baker wrote: >=20 >> High speed database-grade cloud storage is not cheap. Whatever we can = do to decrease the amount of raw storage we need the better. Lower = storage usage means faster replication and quicker backups. Have you = ever tried backing up 1TB of data in the cloud? Spoiler alert: it's not = easy. >=20 > The initial sync is a pain, but after that it's tolerable, especially = if you can efficiently just send diffs eg with zfs send/recv. Yeah, that's how Zach Dysktra set up our MySQL database backups: The = primary MySQL wrote its binlogs to a ZFS volume, and the replica would = receive them and then use a ZFS snapshot to mark the backup point. Then = ZFS itself stored the diff between the snapshot point and the current = data. Long-term, I suspect that'll be what ends up happening w/ the Collector = system: ZFS really seems like the ideal solution for this, as it has = compression like we want, but also snapshotting and replication and = send/recv and etc... If there's anyone with some expertise to lend about = setting up some OpenZFS stuff, that'd be awesome, because I'm not that = up-to-date on it at the moment (I was still thinking it was pretty = unstable on Linux, but a cursory web search I just did makes it seem = like that is no longer the case). Doug Bell [email protected] --Apple-Mail=_98E322A5-8045-431A-A5DD-73FF3928796D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"content-type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"overflow-wrap: break-word; = -webkit-nbsp-mode: space; line-break: = after-white-space;"><br><div><blockquote type=3D"cite"><div>On May 13, = 2025, at 4:38=E2=80=AFPM, David Cantrell <[email protected]> = wrote:</div><br class=3D"Apple-interchange-newline"><div><div>On = 05/05/2025 17:23, Scott Baker wrote:<br><br><blockquote type=3D"cite">High= speed database-grade cloud storage is not cheap. Whatever we can do to = decrease the amount of raw storage we need the better. Lower storage = usage means faster replication and quicker backups. Have you ever tried = backing up 1TB of data in the cloud? Spoiler alert: it's not = easy.<br></blockquote><br>The initial sync is a pain, but after that = it's tolerable, especially if you can efficiently just send diffs eg = with zfs send/recv.</div></div></blockquote><br></div><div>Yeah, that's = how Zach Dysktra set up our MySQL database backups: The primary MySQL = wrote its binlogs to a ZFS volume, and the replica would receive them = and then use a ZFS snapshot to mark the backup point. Then ZFS itself = stored the diff between the snapshot point and the current = data.</div><div><br></div><div>Long-term, I suspect that'll be what ends = up happening w/ the Collector system: ZFS really seems like the ideal = solution for this, as it has compression like we want, but also = snapshotting and replication and send/recv and etc... If there's anyone = with some expertise to lend about setting up some OpenZFS stuff, that'd = be awesome, because I'm not that up-to-date on it at the moment (I was = still thinking it was pretty unstable on Linux, but a cursory web search = I just did makes it seem like that is no longer the = case).</div><div><br></div><div><br></div><div><div style=3D"caret-color: = rgb(0, 0, 0); color: rgb(0, 0, 0);"><div>Doug = Bell</div><div>[email protected]</div><div><br></div><br = class=3D"Apple-interchange-newline"></div></div></body></html>= --Apple-Mail=_98E322A5-8045-431A-A5DD-73FF3928796D--