Re: using zstd compression - any particular build options?
"Dan Langille" <[email protected]>
| Newsgroups | gmane.comp.bacula.user |
|---|---|
| Message-ID | <[email protected]> |
On Thu, Jun 12, 2025, at 12:32 PM, Dan Langille wrote:
> On Thu, Jun 12, 2025, at 12:09 PM, Martin Simmons wrote:
>>>>>>> On Thu, 12 Jun 2025 09:45:20 -0400, Dan Langille said:
>>>
>>> On Thu, Jun 12, 2025, at 8:19 AM, Martin Simmons wrote:
>>> >>>>>> On Wed, 11 Jun 2025 12:38:27 -0400, Dan Langille said:
>>> >>
>>> >> On Wed, Jun 11, 2025, at 10:45 AM, Martin Simmons wrote:
>>> >> >>>>>> On Tue, 10 Jun 2025 20:23:40 -0400, Dan Langille said:
>>> >> >>
>>> >> >> I expected to see std mentioned...
>>> >> >>
>>> >> >> I tried building with --with-zstd in the build options - it did not change the above output.
>>> >> >>
>>> >> >> i.e
>>> >> >> --CONFIGURE_ARGS--
>>> >> >> --disable-conio --enable-batch-insert --enable-smartalloc --sysconfdir=/usr/local/etc/bacula --with-baseport=9101 --with-db-name=bacula --with-db-user=bacula --with-dump-email=root@localhost --with-job-email=root@localhost --with-logdir=/var/log --with-plugindir=/usr/local/lib --with-readline=/usr/local --with-sbin-perm=755 --with-scriptdir=/usr/local/share/bacula --with-tcp-wrappers=/usr/lib --with-working-dir=/var/db/bacula --with-zstd --with-fd-group=wheel --with-fd-user=root --enable-client-only --enable-nls --prefix=/usr/local ${_LATE_CONFIGURE_ARGS}
>>> >> >
>>> >> > It is a bug in the configure script.
>>> >> >
>>> >> > As a workaround, try installing the curl package as a build-time dependency
>>> >> > and add --with-curl=/usr/local to the configure args. AFAICS, this will not
>>> >> > change anything in the built binaries because Bacula doesn't actually use
>>> >> > curl, but it will work around the bug.
>>> >>
>>> >> I have been able to reproduce the bug reported in the ticket. I've been able to get compressin with GZIP and LTO, but not ZSTD.
>>> >>
>>> >> This is the FileSet I'm testing with:
>>> >>
>>> >> FileSet {
>>> >> Name = "basic backup with compression"
>>> >> Include {
>>> >> Options {
>>> >> signature=MD5
>>> >> compression=ZSTD
>>> >> }
>>> >> File = /boot
>>> >> File = /etc
>>> >> ....
>>> >>
>>> >> The build time changes I made were:
>>> >>
>>> >> [15:36 pkg01 dvl /usr/local/poudriere/ports/default/sysutils/bacula15-server] % git diff
>>> >> diff --git a/sysutils/bacula15-server/Makefile b/sysutils/bacula15-server/Makefile
>>> >> index 5ab79d4d4852..bad3f52db132 100644
>>> >> --- a/sysutils/bacula15-server/Makefile
>>> >> +++ b/sysutils/bacula15-server/Makefile
>>> >> @@ -1,6 +1,6 @@
>>> >> PORTNAME= bacula
>>> >> DISTVERSION= 15.0.2
>>> >> -PORTREVISION= 6
>>> >> +PORTREVISION= 6a
>>> >> CATEGORIES?= sysutils
>>> >> MASTER_SITES= SF/bacula/bacula/${DISTVERSION}
>>> >> PKGNAMEPREFIX?= #
>>> >> @@ -78,7 +78,8 @@ CONFIGURE_ARGS+= --disable-conio \
>>> >>
>>> >> .if defined(WITH_CLIENT_ONLY)
>>> >> CONFIGURE_ARGS+= --with-fd-group=wheel \
>>> >> - --with-fd-user=root
>>> >> + --with-fd-user=root \
>>> >> + --with-curl=/usr/local
>>> >> .else
>>> >> CONFIGURE_ARGS+= --with-dir-group=${BACULA_GROUP} \
>>> >> --with-dir-user=${BACULA_USER} \
>>> >> [15:48 pkg01 dvl /usr/local/poudriere/ports/default/sysutils/bacula15-server] %
>>> >>
>>> >> In my initial testing, I did not explicitly add ftp/curl as a run time dependency of the FreeBSD package. curl was already present on the host for other dependencies. With
>>> >> the failed results for "compression=ZSTD", I went back and built the package with the
>>> >> dependency - same results: no compression.
>>> >
>>> > If I remember correctly, poudriere builds in a jail so it will not see the
>>> > curl installed on the host.
>>> >
>>> > It looks like the underlying bug has was fixed in git revision
>>> > 358e838a99ad407e439a149a7dc9b472c63db2f8 (and
>>> > 8d9669fb26bd8bce30e4c48a522d9943c110b5c2 for the autoconf/configure.in). The
>>> > critical parts are the two new "unset error" lines so you could make a patch
>>> > from that.
>>>
>>> Thank you. I grabbed the two patches from:
>>>
>>> https://gitlab.bacula.org/bacula-community-edition/bacula-community/-/commit/358e838a
>>> https://gitlab.bacula.org/bacula-community-edition/bacula-community/-/commit/8d9669fb26bd8bce30e4c48a522d9943c110b5c2
>>>
>>> I added them to my copy of the FreeBSD port and rebuilt the package. By that, I mean the bacula-client package, not the server (i.e. only bacula-fd on the client has been updated in my testing).
>>>
>>> I'm assuming that only the client needs to be updated here, not that server. Am I correct there?
>>
>> bextract also uses it.
>
> but... we're not using bextract here are we?
>
>>> So far, testing with the above patches included is not compressing for 'compression=ZSTD'.
>>>
>>> For testing, I swapped back to 'compression=GZIP - it compressed.
>>
>> Is the poudriere build succeeding?
>
> yes
>
>> What is the output of ldd now?
>
> [16:28 gw01 dvl ~] % ldd /usr/local/sbin/bacula-fd
> /usr/local/sbin/bacula-fd:
> libintl.so.8 => /usr/local/lib/libintl.so.8 (0x32541a345000)
> libz.so.6 => /lib/libz.so.6 (0x32541a773000)
> libbacfind-15.0.2.so => /usr/local/lib/libbacfind-15.0.2.so (0x32541bbe7000)
> libbaccfg-15.0.2.so => /usr/local/lib/libbaccfg-15.0.2.so (0x32541b229000)
> libbac-15.0.2.so => /usr/local/lib/libbac-15.0.2.so (0x32541ca27000)
> libm.so.5 => /lib/libm.so.5 (0x32541d48b000)
> libthr.so.3 => /lib/libthr.so.3 (0x32541e0e4000)
> libwrap.so.6 => /usr/lib/libwrap.so.6 (0x32541fd8b000)
> libssl.so.30 => /usr/lib/libssl.so.30 (0x32541eaa6000)
> libcrypto.so.30 => /lib/libcrypto.so.30 (0x3254218d1000)
> liblzo2.so.2 => /usr/local/lib/liblzo2.so.2 (0x32541f6be000)
> libc++.so.1 => /lib/libc++.so.1 (0x32541ff1f000)
> libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x325420633000)
> libc.so.7 => /lib/libc.so.7 (0x3254234d8000)
> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x325420eda000)
> [vdso] (0x325419b0e000)
>
>> If it doesn't show zstd then what is the output of the poudriere build?
>
> It's here:
> https://services.unixathome.org/poudriere/build.html?mastername=142amd64-default-primary&build=2025-06-12_13h32m20s
>
> Click on success, under Log.
>
> I see
>
> ...
> checking zstd.h usability... no
> checking zstd.h presence... no
> checking for zstd.h... no
> ...
> ZSTD support: no
> ...
>
> Am I missing a dependency?
Yes, yes I am:
LIB_DEPENDS+= libzstd.so:archivers/zstd
With that:
[16:42 gw01 dvl ~] % ldd /usr/local/sbin/bacula-fd | grep std
libzstd.so.1 => /usr/local/lib/libzstd.so.1 (0x37b1178e1000)
and:
Software Compression: 71.6% 3.5:1
Success. : Thank you. Now I'll feed this into the FreeBSD ports tree.
--
Dan Langille
[email protected]