Re: Statically building when curl is present

On 15/06/2023 15:09, John Goerzen wrote:
> Hi Denis,

Hi John,

> 
> I'm the new Debian maintainer for dar.  I got libthreadar accepted into
> Debian, so now I am working to enable the full remote featureset
> including curl support.

Great news! Thanks a lot :)

> 
> I am running into difficulties with dar_static.  Particularly, it looks
> like dar_static is trying to link with curl, and this is problematic.
> 
> I get these messages:
> 
> /bin/bash ../../libtool  --tag=CXX   --mode=link g++  -g -O2 -ffile-prefix-map=/home/jgoerzen/work/dar=. -fstack-protector-strong -Wformat
> -Werror=format-security -all-static  -Wl,-z,relro -Wl,-z,now -o dar_static command_line.o config_file.o dar.o dar_suite.o hide_file.o no_co
> mment.o line_tools.o crit_action_cmd_line.o ../libdar/libdar64.la -lcurl  -L/usr/lib/x86_64-linux-gnu -lgpgme -lassuan -lthreadar -lpthread
>     -largon2 -lpthread -lrsync -lgcrypt -lgpg-error -llz4 -lzstd -llzma -llzo2 -lbz2 -lz -ldl -lcap
> libtool: link: g++ -g -O2 -ffile-prefix-map=/home/jgoerzen/work/dar=. -fstack-protector-strong -Wformat -Werror=format-security -static -Wl
> ,-z -Wl,relro -Wl,-z -Wl,now -o dar_static command_line.o config_file.o dar.o dar_suite.o hide_file.o no_comment.o line_tools.o crit_action
> _cmd_line.o  ../libdar/.libs/libdar64.a -lcurl -L/usr/lib/x86_64-linux-gnu -lgpgme -lassuan -lthreadar -largon2 -lpthread -lrsync -lgcrypt
> -lgpg-error -llz4 -lzstd -llzma -llzo2 -lbz2 -lz -ldl -lcap
> /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libgpg-error.a(libgpg_error_la-sysutils.o): in function `_gpgrt_getpwdir':
> (.text+0x2ca): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version
> used for linking
> /usr/bin/ld: (.text+0x2e8): warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the g
> libc version used for linking

this comes from glibc, this is just a warning and has for consequence a 
duplication of glibc in memory when running the statically linked 
executable. For that reason, when providing dar_static binaries (which 
are distro independents), I got to replaced glibc by "musl", I think 
this was the during investigation where we probably hit a bug in glibc 
see the discussion with Martin Michel having this subject
"Problems with remote repository access via sftp" in September - 
November 2022.

https://sourceforge.net/p/dar/mailman/dar-support/?viewmonth=202209

Unfortunately Debian only relies on glibc AFAIK, so you will not be able 
to get ride of these warning. But for the rest, what you could check is 
the contorsions I had to do with dar dependencies, in particular libcurl 
to have fully featured dar_static without pulling too much of 
dependencies below libcurl.

I kept trace of this work asa script you can find in source code:
misc/dar_static_builder_with_musl_voidlinux.bash

I'm using it to provide any new dar_static binary (outside distros).

> /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libcurl.a(libcurl_la-netrc.o): in function `Curl_parsenetrc':
> (.text+0x79c): warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
> /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libcurl.a(libcurl_la-curl_addrinfo.o): in function `Curl_getaddrinfo_ex':
> (.text+0x6f): warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
> /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libcurl.a(libcurl_la-psl.o): in function `Curl_psl_destroy':
> (.text+0x21): undefined reference to `psl_free'
> /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libcurl.a(libcurl_la-psl.o): in function `Curl_psl_use':
> (.text+0xbc): undefined reference to `psl_latest'
> /usr/bin/ld: (.text+0x157): undefined reference to `psl_builtin'
> /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libcurl.a(libcurl_la-version.o): in function `curl_version':
> (.text+0x129): undefined reference to `BrotliDecoderVersion'

Yes, I remember this. Hopefully libcurl supports alternatives. This 
script should give you some hints like relying on gnutls rather openssl 
which has a much more dependencies, from which the BrotliDecoderVersion 
and similar symbols are expected...

> 
> 
> ... and this goes on for pages and pages.
> 
> Investigating a bit, I see there's a curl-config command.  I note the
> difference:
> 
> jgoerzen@hephaestus:~/work/dar$ curl-config --static-libs
> -Wl,-Bstatic -lcurl -Wl,-Bdynamic -lnghttp2 -lidn2 -lrtmp -lssh2 -lssh2 -lpsl -lssl -lcrypto -lssl -lcrypto -lgssapi_krb5 -llber -lldap -llber -lzstd -lbrotlidec -lz
> jgoerzen@hephaestus:~/work/dar$ curl-config --libs
> -lcurl
> 
> The static .a files won't automatically pull in their dependencies.
> However, I can't just add `curl-config --static-libs` to the g++ call
> because everything after the -Wl,-Bdynamic will cause an error.
> Removing the -Wl,-Bdynamic, and installing a bunch of other dev packages
> gets close, but:
> 
> jgoerzen@sid:~/work/dar/src/dar_suite$ g++ -g -O2 -ffile-prefix-map=/home/jgoerzen/work/dar=. -fstack-protector-strong -Wformat -Werror=format-security -static -Wl,-z -Wl,relro -Wl,-z -Wl,now -o dar_static command_line.o config_file.o dar.o dar_suite.o hide_file.o no_comment.o line_tools.o crit_action_cmd_line.o  ../libdar/.libs/libdar64.a -lcurl -L/usr/lib/x86_64-linux-gnu -lgpgme -lassuan -lthreadar -largon2 -lpthread -lrsync -lgcrypt -lgpg-error -llz4 -lzstd -llzma -llzo2 -lbz2 -lz -ldl -lcap -lssl -lcrypto -lnghttp2 -lidn2 -lrtmp -lssh2 -lssh2 -lpsl -lssl -lcrypto -lssl -lcrypto -lgssapi_krb5 -llber -lldap -llber -lzstd -lbrotlidec
> /usr/bin/ld: cannot find -lgssapi_krb5: No such file or directory
> collect2: error: ld returned 1 exit status

so you miss a static libgssapi_krb5 (kerberos) or miss linking with it, 
which library dar does not use by the way... but which is very probably 
needed by a dependency below openssl I guess.

> 
> And here I run into a problem because libgssapi_krb5 is not available in
> a static (.a) version.

welcome to the team... this is why in the previously mentionned script 
(OK for Void linux not Debian, but the point is the same), you'll see 
that inspite existing packages, the scripts builds from source some 
static flavor of some library (I don't discuss the reason, there is pro 
and cons for distro point of view).


> 
> I have a question at this point: how did others manage to get this to
> build statically with curl? 

I guess they rely on what I provide at
https://dar.edrusb.org/dar.linux.free.fr/Releases/Dar_static/

lol

which is build using the mentionned script

> My best guess is that maybe they're on
> platforms where curl itself was compiled with fewer features.

Well, if you build a static binary/library you can temporarily build 
dependencies that even may be of a different version than the 
corresponding packaged dynamically libraries available on the distro, at 
the end of the process you get the static binary (which you can 
eventually strip to reduce its size), then the rest goes to trash.

> 
> I'm also concerned about the warnings about requiring shared libraries
> for getpwnam() and such.

this is glibc specific, you won't be able to get ride of this... 
unfortunately unless replacing glibc by another standard C library like 
musl, but for Debian this will be I guess very complex, as you will have 
to recompile almost any library dar direclty or indirectly depends upon...

> 
> Anyhow, then the question becomes: how to resolve this?
> 
> As far as I can tell, the only solution is to not link dar_static with
> curl.  I don't see that option in configure, though; it's all or
> nothing with that one.

this is an option. But try building a static flavor of libcurl based on 
gnutls rather openssh (see the script I mentionned) it should be a bit 
more simple I guess...

Maybe also I could be good to ask some Debian Guru, if this is 
acceptable in regard the Debian policy, to have inside dar_static a 
gnutls library compiled with slightly different options gnutls vs 
openssl, but using the same source code, same version as the official one...

> 
> BTW on a totally different topic, just wanted to verify you saw
> https://github.com/Edrusb/DAR/issues/50

Yes, and I think I have answered you by email... let me double check... 
I will update the github issue... :)

> 
> Thanks!
> 
> - John
> 

Thanks to you!
Denis