Libtool Roadmap

Alex Ameen <[email protected]> Sun, 10 Apr 2022 14:46:44 -0500

Newsgroups	gmane.comp.gnu.libtool.general
Message-ID	<[email protected]>

Howdy, a few weeks ago I talked about sending out a road-map.
I've gotten that prepared today. This roadmap is informal, opinionated,
and *I encourage feedback* and discussion on it. I have my own biases
and opinions about things I like/dislike and want added based on my
`libtool' usage over the years. However, I would be doing a disservice
to users if I forced them to adopt "my way of using `libtool'" over
their own. With that in mind it's important for y'all to share your own
use cases and pain points so that I can address them. This might be more
accurately called a "giant RFC dump" than a roadmap, because it's hard
for me to imagine that this remains unchanged.

The only notable item I have not included here is some organizational
changes concerning how bugs/patches are submitted ( currently there's 3
ways to submit bug reports and 2 of them are miserable to process ); but
I'll address that in another thread.

Also note that in my discussion of "CI" ( Continuous Integration )
setup, I intend to provide sufficient abstraction that with some
tailoring automated `libtool' testing can be accessible to other
developers. This will not be a "core" feature of `libtool', it is rather
a practical necessity for maintenance. I am not and will not be making
commitments on behalf of the `libtool' team to gate releases based on
regression for any particular combination of platforms or tools. Rather
you can look at this as "I want to improve the existing Hydra
infrastructure, make it more accessible, and find ways to support
platforms which cannot natively run Nix or Hydra", this may mean
creating more abstract job declarations in a format with can be
processed by a wider range of CI tools, it may mean VMs will be used for
some testing, it may mean that Hydra is dropped altogether ( unlikely ).
Time will tell.

These tasks are what I have in mind for the immediate future. Long term
I'd like to work more closely with the other Autotools teams to improve
integration with the rest of the family, and make `libtool' more
extensible, but those are conversations for another day.

------------------------------------------------------------------------

━━━━━━━━━━━━━━━━━
LIBTOOL-ROADMAP

Alex Ameen
━━━━━━━━━━━━━━━━━

Table of Contents
─────────────────

1. Roadmap
.. 1. Improve `.la' libraries, or phase them out on certain platforms.
..... 1. Libtool should not use `libfoo.la' when `libfoo.EXT' are requested.
..... 2. Make installation of `.lt' libraries optional.
..... 3. Prevent over-linking of ELF binaries.
..... 4. Security
.. 2. Support `$ORIGIN' in ELF binaries’ `RPATH' and `RUNPATH' entries.
.. 3. Avoid use of wrapper scripts for `noinst_' and `nodist_' binaries.
.. 4. Refactor argument parsing, particularly parsing linker flags.
.. 5. Migrate platform, arch, and tool specific conditionals to `autoconf'.
.. 6. Expand usage of `M4SH' for creating `ltmain.sh' and `libtoolize.in'.
.. 7. Simplify `bootstrap' process.
.. 8. Make the creation of test cases more user friendly
.. 9. Create CI pipeline resources for distribution and tool maintainers
.. 10. Permit installable LTLIBRARIES to conditionally be “convenience
libraries”

1 Roadmap
═════════

1.1 Improve `.la' libraries, or phase them out on certain platforms.
────────────────────────────────────────────────────────────────────

1.1.1 Libtool should not use `libfoo.la' when `libfoo.EXT' are requested.
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

What I mean is, if I say `-lfoo', of `-l:libfoo.la', then go right
ahead, do your thing `libtool'. However, when I went out of my way to
say `-l:libfoo.so' or `/trust/the/user/libfoo.a', it is evil for
`libtool' to silently decide that it knows best and decide to use
`libfoo.la' regardless.
• This behavior leads a large number of package managers and
distributions to delete `.la' files before distributing binary
tarballs.
• This frequently leads to incorrect/unexpected libraries and flags
being used because they were written as dependencies or inherited
flags in a `.la' file.

1.1.2 Make installation of `.lt' libraries optional.
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

• Closing Argument:
┌────
│ libtool: link: warning: library `/tmp/usr/lib/libstdc++.la'
was moved.
│ libtool: link: warning: library `/tmp/usr/lib/libstdc++.la'
was moved.
│ libtool: link: warning: library `/tmp/usr/lib/libstdc++.la'
was moved.
│ libtool: link: warning: library `/tmp/usr/lib/libstdc++.la'
was moved.
│ libtool: link: warning: library `/tmp/usr/lib/libstdc++.la'
was moved.
│ libtool: link: warning: library `/tmp/usr/lib/libstdc++.la'
was moved.
│ libtool: link: warning: library `/tmp/usr/lib/libstdc++.la'
was moved.
│ libtool: link: warning: library `/tmp/usr/lib/libstdc++.la'
was moved.
└────
Should people learn to use `DESTDIR'? Sure, but it’s hard enough to
defend the usefulness of `.la' when it has a meltdown like this over
binaries which were safely moved.

1.1.3 Prevent over-linking of ELF binaries.
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

Unless a popular patch for `ltmain.sh' was applied to a project,
`libtool' will over-link unneeded dependencies in ELF binaries. This
often leads to unexpected/incorrect library load/initialization
ordering, particularly in cases where libraries have circular
dependencies.
• “Libraries shouldn’t have circular dependencies” is not a valid
excuse.
• This has a measurable impact on performance in large applications,
and makes interposing symbols nearly impossible.
• This behavior severely aggravates C++ initialization ordering
issues.
• Makes the use of ELF filtering extensions impractical.

1.1.4 Security
╌╌╌╌╌╌╌╌╌╌╌╌╌╌

The ability to pass flags to consumers is useful, but this behavior
requires more visibility, and serious scrutiny concerning security.
• Expecting users to inspect `.la' files recursively, or always build
dry runs to audit flags is utterly fanciful.

1.2 Support `$ORIGIN' in ELF binaries’ `RPATH' and `RUNPATH' entries.
─────────────────────────────────────────────────────────────────────

1.3 Avoid use of wrapper scripts for `noinst_' and `nodist_' binaries.
──────────────────────────────────────────────────────────────────────

“When possible” of course. While these scripts mean well, they are
difficult to circumvent when a use case requires direct execution of a
binary. Further, the availability of `$ORIGIN' on ELF platforms makes
these wrappers unnecessary or counter-intuitive.

1.4 Refactor argument parsing, particularly parsing linker flags.
─────────────────────────────────────────────────────────────────

• It’s incomprehensible, and inline comments indicate that for several
years people have been unable to figure out why certain quirks like
“libraries are in reverse order in this block for some reason”
exist.
• When performing any changes to the parser test cases should be
created to ensure that behavior remains consistent ( unless a latent
bug is fixed ).

1.5 Migrate platform, arch, and tool specific conditionals to `autoconf'.
─────────────────────────────────────────────────────────────────────────

For example, if I configure `./configure --host_os='$(build_os)'', or
if a compiler/linker is known at configure time, much of the runtime
probing which is currently performed is redundant.
• Obviously, for a global `libtool' install on a system, this
optimization would not be performed. Rather this aims to optimize
`ltmain.sh --> libtool' creation performed in a project tree
alongside `autoconf'.

1.6 Expand usage of `M4SH' for creating `ltmain.sh' and `libtoolize.in'.
────────────────────────────────────────────────────────────────────────

In practice I aim to split these scripts into smaller parts which are
easier to read and reason about in isolation.
• More modular snippets may be easier to unit test.
• Use of `M4SH' may also permit users to patch/extend `libtool' more
easily.
• This will allow us to phase out `funclib.sh' over time, and align
idiomatically with the other `autotools'.

1.7 Simplify `bootstrap' process.
─────────────────────────────────

While the bootstrapping script is really more geared for maintainers
and contributors working in the repository, as opposed to the source
tarballs which users download, the existing process’ complexity and
dependence on external repositories complicates efforts to build
`libtool' using CI.
• Use of `git' and reliance on a network connection to vendor external
repositories during the bootstrap phase complicates conventional
hashing approaches used to cache CI artifacts.
• Requiring a network connection for bootstrapping raises security
concerns which would make automatic testing of patch submissions
unsafe.
• Ideally bootstrapping `libtool' should be possible using a minimal
tool-chain. Likely `coreutils' ( or equivalent ), `awk', `sed',
`grep', `tar', `cc', `binutils' ( or equivalent ), `make',
`autoconf', `automake', `m4', `findutils', `tar', and `xz'.
Dependence on `texinfo', `help2man', etc could be optional.
• Parts of `gnulib' which we require should be vendored. The existing
bootstrap process checks out a specific commit of `gnulib' and
`gl/bootstrap', so vendoring this code is functionally equivalent.

1.8 Make the creation of test cases more user friendly
──────────────────────────────────────────────────────

While the existing documentation recommends that folks submitting
issues create a test case which reproduces their problem, the
complexity of the test suite makes that impractical.
• The existing test suite is robust, and I have no intention of
replacing it, however creating an avenue for contributors/users to
draft simple `sh' scripts or minimal `autotools' projects for
reproducing issues should be as user friendly as possible.
• If creating test cases is not convenient, contributors and users
simply won’t do it, and the burden of authoring these tests will
fall to the `libtool' team. Often `libtool' team members do not
readily have access to the effected platform, or lack experience
with a given platform and tool-set’s quirks. A practical and
predictable, consequence is that certain platforms and tool-chains
lack robust testing.

1.9 Create CI pipeline resources for distribution and tool maintainers
──────────────────────────────────────────────────────────────────────

This would likely be placed in a branch or separate repository. The
goal here is that a declarative CI pipeline for testing `libtool' be
drafted such that members of the `libtool' team, or external projects
which want to integrate with `libtool' may self-host a set of CI jobs
used to build and test `libtool' with parameterized inputs.
• For example, if I am a `coreutils' contributor, it may be useful to
drop in an alpha build of `coreutils' to see if it blows up
`libtool'.
• Similarly, if I am a `libtool' contributor, it may be useful to
automatically test a matrix of `autoconf' and `automake' versions to
see if we have broken compatibility with a particular release of
those tools.
• Currently we often don’t hear about changes which broke a niche
combination of platform, arch, and tool-set combinations until after
a release is out, and people submit issues. While it’s not possible
to expand an infinite matrix of combinations, real time feedback for
systems other than a team member’s daily driver could catch a large
number of issues before they are released.

1.10 Permit installable LTLIBRARIES to conditionally be “convenience
libraries”
───────────────────────────────────────────────────────────────────────────────

This aims to support projects which produce different sets of
libraries when a static vs. shared library build is being run. Largely
this effects the handling of internal utility libraries with hidden
symbol visibility.
• For clarity “convenience” libraries are simply `PIC' `libfoo.a'
libraries, or equivalent for a given platform.
• This is particularly applicable to `noinst_PROGRAMS' used at build
time which depend on installable libraries. A `noinst_PROGRAMS'
executable would likely prefer linking with a static “convenience”
library rather than a shared library, since they generally do not
require a wrapper script.
• With a plain `Makefile' it is trivial to produce either a
`libfoo.so', static `libfoo.a', and `PIC' `libfoo.a' during a single
run. `libtool'’s concept of “convenience” libraries make it
difficult to replicate this workflow for arbitrary reasons.
• Ex: Project wants to produce `libfoo.la' and `libbar.la' to make
their public interfaces available to users. Both `libfoo.la' and
`libbar.la' depend on `libquux.la', which does not contain “user
facing” public interfaces. Project may wish to install `libfoo.so'
and `libbar.so' which link `PIC' `libquux.a' with `hidden' symbol
visibility when `-shared' is in effect, but may wish to install
`libfoo.a', `libbar.a', and `libquux.a' when `-static' is in effect.
• Ex: A project may wish to conditionally produce a `libquux.so', or
roll `libquux.a' into `libfoo.so' such that `libbar.so' may satisfy
its undefined references in for `libquux.la' through `libfoo.so'.
• Ex: When performing unit tests linking `PIC' `libfoo.a' rather than
`libfoo.so' into a test executable `mytest' allows `mytest' run
without a wrapper, so that it may be moved to and run in arbitrary
directories. Frankly if I’m authoring tests, the `libtool' wrapper
over executables may be more of a headache than a convenience.
• As niche as these use-cases may sound, there are variety of reasons
for a project to produce any of `libfoo.so', static `libfoo.a', and
`PIC' `libfoo.a' during a single build/install.
• To accomplish this in existing releases of `libtool', project
authors would be required to define `libquux.la' “twice”, once as a
“convenience” library, and again as an installable `LTLIBRARY' with
identical sources and flags. It might also be necessary for users to
configure and/or build twice to get the desired outputs.
My preference is “DRY”.