Re: Standardizing NO_NETWORK and USE_SYSTEM_DEPS environment variables

On 2025-01-23 21:14, Micha=C5=82 G=C3=B3rny wrote:=0D
> Hello,=0D
> =0D
> As a packager, I often find it necessary to ensure two important aspects=
=0D
> of the build process:=0D
> =0D
> 1) that the build process itself doesn't access the Internet,=0D
> in particular that it doesn't download any files on its own,=0D
> =0D
> 2) that the build process uses system shared libraries and other=0D
> dependencies whenever possible, rather than vendored (or downloaded)=0D
> copies.=0D
> =0D
> Unfortunately, these requirements don't always align with what=0D
> the upstream considers best for their own use and what they consider=0D
> the best defaults for their users.  Fortunately, we are often able to=0D
> reach an agreement and get the options to adjust the build system=0D
> behavior.  However, these options are often defined per project=0D
> and they aren't necessarily consistent across different projects.=0D
> =0D
> Given the recent success of how the NO_COLOR variable [1] became a de-=0D
> facto standard, I was wondering if we could perhaps attempt to=0D
> standardize environment variables for these two aspects as well.=0D
> I suppose many other distribution packagers are facing the same=0D
> problems, so I think this mailing list is a good place to discuss this.=0D
> =0D
> What I'd like to propose are two environment variables:=0D
> =0D
> 1) NO_NETWORK -- if it's set to a non-empty value, it requests that=0D
> programs don't access the (TCP/IP) network.=0D
=0D
It may be better to be named NO_INTERNET. Network is a confusing word in Li=
nux =0D
world. They can refer to something from only Internet to any protocols in =
=0D
network subsystem, even include AF_NETLINK... I have been asked why my udev=
 is =0D
broken when my program is in a netns many times. Systemd also have to infor=
m =0D
this point in their document of PrivateNetwork. So use more limited word =0D
Internet to avoid this.=0D
=0D
> =0D
> 2) USE_SYSTEM_DEPS -- if it's set to a non-empty value, it requests that=
=0D
> the build system does not use any vendored dependency for which it=0D
> supports using a system version instead, and that it links to shared=0D
> libraries whenever possible.=0D
> =0D
> Some examples and thoughts below, followed by rationale.=0D
> =0D
> =0D
> For NO_NETWORK, my primary goal is to have build systems not issue=0D
> commands that fetch stuff from the Internet.  For example, if a build=0D
> system supports automatically fetching and building missing=0D
> dependencies, setting NO_NETWORK would imply that it would fail instead.=
=0D
> Technically, this could also be extended to tools like wget, effectively=
=0D
> blocking Internet access on multiple layers -- but that's not strictly=0D
> necessary.=0D
> =0D
> Another use case would be test suites with tests accessing remote=0D
> servers -- the variable could automatically cause them to be skipped. =0D
> It could be also used e.g. by pytest-socket plugin to automatically cut=0D
> the test suite from the Internet when loaded.=0D
> =0D
> I think it would also make sense to imply not accessing local network=0D
> services, in particular local system services.  An example of that are=0D
> test suites that connect to the local database daemons rather than=0D
> starting their own isolated copies.=0D
> =0D
> As for the rationale, my focus would be on security, =0D
> and reliability.  Package management systems in general implement=0D
> streamlined procedures for fetching resources, including verification of=
=0D
> authenticity, use of local mirrors, caching and so on.  Fetching=0D
> resources directly bypasses this.  This could expose information about=0D
> what is happening on a particular machine, cause unnecessary server=0D
> load, cause fees due to data plan use, cause failures due to shoddy=0D
> Internet connection -- similarly for tests.  I've written a more=0D
> detailed explanation once in the Gentoo devmanual [1].=0D
> =0D
> =0D
> For USE_SYSTEM_DEPS, the primary goal is to build against system=0D
> dependencies.  For example, some upstreams either prefer using vendored=0D
> dependencies or fallback to them when the system dependencies aren't=0D
> found.  However, in Gentoo we really do want stuff to use system=0D
> dependencies -- and if we miss to specify them appropriately, we'd=0D
> rather see an error than an implicit fallback to a vendored dependency.=0D
> So if USE_SYSTEM_DEPS is set, the build system should enable using=0D
> system dependencies whenever supported, and disable all possible=0D
> fallbacks to vendored dependencies.=0D
=0D
Some build system (e.g. Meson) have infrastructure of switch between system=
 =0D
library and vendor library, it's good, we only need to expose the switch vi=
a =0D
environment variable.=0D
=0D
But in many build system, especially in "modern" build system like Go, Carg=
o =0D
(Rust) and NPM (Node.js), they are not good on this infrastructure:=0D
=0D
Cargo users normally use feature gate to control whether use system library=
, but =0D
the gate name and the gate direction are not standardized, someone use =0D
'vendored-xxx' and some others use 'system-xxx', and in Cargo we can only =
=0D
control the package behavior we faced directly, not indirect dependencies. =
if we =0D
want to control the bahavior of dependencies, the only way is hoping ALL th=
e =0D
package author make a feature gate to pass this switch to its dependency.=0D
=0D
In NPM, the situation is even worse. NPM ecosystem prefer to bundle everyth=
ing. =0D
They have a --build-from-source in node-gyp, but not all package use it and=
 it =0D
only affect the library that will be load by nodejs. In fact, many nodejs =
=0D
packages, especially which have some web contents, may download a copy of =
=0D
chromium. The switch for it is not standardized and even not existed in som=
e =0D
projects.=0D
=0D
For Golang, it doesn't have any infrastructure for switch. Their library is=
 like =0D
an union: either use system library or use vendored version.=0D
=0D
So the first step may be to build a basic standardized way to use system =0D
resources (include link library and use some executable files) in these lan=
guage =0D
and build system's ecosystem.=0D
=0D
> =0D
> The wording proposed above tries to account for the special case when=0D
> the package does not support a system dependency version at all -- e.g.=0D
> when they are patching the vendored dependency.  It's not ideal, but I'd=
=0D
> like to avoid blocking a major improvement just because we can't get it=0D
> perfect.=0D
> =0D
> The rationale for avoiding vendored dependencies have been repeated many=
=0D
> times, we have one on Gentoo Wiki [2], Fedora has one as well [3].=0D
> The main focus is ensuring that our security team is able to address=0D
> security issues, but being able to fix bugs and simply avoiding=0D
> duplication is also helpful.=0D
> =0D
> I've included mention of shared libraries since some upstreams seem to=0D
> prefer static linking for similar reasons that they prefer vendoring. =0D
> While technically this could be considered a separate issue (and perhaps=
=0D
> deserving a separate flag), I don't think there really are distributions=
=0D
> who want unvendoring but not dynamic linking, so it's simpler to have=0D
> a single "distro packager mode" variable.  Again, it assumes that there=0D
> will be cases when dynamic linking isn't possible, particularly when=0D
> there are no shared libraries.=0D
> =0D
> =0D
> [1] https://devmanual.gentoo.org/ebuild-writing/functions/src_test/index.=
html#tests-that-require-network-or-service-access=0D
> [2] https://wiki.gentoo.org/wiki/Why_not_bundle_dependencies=0D
> [3] https://fedoraproject.org/wiki/Bundled_Libraries=0D
> =0D