Re: Discussion about why GNU/Linux system upgrades cause old programs to break

Arsen Arsenović <[email protected]> Wed, 12 Aug 2026 19:06:07 +0200
Newsgroups	gmane.emacs.devel
Message-ID	<[email protected]>
"Dr. Arne Babenhauserheide" <[email protected]> writes:

> Arsen Arsenović <[email protected]> writes:
>
>> glibc, binutils and gcc releases are tested by various distros
>> rebuilding their software against them.
>
> This leaves out one part: distros often patch packages that do not
> build after an update, to make it possible to update.

Yes, and distro maintainers also generally forward such patches
upstream.  I know I do.

> But this does not help programs that aren’t packaged in the distro.

It does help them in two ways:

1. Dependencies that are packaged can raise errors or even receive fixes
2. Packaged programs very well could observe the same behavior you do

Distros are massive corpora of code.  The more testcases are present,
the more likely breakages are to be caught.

Obviously, it is not ever possible to prevent all breakages.  Sometimes
programs were already broken, and a change just revealed a way in which
they were broken.

I've given a few such examples, specifically of false assumptions being
made.

And sometimes, a breakage is just warranted.

This is because there are two classes of breakages:

1. Changes in actual guarantees.  This is _extremely_ rare in the
   toolchain and libc, but it does happen.  And it can often be
   completely justifiable.

   The case mentioned before falls into this category; there's no number
   of bytes to save ahead of time that's forever correct on an evolving
   CPU architecture like x86_64 or AArch64 (in reference to SIGSTKSZ),
   so it makes sense to make the stack size not a symbolic constant.
   (Though, note that it remains a symbolic constant if compiling in
   POSIX mode, because POSIX defines it that way at the moment)

   The removal of 'gets' is also.  It is simply not possible to use that
   function correctly.

2. Changes in observed but undocumented behaviour.
   The vast majority of 'breakages' fall into this category.  This
   category contains all of the programs I referred to as "broken" in
   prior emails.

   A very frequent example is reliance on transitive includes.  For
   instance, many libstdc++ headers use <cstdint>, and thus,
   accidentally expose std::uint16_t, uint16_t, etc.  So, programmers
   forget to include <cstdint> on their own.

   Such a program is broken; it relies on the false assumption that
   whatever headers it has included also provide the fixed-width integer
   types.  The same program would not compile C++ implementation uses
   that header less frequently.  And, indeed, it does not compile with a
   newer copy of libstdc++.

   This is another example of laxness being a problem.  More info below.

   Though, it is worth noting that an example like this is mostly
   harmless.  The reliance on, for instance, -fno-strict-aliasing, or
   integer wrapping semantics is a far greater problem, as that
   manifests as silent and subtle runtime bugs, rather than very easy to
   find and fix compile-time errors.

It is important to distinguish these cases.  What is to be done about
each differs.

The latter is almost always something that's better fixed in the
program, as that benefits users on systems other than GNU.

IMO, we do a great dis-service to our users by allowing the latter to
happen in the first place.

> They can also file bugs against the new libraries, compilers, etc, that
> cause unchanged programs to no longer compile. But this requires
> upstream to treat such breakage as problem that needs to be fixed.
>
> Where (in the spectrum between always patching programs and always
> ensuring compatibility from libraries/compilers/…) this work is done is
> a community decision. Though often not taken explicitly but by doing
> work where it seems to be done the fastest.
>
> Not an easy one, and not one that is always answered the same way, but
> still a decision.
>
> One that I’ve seen taken without considering unpackaged programs.
>
>>> You make it sound like a sin, but I happen to think it's a virtue.
>>> Backward compatibility is an important aspect of being user friendly
>>> in my book.
>
>> I'm sure many more hours have been spent tracking down stuff that
>> could've been prevented by stronger diagnostics than have been spent
>> trying to compile old code.
>
> I think here is a misconception: when an update to the compiler breaks
> my previously compiling code, I am usually not working actively on the
> project. It hits me while I have no time and energy to fix it. I
> updated the system, after all, not my program.

Again, there's no reason to expect programs to keep working indefinitely
while swapping their parts out carelessly.  Despite that, great effort
is taken to make exactly that situation work out.  That it sometimes
fails is not surprising.

I don't expect the hack-jobs I've written a long time ago to keep
working indefinitely.

Most, however, are yet to break, because I pay special attention to
following the rules of whatever language or API I'm using to the letter.

This is where the distinction between two classes of breakages I alluded
to above comes into play: programs that do not break rules of the
languages/libraries/... they rely on will *only* break in the cases
where 

There is no solution to this problem: for anything with sufficient
users, all behavior, intended or unintended, will be relied upon.  This
is why "bug for bug" compatibility is a thing.

And no software under active development can actually remain bug-for-bug
compatible with itself for long.

That said, GCC in particular is always released with a porting page
which elaborates breaks that might have occurred:

  https://gcc.gnu.org/gcc-16/porting_to.html
  https://gcc.gnu.org/gcc-15/porting_to.html
  https://gcc.gnu.org/gcc-14/porting_to.html
  ...

Generally, such porting takes relatively little effort.  It's almost
always just, like, a missing #include or such.

You've stated previously that you use Guix.  You can pin a particular
set of dependencies for a given set of Guix packages, and thus make it
impossible for it to ever change and, thus, break.  This can be done per
program you don't have time to maintain (I get it, believe me.. much of
my system are parts I put together a long time ago that I couldn't even
enumerate now..), so that reduces security concerns.

Hopefully that helps.

> And I usually cannot decide not to update the system, because security
> issues require me to stay up to date.
>
> And I may have many programs I wrote over the years. They work and I
> started to depend on them, but I no longer change them. They are
> complete.
>
> If breaking of backwards compatibility occurs often, the regular
> maintenance work required just to keep my setup working becomes
> overwhelming after some time.
>
> Example: I had to give up several Python projects I liked a lot because
> of such problems. And I think that that’s bad. I use Python a lot less
> due to that experience.

Sure, but that one is not comparable.

Python is far more relaxed about compatibility than glibc, GCC, etc,
because they provide large deprecation windows and the ability to
install multiple versions of Python.  I've had to deal with some
of those fallouts myself.

This only applies for the standard library, though.  The projects you
refer could've tripped up on some third-party break, such as some
unrelated dependency.  But, this could hardly be blamed on Python.

> Second example: it took a lot of communication work to keep Lilypond
> using Guile after Guile 2.0 broke the performance requirements Lilypond
> had. Continuing after the performance reached a simliar level again.
>
>> But, backwards compatibility in this instance means relaxing compiler
>> diagnostics, thus making the compiler less strict. This is universally
>> a disservice to users.
>
> In these absolute terms I consider this statement to be wrong.
>
> The alternative is not a compiler that can’t be strict, but one where
> strictness is opt-in.
>
> That turns a problem of breaking programs into a communication problem:
> teaching people to activate the stricter mode for new projects.
> (also see the later answer)

This is precisely how GCC was doing for decades (with -pedantic,
-Werror=...  and -pedantic-errors).  These decades are overwhelming,
rather than sufficient proof, that this approach is a dead-end.

This isn't surprising to me.  I see no reason to expect people to learn.

In fact, I've seen time and time again that people refuse to learn.

I've had to deal with people blatantly violating rules of the C language
and trying to convince me that those rules are nonexistent for years.

Laxness as default is a very bad idea.

What I refer to as "laxness" is somewhat abstract, but I hope easy to
explain: the gap between the diagnostics of the compiler and the rules
of the language/library/&c.  A stricter compiler is able to reject more
incorrect programs.

Unfortunately, the nature of a language like C means that even the
strictest C compiler cannot possibly reject many classes of incorrect
programs.

But, that does not mean we should not try.

As I said above, it is a great dis-service to our users to make our
compilers less strict by default, as it more easily allows them to write
code that's wrong and will break, not only when updating their copy of
GNU software, but when they attempt to use other systems.

To demonstrate this, let us consider a scenario:

The compiler is lax by default.

The programmer uses it and, in usual fashion, refuses to read the
documentation until necessary.

Thus, they write wrong code which doesn't get diagnosed.

The user then compiles this wrong code and gets a program that does not
work.

For the user, $?  = 0 after 'make', yet the program is subtly broken.

To me, this seems like a betrayal of both the user and the developer,
rather than a service to either.

Now, obviously, this is an incredibly idealist analysis.  It fails to
consider all pre-existing code.

So, let's expand.

Let's presume that the programmer and user are both dealing, in their
own separate ways, in a codebase that was produced in the above
scenario.

Now, let us presume that the compiler developers finally started
"strictening" the compiler.

This is, indeed, the State of the World, so to say, today.

Let us consider the developer and the user separately now:

- The developer updates their compiler.  Their code breaks.  They can
  look at the compiler manual (say, the porting_to pages linked above),
  and discover that they can make the compiler lax again, and how to fix
  the errors.

  In the best case scenario, they document this for older releases and
  fix their program for newer releases.  In worst case scenario, they
  make the compiler lax for all newer releases.
- The user updates their compiler and attempts to build the program.
  The incorrect program code is rejected.

  They may find the above-mentioned documentation, either from the
  programmer or from the compiler manual itself.

  They don't get an incorrect program out of the compiler, though.

The latter situation here seems at first glance like a bad outcome, but
I'd argue is the better outcome.

The user is prevented from being given a false sense of security, and a
subtly broken program.

>> Indeed, the case I showed in my last email with 'reallocarray' was
>> exactly such a case: the user of flex "benefited" from default backwards
>> compatibility, only to get a program that was slightly broken under
>> specific conditions.
>
> Those conditions only happen when usage changes. In the situation where
> the program is currently being used, it works correctly.

I am not sure what this means.  The program reduced out of flex
presented as an example is always incorrect, but probably only crashes
when the size of ints and pointers differs, and when malloc returns a
pointer >= 2<<31.

The fact that something works out by accident does not mean it's
correct.

>> we've been teaching students that "this code is not correct but the
>> compiler ignored it because old code exists, but you shouldn't ignore
>> it" (to very little effect).
>
> This means addressing a communication problem by a technical change
> that broke existing tools.

This means addressing a historic mistake of making the compiler lax, by
fixing that, actually.

I also don't agree with the idea that this is a "communication problem".

No amount of communication will solve this, as I said above.

> I understand the reasoning for the change, but seeing the collateral
> damage it does (most of which won’t be visible to the ones doing the
> change), I do not like it.

You seem not to have investigated the issue at hand.

The people pushing for the the change you're referring to are precisely
the people who worked to fixed the fallout.

For instance, here are the wiki and bug tracker entries we at Gentoo
used to coordinate our part of the fixing effort:

  https://wiki.gentoo.org/wiki/Modern_C_porting
  https://bugs.gentoo.org/870412
  https://bugs.gentoo.org/880545

... which is relevant because a few Gentoo developers, me included, were
pushing for this change.

(Note that two unrelated things were happening in parallel, so this may
 be a bit confusing.  C23 made some breaking changes in the committee;
 specifically unprototyped functions were made to be treated as
 functions with empty argument lists, as most developers would expect,
 and some identifiers were promoted to keywords.  In parallel, GCC 14
 and Clang started diagnosing incorrect C99+ code 20 years late.  I was,
 in this thread, only referring to the latter.  The former is a much
 simpler case to deal with, by simply passing -std=cXY where XY < 23).

Here's the analogue (I think?) for Fedora (linked on the wiki page
above):

  https://bugzilla.redhat.com/show_bug.cgi?id=2137509

... which is relevant because a few Fedora developers, including the
very developer who submitted the patch making the change, were pushing
for this change.

Other developers pitched in to help also.

So, no, the people dealing with the fallout were most definitely the
people doing the change.  That's why distro testing happens when the
compiler and libc change.

The collateral damage only exists because the compiler was lax with
diagnosing incorrect programs.

The worst of it was probably build system tests (e.g. in Autoconf),
which now started always failing instead of testing what they were
supposed to test.

But why were this test written in a way that's not correct in the first
place?

Clearly, because the compiler permitted it many years ago, and nobody
bothered to fix it, so it percolated into many build systems over the
years.

> Hyrum’s Law does not mean to give up on compatibility.

I explicitly stated that it does not.

> Instead it states that “changes to the system must maintain these
> performance characteristics to continue functioning for its consumers”

No it doesn't.  You seem to have extrapolated an example (and indeed an
undocumented performance characteristic is observable but undocumented
behaviour and thus a special case of the general rule) into a statement
of the rule.

> and “the implicit interface constrains your system design and
> evolution” -- https://www.hyrumslaw.com/ Not a license for breakage
> but a warning that compatibility goes deep.

I think it says more about the idea of communicating constraints than it
does about compatibility, personally.

People do not bother to learn the tools they are using.  They do not
bother to read the specifications they rely on.  They do not bother to
check the API documentation of functions they call.  They will try
something, and if it works out, ship it.  Whether it worked out by
accident or not is secondary to whether it worked out once.

The obvious implication of this is that there is an extremely high value
of preventing misuse.

Thus, breaking compatibility to prevent misuse can easily be worth it.
-- 
Arsen Arsenović
signature.asc (application/pgp-signature, 418 B)