Re: Discussion about why GNU/Linux system upgrades cause old programs to break

"Dr. Arne Babenhauserheide" <[email protected]> Wed, 12 Aug 2026 22:55:02 +0200
Newsgroups	gmane.emacs.devel
Message-ID	<[email protected]>
Arsen Arsenović <[email protected]> writes:

> "Dr. Arne Babenhauserheide" <[email protected]> writes:
>
>> Arsen Arsenović <[email protected]> writes:
>>
>>> glibc, binutils and gcc releases are tested by various distros
>>> rebuilding their software against them.
>>
>> This leaves out one part: distros often patch packages that do not
>> build after an update, to make it possible to update.
>
> Yes, and distro maintainers also generally forward such patches
> upstream.  I know I do.

For packaged programs (because it’s only visible for those in the distro).

>> But this does not help programs that aren’t packaged in the distro.
>
> It does help them in two ways:
>
> 1. Dependencies that are packaged can raise errors or even receive fixes
> 2. Packaged programs very well could observe the same behavior you do

But since the packaged programs get patched (see above), this does not
lead to a fix for the problem in unpackaged programs.

> And sometimes, a breakage is just warranted.

After having lost two projects due to such breakages, I disagree.

> 1. Changes in actual guarantees.  This is _extremely_ rare in the
>    toolchain and libc, but it does happen.  And it can often be

This is rare but extremely painful.

> 2. Changes in observed but undocumented behaviour.

For these you named Hyrum’s Law. But you forgot the part in the law
about avoiding such changes.

> IMO, we do a great dis-service to our users by allowing the latter to
> happen in the first place.

We *did* such a dis-service, but changing this is an even bigger
dis-service, because it breaks programs that used to work. Just because
*another* program updated.

It makes the computing environment as a whole more brittle.

That the original underspecified’ness is to blame does not change that
the one who changes it is the one who forces others to do work just to
keep their system working – often at times when they don’t have the
required time for that.

>> I think here is a misconception: when an update to the compiler breaks
>> my previously compiling code, I am usually not working actively on the
>> project. It hits me while I have no time and energy to fix it. I
>> updated the system, after all, not my program.
>
> Again, there's no reason to expect programs to keep working indefinitely
> while swapping their parts out carelessly. 

I think there is reason to fulfill that expectation.

Because incremental improvement depends on that: solve one problem. Then
solve the next. The previous problem stays solved, so you can build on
it.

If that expectation does not work out, the strength crumbles away that
in Free Software we can stand on the shoulders of giants.

The shoulders of giants then become an ever increasing maintenance
burden.

> There is no solution to this problem: for anything with sufficient
> users, all behavior, intended or unintended, will be relied upon.  This
> is why "bug for bug" compatibility is a thing.

Bug for bug is not what we’re talking about here. We’re talking about
„this caused problems so we broke all programs that relied on it“.

> You've stated previously that you use Guix.  You can pin a particular
> set of dependencies for a given set of Guix packages, and thus make it
> impossible for it to ever change and, thus, break.  

Yes. I do that. And it’s very useful. But there’s a point when it
breaks: when a new version of a library is required to continue working
with the rest of the system but the new version of the library broke
compatibility.

Security is the most frequent cause for that, but graphical libraries
are a close second (and will likely be a cause for a lot of pain in the
years to come).

>> Example: I had to give up several Python projects I liked a lot because
>> of such problems. And I think that that’s bad. I use Python a lot less
>> due to that experience.
>
> Sure, but that one is not comparable.

I’m bringing Python up here as a warning: it shows how much gets broken
when core projects get “more relaxed”.

> Python is far more relaxed about compatibility than glibc, GCC, etc,
> because they provide large deprecation windows and the ability to
> install multiple versions of Python.  I've had to deal with some
> of those fallouts myself.

I don’t think that’s the ordering in which the argument goes. Python
causes more breakage, so distributions have to provide multiple
versions, and Python 2 had to be maintained over a decade longer than
expected.

In 2024 the Linux foundation wrote:

> “there is an ongoing transition from Python 2 to Python 3”
> — page 5 of https://www.linuxfoundation.org/research/census-iii

The Python 3 breakage is a change that should never have been done and
every project should be careful not to get near that amount of breakage.

And that’s despite Python’s popularity, because that mainly persisted
due to getting lucky that Python 3 got taken up for machine learning.

> This only applies for the standard library, though.  The projects you
> refer could've tripped up on some third-party break, such as some
> unrelated dependency.  But, this could hardly be blamed on Python.

One tripped on a third party (PyKDE). The others tripped on Python 3.
That was just horrible and should be remembered as an action no one
should ever take again.

>> That turns a problem of breaking programs into a communication problem:
>> teaching people to activate the stricter mode for new projects.
>> (also see the later answer)
>
> This is precisely how GCC was doing for decades (with -pedantic,
> -Werror=...  and -pedantic-errors).  These decades are overwhelming,
> rather than sufficient proof, that this approach is a dead-end.

Decades of GCC being more and more successful and widespread.

LLVM only started to get traction against GCC, because Apple wanted a
non-copyleft compiler that they can proprietarize, so they funded its
development.

> As I said above, it is a great dis-service to our users to make our
> compilers less strict by default, as it more easily allows them to
> write

I have not seen anyone here say that we should change our compilers to
be less strict. Just that when they were already lax by default, that
should not be changed.

> Let's presume that the programmer and user are both dealing, in their
> own separate ways, in a codebase that was produced in the above
> scenario.
>
> Now, let us presume that the compiler developers finally started
> "strictening" the compiler.
…

> - The developer updates their compiler.  Their code breaks.  They can
…
> - The user updates their compiler and attempts to build the program.
>   The incorrect program code is rejected.
…
> The user is prevented from being given a false sense of security, and a
> subtly broken program.

The user had a working program before, though with bugs, and now they no
longer have a working program, so the tasks for which they depended on
the program are broken.

For the user this is a purely worse situation, because they already used
the program in ways that worked.

The part that this scenario misses is that the number of maintainers of
most free software projects is "less than one". If the compiler breaks
it and no distribution chips in with patches, it’s dead.

That having less than one maintainer is viable is actually a strength,
because it means that the sum of required maintenance work over all of
Free Software is pretty low. We can build upon the work of others and
step by step challenge proprietary programs with far more developers.

But it also means that breaking backwards compatibility can cause
massive damage.

>> Those conditions only happen when usage changes. In the situation where
>> the program is currently being used, it works correctly.
>
> I am not sure what this means.  The program reduced out of flex
> presented as an example is always incorrect, but probably only crashes
> when the size of ints and pointers differs, and when malloc returns a
> pointer >= 2<<31.
>
> The fact that something works out by accident does not mean it's
> correct.

See above, expanded a bit:

If a program doesn’t work correctly for some inputs, people will stop
using it for those inputs. So after a while all situations in which the
program is used will be situations where the problems are
inconsequential.

Fixing the program for those inputs eases the work for people, so adding
warnings is very valuable, if it is still actively maintained.

But breaking the program because it may be broken for some inputs breaks
it for all its current users for all inputs. If it is not actively
maintained, this destroys it for these users.

> So, no, the people dealing with the fallout were most definitely the
> people doing the change.  That's why distro testing happens when the
> compiler and libc change.

This only applies for the packaged programs. Which doesn’t apply to
compiling old versions of programs (they are no longer packaged).

To not be misunderstood: I am very, very happy that distributions
provide patches and test compatibility. I’ve only contributed minor
patches in that direction over the years and I’m really glad that you
did far more.

But I’ve been hit by broken compatibility again and again. Because my
programs weren’t popular enough to be packaged. Except for the one that
got unpackaged because PyKDE died because KDE broke compatibility
because Qt broke compatibility.

> The collateral damage only exists because the compiler was lax with
> diagnosing incorrect programs.

This is the blame game. I want to prevent harm, not cast blame.

This may sound flippant, but it is my actual goal.

I don’t want to blame you for a change that’s done. That ship is sailed.

I just want to see less breakage in the future. And I know that we won’t
have "no breakage", because some of this is outside our reach.

And I also know that GNU is doing a far better job at preserving
backwards compatibility than many other projects out there.

And I am deeply thankful for that.

>> Hyrum’s Law does not mean to give up on compatibility.
>
> I explicitly stated that it does not.
>
>> Instead it states that “changes to the system must maintain these
>> performance characteristics to continue functioning for its consumers”
>
> No it doesn't.  You seem to have extrapolated an example (and indeed an
> undocumented performance characteristic is observable but undocumented
> behaviour and thus a special case of the general rule) into a statement
> of the rule.

I quoted from the website of Hyrum’s Law.

>> and “the implicit interface constrains your system design and
>> evolution” -- https://www.hyrumslaw.com/ Not a license for breakage
>> but a warning that compatibility goes deep.
…
> The obvious implication of this is that there is an extremely high value
> of preventing misuse.
>
> Thus, breaking compatibility to prevent misuse can easily be worth it.

It can be. But usually it does far more damage than good, because the
ones who do the change cannot know who depends on compatibility.

They cannot, because they cannot know what kinds of programs are
built outside the package manager.

And I have yet to experience a case where it was worth it. Until now
every case that I experienced caused more damage than good. Including
projects I contribute to. The worst was that a backwards compatibility
breakage in Guile almost caused Lilypond to be thrown out of Debian a
decade later. The change seemed obvious and inconsequential on the
surface, but it broke a feature baked deeply into Lilypond, and it took
years to heal even some of the wounds that caused.

The damage it caused always was far bigger than expected.

That’s why I nowadays think that breaking backwards compatibility should
only ever be done if it truly cannot be avoided.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
https://www.draketo.de
signature.asc (application/pgp-signature, 1.1 KB)