Re: Discussion about why GNU/Linux system upgrades cause old programs to break

"Dr. Arne Babenhauserheide" <[email protected]> Fri, 14 Aug 2026 11:09:13 +0200
Newsgroups	gmane.emacs.devel
Message-ID	<[email protected]>
Arsen Arsenović <[email protected]> writes:

(I’m trying to slim this down as much as possible, because it’s quite
 long. Please excuse it if sometimes only partial context is included)

> Distro testing catches unintended breaks and more often than not
> prevents them.

Ah, you meant unintended breakage that is fixed in the library, not by
patching the package where the breakage shows up.

There I fully agree.

> Of course, packaged software is only a sample of the collection of all
> relevant software, but it can produce useful information on the real
> effects of a change regardless.

There’s a current example of a breakage where that completely failed to
show up in distro-testing. It is currently hitting a program I maintain:
OpenSUSE changed from update-alternatives to libalternative and that
causes the Tanuki Wrapper to break:

https://bugzilla.opensuse.org/show_bug.cgi?id=1271556

They wrote “I needed to fix about 10 of them using trivial fixes”, but
they didn’t have a single instance of a program using the Tanuki
Wrapper, because the wrapper is usually excluded when packaging (it’s
not needed in that case).

This broke minecraft and hyphanet and who knows how many others.
(the point is: no one knows)

But that’s really the worst case.

The usual case would be those 10 they fixed “using trivial fixes”,
because by their own statistics that makes it probable that their change
broke 1% of all Java programs that are not packaged.

And they don’t know how many of those there are. But still broke them.

>>> 2. Changes in observed but undocumented behaviour.
> There's no avoiding such changes in software that is not frozen.

No avoiding completely, but minimizing. Because the cost of such changes
is almost always higher than the developers of the software expect.

> That's why the only solution to this problem ever found is exactly to
> freeze software.
>
> And, even then, the breakages *still* happen.  Because you sometimes
> have to backport a fix to a frozen release.

Also you often actually need the features from new releases. What can
help is to consider released functionality as frozen as long as that’s
possible.

And to minimize the damage caused by breakages that can’t be avoided.
(I’m happy that at the end of this text we actually get to practical
 questions how that can be done)

>> dis-service, because it breaks programs that used to work.
>
> I disagree, chances that a program that broke as a result of the GCC 14
> changes was correct is very low.

This looks like one part of the core of the discussion here:

Chances that the program was wholly correct are not the same as chances
that the program worked correctly where people actually used it.

> On the other hand, if it was correct, you merely have to pass
> -fpermissive or -std=c89, and it works again.  As the release notes
> state.

That’s something the developer of a maintained program can do. It does
not help the majority of unpackaged FLOSS programs, because they have
"less than one" maintainer.

For packaged ones, it helps, because the distributions chip in.

>> It makes the computing environment as a whole more brittle.
>
> No, the laxness did that.

No: the computing environment worked reliably before. Breaking backwards
compatibility caused parts of the environment to stop working. That’s
why it makes it brittle.

The laxness likely introduced annoying special cases that had to be
worked around, so the tools around the software accumulated annoying
cruft.

These are different problems and there’s a tradeoff between the two. But
the cost for breaking backwards compatibility is usually underestimated
because most of the that cost isn’t visible to those who cause the
breakage.

The cost of cruft however is always visible to everyone working with it.

At least that’s my explanation why such breakages occurs again and
again, because I don’t want to believe that people would knowingly put
onto others many times the work they save.

> No, this wasn't originally underspecified.
>
> GCC chose to implement what you suggested - laxness by default,
> strictness as an opt-in - and we paid the piper for it.

That’s what the Python people said about Python 2, too. „there are
warts, that finally need to go“. It’s what almost everyone says who
breaks backwards compatibility: this breakage will make everything
better in the future.

But once down that path there’s always another breakage to which the
same logic is applied, usually before all the fallout from the previous
one is cleaned up.

(this is not targeted at you: it applies to non-GNU software *far more*
 than to GNU software, so please don’t take it as personal criticism.
 It’s part of my frustration with the software development community now
 that I’m in the position where I have many programs I wrote in the past
 and need to keep working)

> The specification tossed out this class of incorrect programs a very
> long time ago.  (And, yes, with that, some correct programs went, but

Are you saying that "underspecified" doesn’t apply, because the
specification changed earlier than GCC?

>> Because incremental improvement depends on that: solve one problem. Then
>> solve the next. The previous problem stays solved, so you can build on
>> it.
>>
>> If that expectation does not work out, the strength crumbles away that
>> in Free Software we can stand on the shoulders of giants.
>
> Luckily that didn't happen.

Not here. But for example xdotool is dead on most distros by now. I
actually depend on that.

But to make it clear: xdotool breaking causes vastly more fallout than
old code not compiling anymore without changes.

>> The shoulders of giants then become an ever increasing maintenance
>> burden.
>
> Every line of code is a maintenance burden.  Lines of code that cannot
> be verified statically are even greater maintenance burdens.

The maintenance burden doesn’t appear out of nowhere but to a good
degree comes from breakage due to changes in libraries and such.

There is a maintenance burden from undiscovered bugs (when discovered
they must be fixed). Strictness can help there to some degree. But it
doesn’t help with the maintenance burden from changes within the
shoulders of giants.

>>> There is no solution to this problem: for anything with sufficient
>>> users, all behavior, intended or unintended, will be relied upon.  This
>>> is why "bug for bug" compatibility is a thing.
>>
>> Bug for bug is not what we’re talking about here. We’re talking about
>> „this caused problems so we broke all programs that relied on it“.
>
> Thee are two remarks to add here:
>
> 1. The class of programs that accidentally used these "features" that
>    were made errors by GCC 14, and thus had silent bugs introduced, is
>    vastly bigger than the class of programs that used them intentionally
>    and correctly.
> 2. The compiler was refusing to diagnose errors *even when the standard
>    dialect was specifically asked for*.

Do you mean it diagnosed them as warnings instead of errors?

And people ignored those warnings, so they had unexpected behavior?

>> I don’t think that’s the ordering in which the argument goes. Python
>> causes more breakage, so distributions have to provide multiple
>> versions, and Python 2 had to be maintained over a decade longer than
>> expected.
>
> I haven't done the archival research to say one way or the other.  But
> it is clear that the ability to install multiple versions permits making
> breaking changes more easily.  I am sure I am not the first to conclude
> this, and I'm sure that this knowledge informs CPython maintainers.

Gentoo went through a lot of work to provide both python2 and python3.
That was when I still used Gentoo.

And even today I still stumble over scripts that are broken on some
systems, because the distros don’t agree whether python2 or python3
should be /usr/bin/python, an some don’t ship /usr/bin/python3 and
/usr/bin/python2, but only specific versions of python 3 (e.g.
python3.14 and /usr/bin/python but no /usr/bin/python3).

>> And that’s despite Python’s popularity, because that mainly persisted
>> due to getting lucky that Python 3 got taken up for machine learning.
>
> Is it?  I doubt that.

To the best of my knowledge it is.

>> Decades of GCC being more and more successful and widespread.
>
> Indeed.  That success and wide-spread use is why I am confident that
> these decades are overwhelming evidence that opt-in strictness via
> -pedantic utterly failed.
>
> Imagine how much better the software ecosystem would've been if all of
> that success and spread was accompanied with better diagnostics.

Do you limit “better diagnostics” here to “failing to compile”?

> GCC certainly didn't catch on solely (in fact, I imagine this could only
> be a point against it) because it failed to diagnose basic issues by
> default.

I think requiring "solely" isn’t logical here.

>> Just that when they were already lax by default, that should not be
>> changed.
>
> On this specific point, the one that changed in GCC 14, the vast
> majority of instances were *not* intentional usages of K&R C syntax.  In
> most of these cases, they were serious bugs waiting to explode in
> someones' face.

At work we use warnings and linters to get rid of such problems
gradually and only make them build breaking when no single usage
remains.

The latter cannot be done in an open environment, because you can never
know whether everything is fixed (you cannot know all programs).

>>> The user is prevented from being given a false sense of security, and a
>>> subtly broken program.
>>
>> The user had a working program before, though with bugs, and now they
>> no longer have a working program, so the tasks for which they depended
>> on the program are broken.
>
> Had it not been this loud change, it'd have been some subtle change that
> leads to a breakage.  One that's hard to detect, replicate, track down,
> and report.

That assumes that it is inevitable that a breakage would have occured
and this way it only happened earlier.

> As I said, the vast majority of instances of these errors were not
> innocuous, let alone intentional.

I do not doubt that.

Ideal would be (ignoring how hard it could be to implement) if it were
possible to distinguish between currently unmaintained, maintained, and
new programs:

- Maintained ones should nag the maintainer to activate stricter mode
  (and then keep it active) as soon as they have the headspace for that.

- For new programs it should be strict by default.

- For unmaintained programs it should stay lax (while warning, in the hope
  that it may become maintained again at some point).

>> That having less than one maintainer is viable is actually a strength,
>> because it means that the sum of required maintenance work over all of
>> Free Software is pretty low. We can build upon the work of others and
>> step by step challenge proprietary programs with far more developers.
>
> And the compiler being stricter makes that more viable.

Only for those who remain after the breakage.

Every time such a change happens, some programs drop out of the funnel,
so fewer programs remain. And if even one that dropped out was needed by
another one, that other one drops out, too.

> No, it doesn't.  You do not need to recompile a program before each use.
> Indeed, even more care is taken to keep programs in binary form working.
>
> I can't recall when the last serious ABI break in glibc or GCC happened.

I remember one. It broke my Gentoo because a compile failed and I had to
do emergency fixes with file system operations from dolphin because that
still had the old glibc loaded.

But that was a long time ago.

> Gentoo used to advise users to rebuild their systems on a GCC update,
> for instance.  We've gotten rid of that advice a while ago.

That’s great!

>> This is the blame game. I want to prevent harm, not cast blame.
>
> As the amount of code in the world grows, every failure to implement
> something to prevent a bug (harm) will manifest as many future bugs (a
> lot of harm).

When following that argument, it will always be used for every breakage
so there will be an eternal state of programs breaking.

(again: GNU isn’t the main cause here)

>> I quoted from the website of Hyrum’s Law.
>
> Right.  That was an example.  Not the statement of the law.

It is in the explanation of the law by its author.

> Also, FWIW, anecdotally, I've asked around and could only collect
> accounts of about two or three dozen people who strongly disliked this
> change, despite me being present in GCC support rooms and lists and
> reading all bug reports that reach the GCC bugzilla.

Two or three dozen people who publicly disliked a change towards more
strictness is a lot.

> Maybe I would've even supported e.g. the compiler saying "if you're
> compiling old code, you may need '-fpermissive'" or such.

I think that’s a question for Eli: would this have helped?

Maybe something more descriptive: “abc.c::xyz ERROR: <description>. The
majority of usages of this style causes subtle errors on modern systems.
For a clean fix, see <link to documentation>. To compile anyway, use
-fpermissive. To silence this message, add /* silence-warning: xzy */ to
the source just above the line”.

(even better would be something like "for a suggested fix, try to use
gcc --autofix=<identifier> abc.c" -- this is what helps a lot with
getting people to address linter warnings at work)

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
https://www.draketo.de
signature.asc (application/pgp-signature, 1.1 KB)