Re: Discussion about why GNU/Linux system upgrades cause old programs to break

"Dr. Arne Babenhauserheide" <[email protected]> writes:

> Arsen Arsenović <[email protected]> writes:
>
>> "Dr. Arne Babenhauserheide" <[email protected]> writes:
>>
>>> Arsen Arsenović <[email protected]> writes:
>>>
>>>> glibc, binutils and gcc releases are tested by various distros
>>>> rebuilding their software against them.
>>>
>>> This leaves out one part: distros often patch packages that do not
>>> build after an update, to make it possible to update.
>>
>> Yes, and distro maintainers also generally forward such patches
>> upstream.  I know I do.
>
> For packaged programs (because it’s only visible for those in the distro).

Naturally.  Just like I can't test what I don't know exists, I don't
patch what I don't know exists.

>>> But this does not help programs that aren’t packaged in the distro.
>>
>> It does help them in two ways:
>>
>> 1. Dependencies that are packaged can raise errors or even receive fixes
>> 2. Packaged programs very well could observe the same behavior you do
>
> But since the packaged programs get patched (see above), this does not
> lead to a fix for the problem in unpackaged programs.

I'm not sure how this is meant to contradict what I said.

Distro testing catches unintended breaks and more often than not
prevents them.  If the same unintended break that would break some
unpackaged program, would also break a packaged program, the distro
testing will prevent that breakage, this this also applies for the
unpackaged program.

There are breakages, and there are breakages.  I've separated out the
classes earlier.  The above applies for unintentional breakages.

>> And sometimes, a breakage is just warranted.
>
> After having lost two projects due to such breakages, I disagree.

This cannot be judged in the abstract.

That there was breakage, alone, does not let us classify what break
happened, whether it was avoidable, and whether the bug that lead to it
was the break itself, or something that was revealed by a later update.

So, share the breakage.

Someone might even fix them.

>> 1. Changes in actual guarantees.  This is _extremely_ rare in the
>>    toolchain and libc, but it does happen.  And it can often be
>
> This is rare but extremely painful.

Usually it's fairly minor.

For instance, in general, changes in widely used APIs result in strong
negative feedback through the distro testing process mentioned above.

Of course, packaged software is only a sample of the collection of all
relevant software, but it can produce useful information on the real
effects of a change regardless.

>> 2. Changes in observed but undocumented behaviour.
>
> For these you named Hyrum’s Law. But you forgot the part in the law
> about avoiding such changes.

What part?

There's no avoiding such changes in software that is not frozen.

That's why the only solution to this problem ever found is exactly to
freeze software.

And, even then, the breakages *still* happen.  Because you sometimes
have to backport a fix to a frozen release.

>> IMO, we do a great dis-service to our users by allowing the latter to
>> happen in the first place.
>
> We *did* such a dis-service, but changing this is an even bigger
> dis-service, because it breaks programs that used to work. Just
> because *another* program updated.

I disagree, chances that a program that broke as a result of the GCC 14
changes was correct is very low.

On the other hand, if it was correct, you merely have to pass
-fpermissive or -std=c89, and it works again.  As the release notes
state.

> It makes the computing environment as a whole more brittle.

No, the laxness did that.

> That the original underspecified’ness is to blame does not change that
> the one who changes it is the one who forces others to do work just to
> keep their system working – often at times when they don’t have the
> required time for that.

No, this wasn't originally underspecified.

GCC chose to implement what you suggested - laxness by default,
strictness as an opt-in - and we paid the piper for it.

The specification tossed out this class of incorrect programs a very
long time ago.  (And, yes, with that, some correct programs went, but
migrating to the prototype syntax isn't very difficult)

>>> I think here is a misconception: when an update to the compiler breaks
>>> my previously compiling code, I am usually not working actively on the
>>> project. It hits me while I have no time and energy to fix it. I
>>> updated the system, after all, not my program.
>>
>> Again, there's no reason to expect programs to keep working indefinitely
>> while swapping their parts out carelessly. 
>
> I think there is reason to fulfill that expectation.

I agree.  And so does everyone else participating in maintenance of the
tools and libraries in question.

Yet it's impossible.

No program will remain functioning forever unless undisturbed.

That's the trick behind every long-term support thing ever: not touching
it.

> Because incremental improvement depends on that: solve one problem. Then
> solve the next. The previous problem stays solved, so you can build on
> it.
>
> If that expectation does not work out, the strength crumbles away that
> in Free Software we can stand on the shoulders of giants.

Luckily that didn't happen.

> The shoulders of giants then become an ever increasing maintenance
> burden.

Every line of code is a maintenance burden.  Lines of code that cannot
be verified statically are even greater maintenance burdens.

>> There is no solution to this problem: for anything with sufficient
>> users, all behavior, intended or unintended, will be relied upon.  This
>> is why "bug for bug" compatibility is a thing.
>
> Bug for bug is not what we’re talking about here. We’re talking about
> „this caused problems so we broke all programs that relied on it“.

Thee are two remarks to add here:

1. The class of programs that accidentally used these "features" that
   were made errors by GCC 14, and thus had silent bugs introduced, is
   vastly bigger than the class of programs that used them intentionally
   and correctly.
2. The compiler was refusing to diagnose errors *even when the standard
   dialect was specifically asked for*.

>> You've stated previously that you use Guix.  You can pin a particular
>> set of dependencies for a given set of Guix packages, and thus make it
>> impossible for it to ever change and, thus, break.  
>
> Yes. I do that. And it’s very useful. But there’s a point when it
> breaks: when a new version of a library is required to continue working
> with the rest of the system but the new version of the library broke
> compatibility.
>
> Security is the most frequent cause for that, but graphical libraries
> are a close second (and will likely be a cause for a lot of pain in the
> years to come).
>
>>> Example: I had to give up several Python projects I liked a lot because
>>> of such problems. And I think that that’s bad. I use Python a lot less
>>> due to that experience.
>>
>> Sure, but that one is not comparable.
>
> I’m bringing Python up here as a warning: it shows how much gets broken
> when core projects get “more relaxed”.

They didn't, though.

The two changes brought up so far in the core projects, GCC 14
diagnosing K&R C constructs when K&R C was not used, and the SIGSTKSZ,
were not even close to some of the changes that happen in core Python.

As I covered in another post, SIGSTKSZ simply is not a constant in
reality anymore, so if the value was to be bumped (which was suggested,
and AFAIK done for the case when !_DYNAMIC_STACK_SIZE_SOURCE), it was
liable to become too small again later, and thus any program that wasn't
recompiled after the SIGSTKSZ value gets bumped would be liable to new
crashes.

>> Python is far more relaxed about compatibility than glibc, GCC, etc,
>> because they provide large deprecation windows and the ability to
>> install multiple versions of Python.  I've had to deal with some
>> of those fallouts myself.
>
> I don’t think that’s the ordering in which the argument goes. Python
> causes more breakage, so distributions have to provide multiple
> versions, and Python 2 had to be maintained over a decade longer than
> expected.

I haven't done the archival research to say one way or the other.  But
it is clear that the ability to install multiple versions permits making
breaking changes more easily.  I am sure I am not the first to conclude
this, and I'm sure that this knowledge informs CPython maintainers.

> In 2024 the Linux foundation wrote:
>
>> “there is an ongoing transition from Python 2 to Python 3”
>> — page 5 of https://www.linuxfoundation.org/research/census-iii
>
> The Python 3 breakage is a change that should never have been done and
> every project should be careful not to get near that amount of
> breakage.
>
> And that’s despite Python’s popularity, because that mainly persisted
> due to getting lucky that Python 3 got taken up for machine learning.

Is it?  I doubt that.

But, that's not very relevant either.

>> This only applies for the standard library, though.  The projects you
>> refer could've tripped up on some third-party break, such as some
>> unrelated dependency.  But, this could hardly be blamed on Python.
>
> One tripped on a third party (PyKDE). The others tripped on Python 3.
> That was just horrible and should be remembered as an action no one
> should ever take again.
>
>>> That turns a problem of breaking programs into a communication problem:
>>> teaching people to activate the stricter mode for new projects.
>>> (also see the later answer)
>>
>> This is precisely how GCC was doing for decades (with -pedantic,
>> -Werror=...  and -pedantic-errors).  These decades are overwhelming,
>> rather than sufficient proof, that this approach is a dead-end.
>
> Decades of GCC being more and more successful and widespread.

Indeed.  That success and wide-spread use is why I am confident that
these decades are overwhelming evidence that opt-in strictness via
-pedantic utterly failed.

Imagine how much better the software ecosystem would've been if all of
that success and spread was accompanied with better diagnostics.

GCC certainly didn't catch on solely (in fact, I imagine this could only
be a point against it) because it failed to diagnose basic issues by
default.

> LLVM only started to get traction against GCC, because Apple wanted a
> non-copyleft compiler that they can proprietarize, so they funded its
> development.

I do not see the relevance.

>> As I said above, it is a great dis-service to our users to make our
>> compilers less strict by default, as it more easily allows them to
>> write
>
> I have not seen anyone here say that we should change our compilers to
> be less strict.

Sure.  I didn't say anyone said that.

> Just that when they were already lax by default, that should not be
> changed.

Right.  And I'm writing against that.

On this specific point, the one that changed in GCC 14, the vast
majority of instances were *not* intentional usages of K&R C syntax.  In
most of these cases, they were serious bugs waiting to explode in
someones' face.

This was made even more clear during the subsequent cleanup process
which, recall, I also participated in (though I personally did a
relatively small chunk of the work compared to some other maintainers).

>> Let's presume that the programmer and user are both dealing, in their
>> own separate ways, in a codebase that was produced in the above
>> scenario.
>>
>> Now, let us presume that the compiler developers finally started
>> "strictening" the compiler.
> …
>
>> - The developer updates their compiler.  Their code breaks.  They can
> …
>> - The user updates their compiler and attempts to build the program.
>>   The incorrect program code is rejected.
> …
>> The user is prevented from being given a false sense of security, and a
>> subtly broken program.
>
> The user had a working program before, though with bugs, and now they
> no longer have a working program, so the tasks for which they depended
> on the program are broken.
>
> For the user this is a purely worse situation, because they already used
> the program in ways that worked.

Had it not been this loud change, it'd have been some subtle change that
leads to a breakage.  One that's hard to detect, replicate, track down,
and report.

As I said, the vast majority of instances of these errors were not
innocuous, let alone intentional.

> The part that this scenario misses is that the number of maintainers of
> most free software projects is "less than one". If the compiler breaks
> it and no distribution chips in with patches, it’s dead.

If the change of the size of the void* type relative to the int type
breaks it, and no distribution chips in with patches, it's dead.

If the change of the piece of address space in which the allocator
places new pages breaks it, and no distribution chips in with patches,
it's dead.

If any arbitrary set of its false assumptions changes, and no
distribution chips in with patches, it's dead.

> That having less than one maintainer is viable is actually a strength,
> because it means that the sum of required maintenance work over all of
> Free Software is pretty low. We can build upon the work of others and
> step by step challenge proprietary programs with far more developers.

And the compiler being stricter makes that more viable.

> But it also means that breaking backwards compatibility can cause
> massive damage.
>
>>> Those conditions only happen when usage changes. In the situation where
>>> the program is currently being used, it works correctly.
>>
>> I am not sure what this means.  The program reduced out of flex
>> presented as an example is always incorrect, but probably only crashes
>> when the size of ints and pointers differs, and when malloc returns a
>> pointer >= 2<<31.
>>
>> The fact that something works out by accident does not mean it's
>> correct.
>
> See above, expanded a bit:
>
> If a program doesn’t work correctly for some inputs, people will stop
> using it for those inputs. So after a while all situations in which the
> program is used will be situations where the problems are
> inconsequential.

That example is not an input.  That example is an unrelated factor
dependent on details of the execution environment practically equivalent
to noise.

> Fixing the program for those inputs eases the work for people, so adding
> warnings is very valuable, if it is still actively maintained.
>
> But breaking the program because it may be broken for some inputs breaks
> it for all its current users for all inputs. If it is not actively
> maintained, this destroys it for these users.

No, it doesn't.  You do not need to recompile a program before each use.
Indeed, even more care is taken to keep programs in binary form working.

I can't recall when the last serious ABI break in glibc or GCC happened.

If anything, we got better on this front.

Gentoo used to advise users to rebuild their systems on a GCC update,
for instance.  We've gotten rid of that advice a while ago.

The bugs triggered by K&R C constructs also don't happen on 'some
inputs' that can be avoided.  Most of the time, it's something like the
example above.

>> So, no, the people dealing with the fallout were most definitely the
>> people doing the change.  That's why distro testing happens when the
>> compiler and libc change.
>
> This only applies for the packaged programs. Which doesn’t apply to
> compiling old versions of programs (they are no longer packaged).
>
> To not be misunderstood: I am very, very happy that distributions
> provide patches and test compatibility. I’ve only contributed minor
> patches in that direction over the years and I’m really glad that you
> did far more.
>
> But I’ve been hit by broken compatibility again and again. Because my
> programs weren’t popular enough to be packaged. Except for the one that
> got unpackaged because PyKDE died because KDE broke compatibility
> because Qt broke compatibility.
>
>> The collateral damage only exists because the compiler was lax with
>> diagnosing incorrect programs.
>
> This is the blame game. I want to prevent harm, not cast blame.
>
> This may sound flippant, but it is my actual goal.
>
> I don’t want to blame you for a change that’s done. That ship is
> sailed.

As the amount of code in the world grows, every failure to implement
something to prevent a bug (harm) will manifest as many future bugs (a
lot of harm).

> I just want to see less breakage in the future. And I know that we won’t
> have "no breakage", because some of this is outside our reach.
>
> And I also know that GNU is doing a far better job at preserving
> backwards compatibility than many other projects out there.
>
> And I am deeply thankful for that.
>
>>> Hyrum’s Law does not mean to give up on compatibility.
>>
>> I explicitly stated that it does not.
>>
>>> Instead it states that “changes to the system must maintain these
>>> performance characteristics to continue functioning for its consumers”
>>
>> No it doesn't.  You seem to have extrapolated an example (and indeed an
>> undocumented performance characteristic is observable but undocumented
>> behaviour and thus a special case of the general rule) into a statement
>> of the rule.
>
> I quoted from the website of Hyrum’s Law.

Right.  That was an example.  Not the statement of the law.

>>> and “the implicit interface constrains your system design and
>>> evolution” -- https://www.hyrumslaw.com/ Not a license for breakage
>>> but a warning that compatibility goes deep.
> …
>> The obvious implication of this is that there is an extremely high value
>> of preventing misuse.
>>
>> Thus, breaking compatibility to prevent misuse can easily be worth it.
>
> It can be. But usually it does far more damage than good, because the
> ones who do the change cannot know who depends on compatibility.
>
> They cannot, because they cannot know what kinds of programs are
> built outside the package manager.
>
> And I have yet to experience a case where it was worth it. Until now
> every case that I experienced caused more damage than good. Including
> projects I contribute to. The worst was that a backwards compatibility
> breakage in Guile almost caused Lilypond to be thrown out of Debian a
> decade later. The change seemed obvious and inconsequential on the
> surface, but it broke a feature baked deeply into Lilypond, and it took
> years to heal even some of the wounds that caused.
>
> The damage it caused always was far bigger than expected.
>
> That’s why I nowadays think that breaking backwards compatibility should
> only ever be done if it truly cannot be avoided.

We had something that could be considered such a case earlier in this
thread, wrt SIGSTKSZ.

I won't repeat myself on my assessment of the damage caused by
diagnosing K&R C syntax.

Also, FWIW, anecdotally, I've asked around and could only collect
accounts of about two or three dozen people who strongly disliked this
change, despite me being present in GCC support rooms and lists and
reading all bug reports that reach the GCC bugzilla.

That said, it could be argued that -fpermissive should've been
advertised better.

Maybe I would've even supported e.g. the compiler saying "if you're
compiling old code, you may need '-fpermissive'" or such.

But I am worried that even that would've lead to too many new, broken
programs.

Then again, a programmer that'd have taken the above sentence to mean
"you need to pass -fpermissive" rather than "oh, there's something wrong
in the program I am currently working on" is unlikely to have been
writing a good program anyway.
-- 
Arsen Arsenović

-----BEGIN PGP SIGNATURE-----

iQECBAEWCgCqFiEE/uKz0RP8AKMWLWBhUsKUMB6ixJMFAmp+NQIbFIAAAAAABAAO
bWFudTIsMi41KzEuMTIsMiwyXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25z
Lm9wZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRGRUUyQjNEMTEzRkMwMEEzMTYyRDYw
NjE1MkMyOTQzMDFFQTJDNDkzEBxhcnNlbkBhYXJzZW4ubWUACgkQUsKUMB6ixJNu
vAEA2Yq605YexqSLkVPkR0UZ3W1NpjVos0V5VcEEUgTHUwYBAMWalRqJDepYudMW
G+FbyQ6qCXOModoLvaK3i8WDJsoC
=IbJo
-----END PGP SIGNATURE-----