Re: Discussion about why GNU/Linux system upgrades cause old programs to break

Thanks Arsen for explaining lots of detailed technical points, it is
interesting reading.  I have a few questions interspersed below.


Arsen Arsenović <[email protected]> writes:

> Eli Zaretskii <[email protected]> writes:
>
>>> > What happened was a system upgrade.  I don't expect a system upgrade
>>> > to remove capabilities.  Am I alone in this expectation?

I agree this is a reasonable expectation.

>> My point is that at least in some cases we decide "not to solve" this
>> too soon.  We could do better.  Having a program break 25 years after
>> it was built is much better than if it stops working after 5, I hope
>> you agree.
>
> Absolutely, some such cases happened; I listed one such case above.

There is more agreement that a program should continue to work for a
longer period of time than sometimes it currently does.

>
>> You are talking about programs that are part of the OS.  By contrast,
>> I'm talking about programs that were built by the user -- those are
>> not upgraded when the system is, and one of the worst effects of a
>> system upgrade is to see programs which worked for you for years
>> suddenly stop working.
>
> No, I'm not.
>
> My paragraph was worded a bit poorly, let me restate what I stated
> above, and expand on it:
>
>>> The system vendor (be it Apple or Microsoft or Google or whoever)
>>> simply ships you, the developer, a set of dependencies dated to, say,
>>> some year, so you end up seeing breakages when you change which date
>>> the system-vendor-provided dependencies attach themselves to, because
>>> you, the developer, are the one acting as an integrator for the vast
>>> majority of the dependencies, and your own code, with the
>>> vendor-provided code.
>>> 
>>> The users end up not seeing this at all because each program gets
>>> distributed with nearly all dependencies, with the ones distributed
>>> as part of the OS being installed multiple times.
>
> These system-vendor-provided dependencies are equivalent to the "core"
> GNU system: GCC runtime libraries, glibc, coreutils, GTK2/3/4 (at some
> point at least), etc.  These are in that year-versioned bundle that I
> was referring to, that gets installed a few times on each users' system.
>
> The dependencies outside of these, however (say, libjpeg or
> tree-sitter), are integrated by the developer, and get shipped with the
> program.
>
> So, when any of these dependencies - whichever category they belong to -
> change, builds break *for the developer*.  The user doesn't notice,
> because it was made the developers' problem.  The user just gets a ton
> of binaries, with all the parts epoxied together.
>
> But the user in our case is given source-code, and the freedom and
> compulsion to vary dependencies.  Exactly as the developer was in these
> alternative models.  The user becomes the integrator.
>
> It is not possible to avoid breakages as a result.
>
> Great effort is already taken to avoid them, yet they still happen.
>
>>> (those who have used MS-w are aware of "VC++ Redistributable XYZW" - or
>>> whatever the name was - which they have to install on occasion for
>>> various XYZW).
>>
>> That's just a packaging issue: those redistributables should have been
>> bundled with the program that needs them, but could not be due to
>> copyright issues and such.  The important part is that, once you do
>> have the redistributable installed, no system upgrade will remove it.
>
> I'm not sure whether a MS-w upgrade by a major version would remove
> them.
>
> But, even if not, they're the exception to the rule on that OS; most
> dependencies are shipped with most programs, thus they're mostly
> isolated from each-other.
>
> Which is its own mess, of course (tried using TeXLive and Inkscape
> together through svg.sty?  what a mess..).
>
> The issue we're discussing is also a packaging issue, though; programs
> in the Linux and BSD and ... ecosystems generally don't get distributed
> in binary form with all of their dependencies built in, they get
> distributed in source form, without dependencies.
>
> Whereas programs in the ecosystems I refer to above are all distributed
> in binary form with nearly all of their dependencies bundled.
>
> The former means that the same sources are expected to work with a
> myriad of combinations of various dependency versions.
>
> Obviously, the developer testing them all is unfeasible.  Especially as
> many of those combinations do not yet exist at time of writing.
>
> Thus, problems arise sometimes quite late.
>
> This is where compiler laxness comes into play.  I'll return to this
> below.
>

>>> This is also why many cling to static linking.
>>
>> Static linking is a two-edged sword: it could be more robust in the
>> long run, but in the shorter run you lose the ability to easily update
>> a dependency library if and when its developers fix some grave bug in
>> it, without the need to rebuild your application.
>>
>> Most programs nowadays use dynamic linking, IME.
>
> There's other problems with static linking also, but, yes, most programs
> are dynamic linked nowadays.
>
> What I was saying is that there's a contingent of developers who cling
> to static linking as a solution for the problems you lie out in this
> thread.
>
> Which are real problems.  But they aren't problems solvable at the
> source level, or even at the binary level.  We still try to, though.
>
> The solutions in the ecosystems I referred to above are far closer to
> static linking (because you can't really independently update components
> as the user) than what we usually do in the Linux and BSD and
> ... ecosystems.
>
> You can't have your cake and eat it, too.  You either allow varying
> dependencies and thus have breakages, or you don't, and everything works
> (you even get gratis bugs because you can't independently fix them).
>
> I think we've largely made the right choice.  The free software world
> permits anyone to fix problems that arise as a result of Hyrum's Law,
> and we're able to integrate masses of code written by many thousands of
> people and work on them collaboratively.
>
> But, this does mean that the constraint of "unmodified program works
> indefinitely" is impossible to satisfy.

Arsen, do you have a suggestion for improving the longevity of Emacs
source?

Are you suggesting it amounts to forms of more 'static linking'?
Should it be something to recommend to distros (Gentoo, Debian/Trisquel,
...) ?
Would there be a subset of dependencies such that there is greater
longevity anticipated for, by way of illustration, Emacs compiled
without GTK.  Namely, a subset of e.g. terminal-based Emacs?

>>> > One example of this is compiling a program that used K&R syntax.
>>> > E.g., Emacs switched to C90-style function definitions only in version
>>> > 24.1, which was released in 2012.  All the older versions used K&R
>>> > syntax; they are now "broken" by your definitions, and cannot be
>>> > easily compiled.
>>> 
>>> Well, yes, if Emacs versions up to 2012 were telling the compiler that
>>> they were C99 programs, but were not valid C99, they were certainly
>>> broken, and only worked due to a defect in GCC.
>>
>> See above: that's not what happens.  In reality, the build invokes the
>> compiler without specifying the dialect, so whatever is the default C
>> dialect is being used.  So when the default changes to be
>> backward-incompatible, it's bound to break old programs.
>
> Indeed, the default dialect can change.  This is why I find the
> autotools choice to only provide -std=... conditionally dubious.

Do you have a suggestion to improve this?


>>> I've argued before, and stand by it now, that source-level backwards
>>> compatibility should not be the default for compiled languages.  When a
>>> compiler is present, it should only be backwards-compatible if told to
>>> be explicitly.
>>
>> This makes recompiling old sources harder, so I think it's a mistake,
>> unless we don't care about these case.
>
> I'm sure many more hours have been spent tracking down stuff that
> could've been prevented by stronger diagnostics than have been spent
> trying to compile old code.
>
> I do not mean to imply that backwards compatibility is a "sin".  But,
> backwards compatibility in this instance means relaxing compiler
> diagnostics, thus making the compiler less strict.  This is universally
> a disservice to users.  I've went over why earlier, but I can restate it
> in brief again: anything that can be caught early but isn't harms
> everyone, including program authors and users.
>
> There are two different classes of users of compilers: program
> authors and program users building the source form.
>
> The former is harmed, substantially, by such default laxness, because
> they're enabled to write programs that are slightly wrong.  The latter
> receives some slight relief from default backwards compatibility.
>
> But, this is a Pyrrhic victory for the latter: as a result of default
> backwards compatibility, rather than fixing defects in an early versions
> of programs, those defects go unnoticed are kept around long enough to
> break in more insidious ways later, blowing up in the users' face.
>
> Indeed, the case I showed in my last email with 'reallocarray' was
> exactly such a case: the user of flex "benefited" from default backwards
> compatibility, only to get a program that was slightly broken under
> specific conditions.
>
> Luckily, in that case, the breakage manifested as an easy-to-find crash.
> In many cases, it will manifest as a rare and difficult to diagnose
> runtime bug.
>
> Meanwhile the developer of flex was left unaware of this breakage for a
> longer time.
>
> It was a disservice to both classes of users to postpone or entirely
> omit diagnostics on code that GCC knew was very likely to be wrong.
>
> The driver behind the GCC 14 change was not that anyone else decided to
> do it (though arguably Clang doing it acted as a trigger), it was that
> we've been fixing related bugs for many years in many programs, and that
> we've been teaching students that "this code is not correct but the
> compiler ignored it because old code exists, but you shouldn't ignore
> it" (to very little effect).

Arsen, do you have more ideas on how to make recompiling old Emacs sources
specifically have greater longevity than at present?