Re: Architecture variants for Debian / Ubuntu

On 03.05.24 11:27, Julian Andres Klode wrote:
> On Wed, Sep 06, 2023 at 11:27:02AM +0200, Guillem Jover wrote:
>> Hi!
>>
>> On Fri, 2023-09-01 at 08:43:55 +1200, Michael Hudson-Doyle wrote:
>>> Recently the topic of exploiting newer instructions without dropping
>>> support for older machines has come up several times inside Ubuntu
>>> engineering. I understand this topic has come up several times in the past
>>> for Debian as well, but nothing has really come of it to date.
>>
>> I also had a chat about this with Matthias Klose (CCed) around 2022-05.
>>
>>> I've spent a while thinking through the options and coming up with a design
>>> and wrote some notes into a wiki page:
>>> https://wiki.debian.org/ArchitectureVariants
>>
>> I think we are already doing 1, 2 and 3. I agree 4 is just wrong. And
>> something like 5 is what I suggested to Matthias for Ubuntu when we
>> last discussed it as the best way to go about this.
>>
>> I'm not sure I entirely agree with the requirements you set forth
>> though:
>>
>>   - I think such optimized builds might need to be done with "special
>>     toolchains" (these could simply be wrappers over the host compiler
>>     passing the appropriate flags via command-line or via specs or
>>     similar, not necessarily full blown toolchains), passing these via
>>     something like dpkg-buildflags seems currently unreliable, as I don't
>>     think we have full coverage in packages (neither for all compilers
>>     available)? Although it would be better as it would centralize the
>>     management. (For reference this is in part how rpm handles this:
>>      https://github.com/rpm-software-management/rpm/blob/master/rpmrc.in)
>>   - Perhaps that's a limitation from the archive software side, but
>>     requiring to place the binary packages in the same pool seems
>>     rather restrictive (it forces different filenames for example).
>>   - I guess it might be nice for the ISA to be passed down to the
>>     dpkg tools, but I don't think this is strictly necessary? A
>>     frontend like apt could also decide based on metadata in say the
>>     Release file, although not having the actual installed package
>>     metadata on whether it was a different ISA build or not would make
>>     its job more inconvenient. In any case I don't have a big issue
>>     with recording this via dpkg-gencontrol or similar if necessary.
>>
>> On the specific implementation details:
>>
>>   - Changing the Architecture format (as in adding colons there) seems
>>     like a non-starter, and I expect that would break lots of things
>>     (I mean it could be done but I'm not sure it's worth it for this).
>>     Recording this mostly as a hint than anything else, via another
>>     field (if necessary at all) I think would be best.
>>   - As covered in previous discussions, dpkg could (but I don't think
>>     it's necessary) check whether the .deb is runnable on the current
>>     hw, but that's tricky as chrootless installs need to be taken
>>     into account, etc. It should certainly not be part of dependency
>>     resolution.
>>   - I'm not fond of having to change the binary package name format
>>     either for this (name_version_arch.deb) even if at least dpkg
>>     itself does not care (but I know other tools do care), and
>>     depending on the format I'd expect things to break (this goes
>>     back to the shared pool concern).
>>   - If dpkg-architecture needs to be aware of this, then this might need
>>     to be auto-detectable from just the current toolchain being used.
>>
>> Some of the above problems could perhaps be avoided if we introduced
>> a concept of architecture aliases/ISAs (similar to what rpm has), which
>> would side-step the pool sharing issue, the binary package renaming,
>> etc. One big issue with this is that it requires for dpkg to have an
>> exhaustive table of all such aliases, and if there's ever a new alias
>> added, old dpkg versions need to be updated or they will not understand
>> what they match with. So this does not seem ideal either. So I guess this
>> is a variation over your proposal, but perhaps this could still be used
>> in specific contexts, say only at build-time (but not for dependency
>> relationships), for repo management (say binary-arm64v9/Packages.xz),
>> or binary package names where the field would specify the actual name
>> for the filename, say:
>>
>>    Architecture: arm64
>>    ArchitectureIsa: arm64v9
>>
>> or maybe better:
>>
>>    Architecture: arm64
>>    ArchitectureIsa: v9
>>
>> resulting in dpkg-deb generating:
>>
>>    binpkg_1.0-1_arm64v9.deb
>>
>> but targeting arm64. I also think I prefer naming this explicitly as ISA
>> variants, if you will, than just architecture variants as that gives
>> way too much room (which perhaps we want, but then that has other
>> implications over compatibility), and for the field perhaps just Isa is
>> better, to avoid the implicit repetition of
>> ArchitectureInstructionSetArchitecture :), but that makes it less easy
>> to associate both as related.
> 
> I have thought more about this and I'm not particularly fond of the
> ArchitectureIsa name. While *this specific use case* is a variant of
> the architecture instruction set; you could just as well build other
> variants such as "compiled with -O3", "compiled with frame pointers",
> "compiled with -O0", or other shenanigans (I haven't thought about
> others outside compiler flags)

or
  - DistroBuiltWithClang, e.g. using libc++ instead of libstdc++.
  - distro built with some of the sanitizers turned on by default

> Hence I prefer Architecture-Variant, Subarchitecture, or anything
> like that rather than have to invent another field or abuse this
> one the next time we want to build a special variant of an architecture
> with special optimizations for a special customer or whatnot.

yes, it sounds like a bit too specific.