Re: Upcoming CI changes - transition to VM based CI

Am 03.06.25 um 11:42 schrieb Ben Cooksley:
> On Tue, Jun 3, 2025 at 9:03 AM Albert Astals Cid <[email protected]> wrote:
>
>     El dilluns, 2 de juny del 2025, a les 13:39:21 (Hora d’estiu d’Europa
>     central), Ben Cooksley va escriure:
>     > Hi all,
>     >
>     > For some time now we have had a variety of issues with our
>     Docker/Podman
>     > based CI builds. These have included the lack of GUI test support on
>     > Windows, periodic crashes on FreeBSD, poor IO performance of Windows
>     > builds, issues supporting builds for Flatpak and Snaps and
>     inability to
>     > support either builds or tests where elevated privileges or
>     system session
>     > resources are needed.
>     >
>     > In addition to this we've had issues where Linux CI builds have the
>     > capability to trigger OOM events on the CI hosts, which in turn
>     takes out
>     > Windows and (less often) FreeBSD builders. While this does not
>     occur too
>     > often, it does happen from time to time and eventually
>     negatively impacts
>     > the build queue for those platforms.
>     >
>     > The need to have dedicated VMs for FreeBSD and Windows on our
>     builders also
>     > makes setting up of a CI build node for KDE software a more
>     complicated and
>     > time intensive task than it otherwise needs to be (and means
>     that the
>     > amount of systems to care for increases by 3 for every CI node
>     we add).
>     >
>     > While individually relatively minor, together these issues more than
>     > justify making a significant change to the way we run our CI
>     system - in
>     > this case transitioning from container based builds to VM based
>     builds.
>     >
>     > These builds will still take place on dedicated hardware that we
>     control,
>     > however instead of taking place within a container (managed by
>     Podman on
>     > Linux and FreeBSD, or Docker on Windows) they will instead take
>     place
>     > within a VM using a copy-on-write disk image.
>     > VM based builds will unfortunately take a little longer to start
>     (it takes
>     > ~10 seconds for a VM from any of Linux, FreeBSD or Windows to
>     boot on my
>     > personal system) however the benefits we gain should more than
>     outweigh
>     > this small downside.
>     >
>     > This has been under development for the past couple of weeks and
>     is now
>     > reaching the point where the only remaining steps are to get it
>     integrated
>     > with the Gitlab CI agent (gitlab-runner) for which prototype code is
>     > already in place, and complete porting of our images over. Once that
>     > happens a complete rebuild of all of our builders will be swiftly
>     > undertaken to transition them completely over to the new VM based
>     > infrastructure.
>     >
>     > Specs wise, at this time it is planned for each spawned standard
>     VM to be
>     > provided with 2/3's of the system CPU cores (so 12 cores), 16GB
>     RAM and
>     > 100GB of disk space (although some of that will be occupied by
>     the system
>     > image - approximately 10GB for standard Linux builds and ~30GB
>     or so for
>     > Windows builds). There will be a higher resource tier available
>     for certain
>     > builds however that will be on request only and would need to be
>     justified
>     > (such as Craft needing to build QtWebEngine).
>     >
>     > As launching VMs is not the most efficient approach for all
>     workloads,
>     > limited support for running Docker containers will be preserved,
>     however
>     > this support is primarily intended for running linters, sanity
>     checks and
>     > website builds, and is not intended for running general CI/CD
>     builds.
>     >
>     > The tooling used by the CI nodes to run VMs is something that
>     should be
>     > fairly trivial for people to run on their own local system
>     should they wish
>     > to run any of those images (say for FreeBSD or Android),
>     although you will
>     > need to setup libvirt yourself (SUSE has very good instructions
>     for this,
>     > Debian less so as their instructions lack installing the
>     packages needed to
>     > provide UEFI and TPM support). The tooling itself was merged
>     this evening
>     > to sysadmin/ci-images (vm-common/ folder) and can be used with
>     the VM
>     > images found at https://storage.kde.org/vm-images/
>     >
>     > There is however one downside to this - Qt 5 support.
>     >
>     > Over the past few months distributions have been steadily
>     removing packages
>     > and other supporting infrastructure needed to keep Qt 5 builds
>     alive. In
>     > the case of Windows, support for the entire Qt 5 tree has been
>     unmaintained
>     > for some time. For FreeBSD and SUSE a significant number of
>     packages have
>     > been removed - which in the case of SUSE also includes packages
>     needed to
>     > support the building of KJS.  Accordingly, because builds of
>     Frameworks are
>     > a first stepping stone to support building anything else, it
>     will not be
>     > possible for us to produce Qt 5 based VM build images for any of
>     the 3
>     > platforms.
>     >
>     > We will therefore have to remove Qt 5 support from the CI system
>     with the
>     > transition to VM based CI.
>
>     From previous discussions I had the impression this was only for
>     things that
>     wanted to create packages and not for "want to have CI to
>     compile/run tests".
>
>     Can you confirm you are proposng a total annihilation of Qt5
>     support in our
>     CI?
>
>
> At the time we had that discussion it was still possible to build some 
> of the Qt 5 images, however that is no longer the case - all of them 
> now fail to build.
>
> In the case of the suse-qt515 image, the removal of libpcre in SUSE 
> means it is no longer possible to build KJS.
> Consequently, we're no longer able to build all Frameworks (making 
> 'kf5' branch CI for Frameworks non-functional) so there isn't much 
> point in looking further to support Qt 5 on Linux.

Not that I have much sympathy left for Qt5, but wouldn't it be possible 
to exclude kjs and keep the remaining frameworks? I don't think any of 
the unported projects need kjs anyway.


> For FreeBSD the story is much the same as SUSE - packages are being 
> removed as apps upgrade to Qt 6 and the Qt 5 version of libraries 
> becomes surplus to requirements.
>
> For Windows the continued operation of that CI has only been possible 
> because our existing images are still around - new ones cannot be built.
> That has been the case for a significant amount of time now, and it is 
> not worth the investment to fix it as everyone who works on Windows 
> has moved on to Qt 6.
>
> In essence there is little we can do to keep this alive - 
> distributions are removing support so we must follow suit.
>
> The correct course of action is to accelerate the porting of the 
> remaining applications, not to delay and keep Qt 5 alive.
>
>
>     Cheers,
>       Albert
>
>
> Regards,
> Ben
>
>
>     >
>     > Please let me know if there are any questions on the above.
>     >
>     > Thanks,
>     > Ben
>
>
>
>