Re: CI Outages

Thanks a lot for the quick response! =F0=9F=91=8F

Aleix

On Fri, May 1, 2026 at 12:02=E2=80=AFPM Ben Cooksley <[email protected]> wr=
ote:
>
> Hi all,
>
> Over the past 48 hours or so we've had a series of two unfortunate incide=
nts that have significantly degraded the CI system.
>
> The first part of this took place approximately 36 hours ago, when an adm=
in, in response to the announcement of the https://copy.fail/ Linux kernel =
exploit, installed updates on one or more of our VM Runners. The unfortunat=
e side effect of this is that it also installed updates for gitlab-runner a=
nd upgraded it to a newer version. As part of it's work, Gitlab Runner requ=
ires the assistance of a helper binary within the VM, and this helper shoul=
d ideally be the same version as is deployed on the VM runner hosts, or at =
the very least be a newer version.
>
> In the case of this update, there were incompatible changes as part of ch=
anges to how artifacts are captured, which is why we are seeing breakages r=
elated to an unrecognised timeout parameter which is causing a complete fat=
al failure of the CI jobs.
>
> The images that support the majority of our CI builds (Linux - Qt 6.11, Q=
t 6.12 and Qt 5.15, Android, Flatpak, Snap and Appimages) have been rebuilt=
 to include the newer Gitlab Runner helper and those builds should now be f=
unctional again.
> Custom VM images utilised by Yocto, Buildstream, Neon and KDE Linux have =
also been rebuilt and should also be functional again.
>
> Windows builds require a replacement base image as the Gitlab Runner help=
er is burned into the base image - and that is in the process of being uplo=
aded currently.
> Once uploaded, i'll rebuild the image that supports both general Windows =
CI and Craft builds which will restore those builds to working order as wel=
l.
>
> For FreeBSD, we will need our custom package repository updated to includ=
e the newer Gitlab Runner helper.
> This has been requested and should be completed in the next few days so t=
hose builds will remain broken for a bit longer i'm afraid.
>
> The second incident involved a service outage of the builder that support=
s Docker based jobs. This was caused by hoster maintenance related to the S=
AN that supports those hosts, and also caused service disruptions to all No=
tary Service operations, WebSVN and Sentry.
> This outage impacted us for approximately 12 hours and has now been corre=
cted with all services fully returned to normal.
>
> Apologies for the disruption caused by these incidents, it is most regret=
table - and in the case of the issue affecting VM builds - completely avoid=
able.
>
> Please let me know if you have any questions on the above.
>
> Many thanks,
> Ben