Re: CI Outages

--00000000000008c5990650da41c3
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi all,

I'm happy to announce that final repairs on both Windows and FreeBSD were
completed yesterday and those builds are now fully functional again which
completes the restoration of normal CI service.

Please let us know if you see any unusual failures.

Many thanks,
Ben

On Fri, May 1, 2026 at 10:02=E2=80=AFPM Ben Cooksley <[email protected]> wr=
ote:

> Hi all,
>
> Over the past 48 hours or so we've had a series of two unfortunate
> incidents that have significantly degraded the CI system.
>
> The first part of this took place approximately 36 hours ago, when an
> admin, in response to the announcement of the https://copy.fail/ Linux
> kernel exploit, installed updates on one or more of our VM Runners. The
> unfortunate side effect of this is that it also installed updates for
> gitlab-runner and upgraded it to a newer version. As part of it's work,
> Gitlab Runner requires the assistance of a helper binary within the VM, a=
nd
> this helper should ideally be the same version as is deployed on the VM
> runner hosts, or at the very least be a newer version.
>
> In the case of this update, there were incompatible changes as part of
> changes to how artifacts are captured, which is why we are seeing breakag=
es
> related to an unrecognised timeout parameter which is causing a complete
> fatal failure of the CI jobs.
>
> The images that support the majority of our CI builds (Linux - Qt 6.11, Q=
t
> 6.12 and Qt 5.15, Android, Flatpak, Snap and Appimages) have been rebuilt
> to include the newer Gitlab Runner helper and those builds should now be
> functional again.
> Custom VM images utilised by Yocto, Buildstream, Neon and KDE Linux have
> also been rebuilt and should also be functional again.
>
> Windows builds require a replacement base image as the Gitlab Runner
> helper is burned into the base image - and that is in the process of bein=
g
> uploaded currently.
> Once uploaded, i'll rebuild the image that supports both general Windows
> CI and Craft builds which will restore those builds to working order as
> well.
>
> For FreeBSD, we will need our custom package repository updated to includ=
e
> the newer Gitlab Runner helper.
> This has been requested and should be completed in the next few days so
> those builds will remain broken for a bit longer i'm afraid.
>
> The second incident involved a service outage of the builder that support=
s
> Docker based jobs. This was caused by hoster maintenance related to the S=
AN
> that supports those hosts, and also caused service disruptions to all
> Notary Service operations, WebSVN and Sentry.
> This outage impacted us for approximately 12 hours and has now been
> corrected with all services fully returned to normal.
>
> Apologies for the disruption caused by these incidents, it is most
> regrettable - and in the case of the issue affecting VM builds - complete=
ly
> avoidable.
>
> Please let me know if you have any questions on the above.
>
> Many thanks,
> Ben
>

--00000000000008c5990650da41c3
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi all,<div><br></div><div>I&#39;m happy to announce that =
final repairs on both Windows and FreeBSD were completed yesterday and thos=
e builds are now fully functional again which completes the restoration of =
normal CI service.</div><div><br></div><div>Please let us know if you see a=
ny unusual failures.</div><div><br></div><div>Many thanks,</div><div>Ben</d=
iv></div><br><div class=3D"gmail_quote gmail_quote_container"><div dir=3D"l=
tr" class=3D"gmail_attr">On Fri, May 1, 2026 at 10:02=E2=80=AFPM Ben Cooksl=
ey &lt;<a href=3D"mailto:[email protected]">[email protected]</a>&gt; wrote=
:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.=
8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"lt=
r">Hi all,<div><br></div><div>Over the past 48 hours or so we&#39;ve had a =
series of two unfortunate incidents that have significantly degraded the CI=
 system.</div><div><br></div><div>The first part of this took place approxi=
mately 36 hours ago, when an admin, in response to the announcement of the=
=C2=A0<a href=3D"https://copy.fail/" target=3D"_blank">https://copy.fail/</=
a> Linux kernel exploit, installed updates on one or more of our VM Runners=
. The unfortunate side effect of this is that it also installed updates for=
 gitlab-runner and upgraded it to a newer version. As part of it&#39;s work=
, Gitlab Runner requires the assistance of a helper binary within the VM, a=
nd this helper should ideally be the same version as is deployed on the VM =
runner hosts, or at the very least be a newer version.=C2=A0</div><div><br>=
</div><div>In the case of this update, there were incompatible changes as p=
art of changes to how artifacts are captured, which is why we are seeing br=
eakages related to an unrecognised timeout parameter which is causing a com=
plete fatal failure of the CI jobs.</div><div><br></div><div>The images tha=
t support the majority=C2=A0of our CI builds (Linux - Qt 6.11, Qt 6.12 and =
Qt 5.15, Android, Flatpak, Snap and Appimages) have been rebuilt to include=
 the newer Gitlab Runner helper and those builds should now be functional a=
gain.=C2=A0</div><div>Custom VM images utilised by Yocto, Buildstream, Neon=
 and KDE Linux have also been rebuilt and should also be functional again.<=
/div><div><br></div><div>Windows builds require a replacement base image as=
 the Gitlab Runner helper is burned into the base image - and that is in th=
e process of being uploaded currently.=C2=A0</div><div>Once uploaded, i&#39=
;ll rebuild the image that supports both general Windows CI and Craft build=
s which will restore those builds to working order as well.</div><div><br><=
/div><div>For FreeBSD, we will need our custom package repository updated t=
o include the newer Gitlab Runner helper.=C2=A0</div><div>This has been req=
uested and should be completed in the next few days so those builds will re=
main broken for a bit longer i&#39;m afraid.</div><div><br></div><div>The s=
econd incident involved a service outage of the builder that supports Docke=
r based jobs. This was caused by hoster maintenance related to the SAN that=
 supports those hosts, and also caused service disruptions to all Notary Se=
rvice operations, WebSVN and Sentry.</div><div>This outage impacted us for =
approximately 12 hours and has now been corrected with all services fully r=
eturned to normal.</div><div><br></div><div>Apologies for the disruption ca=
used by these incidents, it is most regrettable - and in the case of the is=
sue affecting VM builds - completely avoidable.</div><div><br></div><div>Pl=
ease let me know if you have any questions on the above.</div><div><br></di=
v><div>Many thanks,</div><div>Ben</div></div>
</blockquote></div>

--00000000000008c5990650da41c3--