Re: [PATCH for-11.1] gitlab: disable provenance attestations to work around CI bug

On Tue, Aug 04, 2026 at 12:54:14PM -0400, Stefan Hajnoczi wrote:
> QEMU's CI pipeline involves building container images that will be used
> to run builds and tests. A recent Docker change triggered the following
> error:
> 
>   $ docker push "$TAG"
>   ...
>   error from registry: blob unknown to registry - sha256:4401f6f779caf8841cafd5f483e642fcac56a23a4e4a59523231e101c890dad9
> 
> https://gitlab.com/qemu-project/qemu/-/jobs/15701875927#L2372
> 
> This happens because Docker now pushes out-of-order and the GitLab
> Container Registry rejects due to an unknown reference:
> https://forum.gitlab.com/t/started-yesterday-docker-push-error-from-registry-blob-unknown-to-registry/134733/5
> 
> It is unclear at this point whether GitLab will modify the behavior of
> Container Registry or whether Docker will ship a fix.

The forum post links to an issue where this comment:

  https://github.com/moby/buildkit/issues/7007#issuecomment-5178990414

suggests we can just set

  BUILDX_NO_DEFAULT_OCI_ARTIFACT=true

...except I don't think they've updated the docker docker image
with this fix yet. Once they do, that will be better as we can
set it in the GitLab env such that it applies to all branches,
instead of havnig to cherry-pick 

> 
> The current workaround is to disable the provenance attestation that is
> involved in this issue. QEMU's CI pipeline container images are used
> internally for testing and are not widely distributed. Provenance
> attestation can be disabled as there are no external consumers of these
> images. Expect to revert this commit in the future when GitLab or Docker
> have released their own fixes.
> 
> Cc: Alex Bennée <[email protected]>
> Cc: Daniel P. Berrangé <[email protected]>
> Cc: Thomas Huth <[email protected]>
> Signed-off-by: Stefan Hajnoczi <[email protected]>
> ---
> Note that this is an urgent patch so that the CI will pass again and I
> can tag v11.1.0-rc3 later today. Currently it is failing due to this
> issue!
> 
>  .gitlab-ci.d/container-template.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml
> index 8c7311cca57..88317d650e2 100644
> --- a/.gitlab-ci.d/container-template.yml
> +++ b/.gitlab-ci.d/container-template.yml
> @@ -14,7 +14,7 @@
>      - echo "TAG:$TAG"
>      - echo "COMMON_TAG:$COMMON_TAG"
>      - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
> -      --build-arg BUILDKIT_INLINE_CACHE=1
> +      --build-arg BUILDKIT_INLINE_CACHE=1 --provenance=false
>        -f "tests/docker/dockerfiles/$NAME.docker" "."
>      - docker push "$TAG"
>    after_script:

Reviewed-by: Daniel P. Berrangé <[email protected]>

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|