Re: Proposal: Utilizing Container Registry for Shared BitBake Sstate-Cache

On Mon, Jun 8, 2026 at 11:10 AM Patrick Williams <[email protected]> wrote:
> Welcome Yash.

Thanks for the feedback and the welcome!

> One of the issues we have right now is that we do not have a good way to
> clean up the sstate/downloads directories out of each Jenkins node.
> Eventually they run out of space.  Managing it through containers will
> hopefully make it easier to prune old state and keep the nodes from
> running out of space.

I agree and managing the cache entirely within disposable container
layers should help address this.

> In addition to per-platform containers, you'll also need the branch
> included there.  We should make sure to build this for Wrynose since
> that is the Yocto LTS branch and we've committed to supporting that for
> a few years.

We will add the branch name to the image tagging convention.

> I would suggest a separate job for two reasons:
>     - Ideally we need to trigger this more often than once per day if
>       you want other Jenkins nodes to use that for a starting point of
>       their sstate / downloads cache.
>
>     - You are going to want to create sub-jobs per machine / branch (and
>       like I mentioned, we should do this at least for the Wrynose
>       branch in addition to master).

Agreed. Our current plan is to create a dedicated Jenkins job with
sub-jobs per platform and branch.

> Make sure when you do this that you don't use 'FROM: <old container>'
> because that will just create a chain of container subsets that will
> grow monsterrously large over time.  You have to create a build
> container and a final container.  The build container uses the 'FROM' to
> pull the previous container, but then the final container just takes the
> resulting final sstate/downloads as a directory.

The current implementation avoids this by copying the resulting
sstate/downloads into a fresh final image rather than building a chain
of container layers.

> Have you done any experimentation on if the GHCR will rate limit us in a
> way to make this non-useful?  Do we have enough space on the Jenkins
> server to run a container repository there?  There are a few smaller
> ones that are self-contained in containers like how we run
> Jenkins/Gerrit.

We have not done any extensive testing around GHCR rate limits yet.
Our initial focus has been validating the container workflow itself.
Evaluating GHCR limits and alternative hosting options is something we
plan to investigate once we have the Jenkins generation and
build-setup.sh integration in place.

We have a generic container available for testing. To try it:

git reset --hard 71085e902b0d22bcd854ae0db4c46c6fc666d17a

docker run -it --rm --network=host --pids-limit=-1 \
  -e LOCAL_UID=$(id -u) \
  -e LOCAL_GID=$(id -g) \
  -e LOCAL_USER=$(id -un) \
  -v $PWD:$PWD \
  -w $PWD \
  ghcr.io/pebbleeee/sstate:p10bmc-x86-generic \
  bash

. setup p10bmc

echo 'SSTATE_MIRRORS = "file://.*
file:///var/lib/openbmc/sstate-cache/PATH"' >> conf/local.conf
echo 'BB_SIGNATURE_HANDLER = "OEBasicHash"' >> conf/local.conf

MACHINE=p10bmc bitbake obmc-phosphor-image

Our immediate next steps are to generate the containers through a
Jenkins job and update build-setup.sh to automatically detect and
utilize them.

Thanks for the feedback.