Re: Optimizing C/R Image Format for Kubernetes

On Thu, Jun 19, 2025 at 4:06 AM Adrian Reber <[email protected]> wrote:
...
>
> > Here's my vision for an ideal image format for C/R-ed containers:
> > * Filesystem Delta as an Overlay Layer: The filesystem delta should be
> >   treated just like any other container image delta. This means it would
> >   be specified as one of the overlay layers when a container is mounted.
>
> Yes. The current format was my wrong decision as I was not familiar
> with how those delta layers are working.
>
> > * Directly Accessible CRIU Images: Once an image is pulled locally, the
> >   CRIU images should not be bundled in a tar archive. Instead, they
> >   should be placed directly in a directory, allowing CRIU to use them
> >   immediately without any extra extraction steps.
>
> This is not actually true. The OCI image does not contain the tar
> archive but the actual checkpoint files directly:
>
> # podman pull quay.io/adrianreber/checkpoint-test:tag73
> Trying to pull quay.io/adrianreber/checkpoint-test:tag73...
> Getting image source signatures
> Copying blob e65839d7ec1b done
> Copying config 27d63848a3 done
> Writing manifest to image destination
> Storing signatures
> 27d63848a32d24c68b131f99880411c11af6519820ef22b989a86b7f10038c79
> # podman image mount quay.io/adrianreber/checkpoint-test:tag73
> /var/lib/containers/storage/overlay/98aaf3c7dc28cfb2e79893ef952380b00169dcce910be48bbea1143b07ae2a0e/merged
> # ls -la /var/lib/containers/storage/overlay/98aaf3c7dc28cfb2e79893ef952380b00169dcce910be48bbea1143b07ae2a0e/merged
> total 44
> dr-xr-xr-x. 1 root root  4096 Jun 19 10:53 .
> drwx------. 6 root root  4096 Jun 19 10:53 ..
> -rw-------. 1 root root  1120 Feb  1 11:11 bind.mounts
> drw-------. 2 root root  4096 Feb  1 11:11 checkpoint
> -rw-------. 1 root root   616 Feb  1 11:11 config.dump
> -rw-------. 1 root root     0 Feb  1 11:11 dump.log
> -rw-r--r--. 1 root root   315 Feb  1 11:11 io.kubernetes.cri-o.LogPath
> -rw-r--r--. 1 root root  2048 Feb  1 11:11 rootfs-diff.tar
> -rw-------. 1 root root 11276 Feb  1 11:11 spec.dump
> -rw-r--r--. 1 root root    49 Feb  1 11:11 stats-dump
>
> We currently have some metadata defined in
> github.com/checkpoint-restore/checkpointctl which we want to use in
> all three projects (podman, containerd and cri-o).

You know, maybe there's a difference between CRI-O and containerd.
I followed the steps from the containerd test to create an image:
https://github.com/containerd/containerd/blob/main/contrib/checkpoint/checkpoint-restore-kubernetes-test.sh#L105

root@gke-cluster-1-default-pool-595f3f31-2wft:/home/avagin# docker
create --name test-image avagin/test-cpt:0.5 ls
c80dbf467d99a0e3a6684d6cb36d29c212a6b12bdfc9af8abe2ffe3fcb69a5de
root@gke-cluster-1-default-pool-595f3f31-2wft:/home/avagin# docker
export test-image | tar -t
.dockerenv
blobs/
blobs/sha256/
blobs/sha256/5159244823d7bfa959a4249c912ffef669c5596fcf41a866264823152b6dbba9
blobs/sha256/9178f6d56b033b8221dda746c3fd9ad98552569f05e66241365ef8a722da96be
blobs/sha256/eca4c8bdd20acb007a5594777ace63727d2c17413a54d3a5a817e252d0390902
dev/
dev/console
dev/pts/
dev/shm/
etc/
etc/hostname
etc/hosts
etc/mtab
etc/resolv.conf
index.json
oci-layout
proc/
sys/
root@gke-cluster-1-default-pool-595f3f31-2wft:/home/avagin# docker
export test-image | tar -x -C test-img/
# tar -tf test-img/blobs/sha256/eca4c8bdd20acb007a5594777ace63727d2c17413a54d3a5a817e252d0390902
checkpoint/
checkpoint/cgroup.img
checkpoint/core-1.img
checkpoint/core-8.img
checkpoint/descriptors.json
checkpoint/fdinfo-2.img
checkpoint/fdinfo-3.img
checkpoint/files.img
checkpoint/fs-1.img
...

>
> What I also would like to see is that we can directly write to an OCI
> image and not just first to a local tar archive and then convert it to
> an OCI image (like Podman already does today). But that requires buy-in
> from Kubernetes and changes to the CRI-API which has always been
> extremely difficult for me to get accepted by Kubernetes. The main
> problem is that checkpoint/restore is not seen as an important feature
> from most Kubernetes contributors (especially approvers and reviewers).
>
> So having someone who supports our work instead of blocking it would
> help us a lot.
>
> There is also the fear of exposing secret information which often blocks
> and progress in the Kubernetes area. Having encryption in CRIU would
> also make those discussions easier (even if the data is not always
> encrypted, but being able to check the encryption box would make
> discussions easier).

Radostin, what is the current state of encryption for CRIU images?