Optimizing C/R Image Format for Kubernetes

Hi everyone,

I've been spending the last few days diving into checkpoint/restore
(C/R) within Kubernetes, specifically focusing on the restore process
and the current image format.

I found the current container image format to be suboptimal. I've
examined containerd, and I suspect CRI-O has similar issues.
Essentially, it's a container image that encapsulates a
checkpoint-restore archive. Each container start requires multiple
unpacking steps:
* Extracting the C/R archive: This yields two tar archives—one for the
  filesystem delta and another for CRIU images.
* Applying the filesystem delta: We need to mount the container's root
  filesystem, then extract and apply this delta.
* Restoring the container: Finally, we extract the CRIU images and
  proceed with the restore.

I believe this format, with its nested tar archives, leads to a
significant amount of time wasted on unpacking, which directly impacts
performance.

With the growing interest in using C/R to optimize application startup
time. I've run some experiments. My findings indicate that the current
image format significantly reduces the benefits of C/R, and in many
cases, restoring a container from these images is actually slower than
starting it from scratch.

Here's my vision for an ideal image format for C/R-ed containers:
* Filesystem Delta as an Overlay Layer: The filesystem delta should be
  treated just like any other container image delta. This means it would
  be specified as one of the overlay layers when a container is mounted.
* Directly Accessible CRIU Images: Once an image is pulled locally, the
  CRIU images should not be bundled in a tar archive. Instead, they
  should be placed directly in a directory, allowing CRIU to use them
  immediately without any extra extraction steps.

Thanks,
Andrei