Re: [REGRESSION] Return change in 6.12.80+ with volatile mounting

We can fix the test case for the "volatile" option and expect either
"volatile" or "fsync=volatile". Existing use within containerd beyond
the test code involves us producing the "volatile" option and passing
it to the mount syscall (or _not_ passing it, depending on the
situation). This seems like it should still work if "fsync=volatile"
is an alias rather than a replacement.  However, containerd is not in
full control of the mount options; as part of snapshot (layer)
management external snapshotter implementations can provide mount
options to containerd which containerd then passes to the kernel.  The
protection in RemoveVolatileOption will break if a newer snapshotter
produces "fsync=volatile" and provides it to an unpatched version of
containerd.  I think the breakage scenario then becomes new kernel +
new snapshotter (producing the new option) + old containerd.

-- 
Samuel Karp

-- 
Samuel Karp


On Tue, Apr 21, 2026 at 10:20 AM Chenglong Tang
<[email protected]> wrote:
>
> CC: Samnuel from containerd
>
> On Mon, Apr 20, 2026 at 12:59 AM Amir Goldstein <[email protected]> wrote:
> >
> > On Mon, Apr 20, 2026 at 8:31 AM Chenglong Tang <[email protected]> wrote:
> > >
> > > Hi, Amir,
> > >
> > > Thanks for looking into this! To answer your questions:
> > >
> > > 1. Production vs. Test Suite Impact
> > >
> > > The immediate failure we encountered is in containerd's integration
> > > test suite (TestImageVolumeCheckVolatileOption). The test explicitly
> > > reads /proc/mounts and expects the exact string "volatile".
> > >
> > > In default production, containerd passes the legacy "volatile" string
> > > to the mount syscall, which your patch correctly handles under the
> > > hood. So the standard "happy path" is not broken in production.
> > >
> > > 2. The purpose of WithTempMount() / RemoveVolatileOption
> > >
> > > Containerd regularly makes temporary overlay mounts (e.g., for
> > > unpacking layers). Because overlayfs rejects reusing upper/work dirs
> > > from a volatile mount, containerd uses RemoveVolatileOption to strip
> > > the volatile flag before these temporary mounts.
> > >
> > > Currently, containerd's RemoveVolatileOption does an exact string
> > > match for "volatile". While it works for the default path, there is a
> > > production edge case: if a user explicitly configures their container
> > > runtime to use the new "fsync=volatile" option, older containerd
> > > binaries will fail to strip it, and the temporary mounts will be
> > > rejected by the kernel.
> >
> > I need to challenge this specific argument because I do not agree
> > that this edge case could be considered a regression at all.
> >
> > An admin from the past could not have set an explicit
> > "fsync=volatile" mount option.
> >
> > Though experiment - overlayfs adds a new mode fsync=off
> > which is more loose than "volatile" because "volatile" can actually
> > return error on fsync in some cases.
> >
> > Overlayfs would also not allow to reuse workdir from such
> > a mount.
> >
> > So would you then claim that adding the new mount option
> > "fsync=off" is a regression because of the edge case that an admin
> > decided to explicitly add "fsync=off" and RemoveVolatileOption()
> > does not handle it.
> >
> > I don't buy it.
> >
> > There is a more correct way to handle this situation and it is
> > documented in overlayfs.rst:
> >
> > "When overlay is mounted with "volatile" option, the directory
> > "$workdir/work/incompat/volatile" is created.  During next mount, overlay
> > checks for this directory and refuses to mount if present. This is a strong
> > indicator that the user should discard upper and work directories and create
> > fresh ones. In very limited cases where the user knows that the system has
> > not crashed and contents of upperdir are intact, the "volatile" directory
> > can be removed."
> >
> > containerd unpacking layers falls under this very limited case.
> > containerd can syncfs() workdir after unpack & unmount of temp
> > overlayfs and remove the "incompat/volatile" directory and then
> > the upperdir/workdir are are free to be remounted.
> >
> > >
> > > Conclusion
> > >
> > > While containerd could theoretically patch their code to accept
> > > strings.Contains() or fsync=volatile going forward, there are many
> >
> > Following the documentation advise would be better.
> >
> > > existing containerd binaries in the wild. Given that this patch breaks
> > > containerd's CI tests and introduces an edge case for
> > > RemoveVolatileOption, it might be safest to fix ovl_show_options in
> > > the kernel to continue outputting the legacy "volatile" string to
> > > strictly guarantee backwards compatibility with userspace.
> > >
> >
> > Considering my comments above, I think we have not yet reached
> > the point where the backward compat "volatile" string is called for,
> > but with other reports from production workloads, we could get there.
> >
> > Thanks for the clear and honest explanations of the containerd situation!
> > Amir.