Re: [PATCH] ovl: make fsync after metadata copy-up opt-in mount option

[CC ext4]

On Fri, Mar 27, 2026 at 6:45 AM Christoph Hellwig <[email protected]> wrote:
>
> On Wed, Mar 25, 2026 at 02:11:31PM +0100, Amir Goldstein wrote:
> > When an overlayfs file is modified for the first time, copy up will
> > create a copy of the lower file and its parent directories in the upper
> > layer.  Since the Linux filesystem API does not enforce any particular
> > ordering on storing changes without explicit fsync(2) calls, in case
> > of a system crash, the upper file could end up with no data at all
> > (i.e. zeros), which would be an unusual outcome.  To avoid this
> > experience, overlayfs calls fsync(2) on the upper file before completing
> > data copy up with rename(2) to make the copy up "atomic".
>
> Sounds good so far.
>
> > By default, overlayfs does not call fsync(2) on copied up directories,
> > so after a crash, a copied up directory could be observed in the upper
> > layer without some of its attributes.
>
> This does sound a bit scary.  How does a directory copy up work?
> mkdir + adding the copies up entries, probably with some chmod or
> chown thrown in?
>

Don't worry, there is no attempt to implement "directory content copy up"

There is only "directory inode copy up"
or in a more generic description there is:
1. "inode metadata copy up" - attributes, xattr and some fileattr
2. "inode data copy up"

Copy up of directory a is:
mkdir workdir/tmpdir
set attrs on workdir/tmpdir
fsync workdir/tmpdir (fsync == strict)
mv workdir/tmpdir upperdir/a

Copy up of a/b/c/file is
copy up a (if needed)
copy up a/b (if needed)
copy up a/b/c (if needed)
open O_TMPFILE
write data to tmpfile
fsync tmpfile (fsync != volatile)
set attr on tmpfile
fsync tmpfile (fsync == strict)
link tmpfile to upperdir/a/b/c/file

But note that the trigger to copy up is file data or metadata modification.
Overlayfs provides no guarantee to persist the modification unless
user does fsync themselves.

Overlayfs only provides the guarantee that if the copy up is observed,
the observed data is not zeros because data is synced before the link(2).

> > - "ordered": (default)
> >     Call fsync(2) on upper file before completion of data copy up.
> >     No fsync(2) is called on directory or metadata-only copy up.
>
> "ordered" sounds like an odd name here.  It's more like lazy or

The inspiration is the journal=orderded mode which provides
similar guarantee to ext4 (after the delalloc mitigation) -
no zeros observed after write+rename even without  explicit fsync.

> "nodirfsync".  And it might help to explain what this implies, which
> is that the fsync on the files in the directory also sync the
> directories out, because they are usually modified in the same
> transaction, and a traditional simple log model implies that.  That
> traditional single log model also implies that you get the metadata
> file fsync for free in that case.  I.e. if you did:

I wish to avoid a naming discussion, so this is going to be fsync=auto
and documentation will elaborate on what it does.
See rephrased doc below.

>
>         for each file:
>                 sync_file_range(file, .., SYNC_FILE_RANGE_WRITE |
>                                     SYNC_FILE_RANGE_WAIT_AFTER);
>
>         fsync(dir)
>         for each file:
>                 fsync(file)

This doesn't happen but I get what you mean.

>
> at least for xfs (and probably the others) you should get the
> performance of your ordered mode with the durability guarantees
> of the strict version.
>

Honestly, we did not think that adding fsync on the parent dirs
would impact performance so much, that is why we did not
do this opt-in to begin with.

My guess is that ext4 fell from a fast commit workload to
non-fast commit workload due to this change.

If ext4 developers want to investigate, then may do so with the fsync=strict
mount option. The regression report is from Google COS so...

I just want to make this ovl behavior change opt-in because I do not
want any more surprises from any other upper fs.

Thanks,
Amir.

Durability and copy up
----------------------

The fsync(2) system call ensures that the data and metadata of a file
are safely written to the backing storage, which is expected to
guarantee the existence of the information post system crash.

Without an fsync(2) call, there is no guarantee that the observed
data after a system crash will be either the old or the new data, but
in practice, the observed data after crash is often the old or new data
or a mix of both.

When an overlayfs file is modified for the first time, copy up will
create a copy of the lower file and its parent directories in the upper
layer.  Since the Linux filesystem API does not enforce any particular
ordering on storing changes without explicit fsync(2) calls, in case
of a system crash, the upper file could end up with no data at all
(i.e. zeros), which would be an unusual outcome.  To avoid this
experience, overlayfs calls fsync(2) on the upper file before completing
data copy up with rename(2) or link(2) to make the copy up "atomic".

By default, overlayfs does not explicitly call fsync(2) on copied up
directories or on metadata-only copy up, so it provides no guarantee to
persist the user's modification unless the user calls fsync(2).
The fsync during copy up only guarantees that if a copy up is observed
after a crash, the observed data is not zeroes or intermediate values
from the copy up staging area.

On traditional local filesystems with a single journal (e.g. ext4, xfs),
fsync on a file also persists the parent directory changes, because they
are usually modified in the same transaction, so metadata durability during
data copy up effectively comes for free.  Overlayfs further limits risk by
disallowing network filesystems as upper layer.

Overlayfs can be tuned to prefer performance or durability when storing
to the underlying upper layer.  This is controlled by the "fsync" mount
option, which supports these values:

- "auto": (default)
    Call fsync(2) on upper file before completion of data copy up.
    No explicit fsync(2) on directory or metadata-only copy up.
- "strict":
    Call fsync(2) on upper file and directories before completion of any
    copy up.
- "volatile": [*]
    Prefer performance over durability (see `Volatile mount`_)

[*] The mount option "volatile" is an alias to "fsync=volatile".