Re: [Cluster-devel] [PATCH v7 12/13] ext4: switch to multigrain timestamps

On Wed, 2023-09-20 at 12:17 +0200, Jan Kara wrote:
> On Wed 20-09-23 10:41:30, Christian Brauner wrote:
> > > > f1 was last written to *after* f2 was last written to. If the times=
tamp of f1
> > > > is then lower than the timestamp of f2, timestamps are fundamentall=
y broken.
> > > >=20
> > > > Many things in user-space depend on timestamps, such as build syste=
m
> > > > centered around 'make', but also 'find ... -newer ...'.
> > > >=20
> > >=20
> > >=20
> > > What does breakage with make look like in this situation? The "fuzz"
> > > here is going to be on the order of a jiffy. The typical case for mak=
e
> > > timestamp comparisons is comparing source files vs. a build target. I=
f
> > > those are being written nearly simultaneously, then that could be an
> > > issue, but is that a typical behavior? It seems like it would be hard=
 to
> > > rely on that anyway, esp. given filesystems like NFS that can do lazy
> > > writeback.
> > >=20
> > > One of the operating principles with this series is that timestamps c=
an
> > > be of varying granularity between different files. Note that Linux
> > > already violates this assumption when you're working across filesyste=
ms
> > > of different types.
> > >=20
> > > As to potential fixes if this is a real problem:
> > >=20
> > > I don't really want to put this behind a mount or mkfs option (a'la
> > > relatime, etc.), but that is one possibility.
> > >=20
> > > I wonder if it would be feasible to just advance the coarse-grained
> > > current_time whenever we end up updating a ctime with a fine-grained
> > > timestamp? It might produce some inode write amplification. Files tha=
t
> >=20
> > Less than ideal imho.
> >=20
> > If this risks breaking existing workloads by enabling it unconditionall=
y
> > and there isn't a clear way to detect and handle these situations
> > without risk of regression then we should move this behind a mount
> > option.
> >=20
> > So how about the following:
> >=20
> > From cb14add421967f6e374eb77c36cc4a0526b10d17 Mon Sep 17 00:00:00 2001
> > From: Christian Brauner <[email protected]>
> > Date: Wed, 20 Sep 2023 10:00:08 +0200
> > Subject: [PATCH] vfs: move multi-grain timestamps behind a mount option
> >=20
> > While we initially thought we can do this unconditionally it turns out
> > that this might break existing workloads that rely on timestamps in ver=
y
> > specific ways and we always knew this was a possibility. Move
> > multi-grain timestamps behind a vfs mount option.
> >=20
> > Signed-off-by: Christian Brauner <[email protected]>
>=20
> Surely this is a safe choice as it moves the responsibility to the sysadm=
in
> and the cases where finegrained timestamps are required. But I kind of
> wonder how is the sysadmin going to decide whether mgtime is safe for his
> system or not? Because the possible breakage needn't be obvious at the
> first sight...
>=20

That's the main reason I really didn't want to go with a mount option.
Documenting that may be difficult. While there is some pessimism around
it, I may still take a stab at just advancing the coarse clock whenever
we fetch a fine-grained timestamp. It'd be nice to remove this option in
the future if that turns out to be feasible.

> If I were a sysadmin, I'd rather opt for something like
> finegrained timestamps + lazytime (if I needed the finegrained timestamps
> functionality). That should avoid the IO overhead of finegrained timestam=
ps
> as well and I'd know I can have problems with timestamps only after a
> system crash.

> I've just got another idea how we could solve the problem: Couldn't we
> always just report coarsegrained timestamp to userspace and provide acces=
s
> to finegrained value only to NFS which should know what it's doing?
>=20

I think that'd be hard. First of all, where would we store the second
timestamp? We can't just truncate the fine-grained ones to come up with
a coarse-grained one. It might also be confusing having nfsd and local
filesystems present different attributes.


> > ---
> >  fs/fs_context.c     | 18 ++++++++++++++++++
> >  fs/inode.c          |  4 ++--
> >  fs/proc_namespace.c |  1 +
> >  fs/stat.c           |  2 +-
> >  include/linux/fs.h  |  4 +++-
> >  5 files changed, 25 insertions(+), 4 deletions(-)
> >=20
> > diff --git a/fs/fs_context.c b/fs/fs_context.c
> > index a0ad7a0c4680..dd4dade0bb9e 100644
> > --- a/fs/fs_context.c
> > +++ b/fs/fs_context.c
> > @@ -44,6 +44,7 @@ static const struct constant_table common_set_sb_flag=
[] =3D {
> >  =09{ "mand",=09SB_MANDLOCK },
> >  =09{ "ro",=09=09SB_RDONLY },
> >  =09{ "sync",=09SB_SYNCHRONOUS },
> > +=09{ "mgtime",=09SB_MGTIME },
> >  =09{ },
> >  };
> > =20
> > @@ -52,18 +53,32 @@ static const struct constant_table common_clear_sb_=
flag[] =3D {
> >  =09{ "nolazytime",=09SB_LAZYTIME },
> >  =09{ "nomand",=09SB_MANDLOCK },
> >  =09{ "rw",=09=09SB_RDONLY },
> > +=09{ "nomgtime",=09SB_MGTIME },
> >  =09{ },
> >  };
> > =20
> > +static inline int check_mgtime(unsigned int token, const struct fs_con=
text *fc)
> > +{
> > +=09if (token !=3D SB_MGTIME)
> > +=09=09return 0;
> > +=09if (!(fc->fs_type->fs_flags & FS_MGTIME))
> > +=09=09return invalf(fc, "Filesystem doesn't support multi-grain timest=
amps");
> > +=09return 0;
> > +}
> > +
> >  /*
> >   * Check for a common mount option that manipulates s_flags.
> >   */
> >  static int vfs_parse_sb_flag(struct fs_context *fc, const char *key)
> >  {
> >  =09unsigned int token;
> > +=09int ret;
> > =20
> >  =09token =3D lookup_constant(common_set_sb_flag, key, 0);
> >  =09if (token) {
> > +=09=09ret =3D check_mgtime(token, fc);
> > +=09=09if (ret)
> > +=09=09=09return ret;
> >  =09=09fc->sb_flags |=3D token;
> >  =09=09fc->sb_flags_mask |=3D token;
> >  =09=09return 0;
> > @@ -71,6 +86,9 @@ static int vfs_parse_sb_flag(struct fs_context *fc, c=
onst char *key)
> > =20
> >  =09token =3D lookup_constant(common_clear_sb_flag, key, 0);
> >  =09if (token) {
> > +=09=09ret =3D check_mgtime(token, fc);
> > +=09=09if (ret)
> > +=09=09=09return ret;
> >  =09=09fc->sb_flags &=3D ~token;
> >  =09=09fc->sb_flags_mask |=3D token;
> >  =09=09return 0;
> > diff --git a/fs/inode.c b/fs/inode.c
> > index 54237f4242ff..fd1a2390aaa3 100644
> > --- a/fs/inode.c
> > +++ b/fs/inode.c
> > @@ -2141,7 +2141,7 @@ EXPORT_SYMBOL(current_mgtime);
> > =20
> >  static struct timespec64 current_ctime(struct inode *inode)
> >  {
> > -=09if (is_mgtime(inode))
> > +=09if (IS_MGTIME(inode))
> >  =09=09return current_mgtime(inode);
> >  =09return current_time(inode);
> >  }
> > @@ -2588,7 +2588,7 @@ struct timespec64 inode_set_ctime_current(struct =
inode *inode)
> >  =09=09now =3D current_time(inode);
> > =20
> >  =09=09/* Just copy it into place if it's not multigrain */
> > -=09=09if (!is_mgtime(inode)) {
> > +=09=09if (!IS_MGTIME(inode)) {
> >  =09=09=09inode_set_ctime_to_ts(inode, now);
> >  =09=09=09return now;
> >  =09=09}
> > diff --git a/fs/proc_namespace.c b/fs/proc_namespace.c
> > index 250eb5bf7b52..08f5bf4d2c6c 100644
> > --- a/fs/proc_namespace.c
> > +++ b/fs/proc_namespace.c
> > @@ -49,6 +49,7 @@ static int show_sb_opts(struct seq_file *m, struct su=
per_block *sb)
> >  =09=09{ SB_DIRSYNC, ",dirsync" },
> >  =09=09{ SB_MANDLOCK, ",mand" },
> >  =09=09{ SB_LAZYTIME, ",lazytime" },
> > +=09=09{ SB_MGTIME, ",mgtime" },
> >  =09=09{ 0, NULL }
> >  =09};
> >  =09const struct proc_fs_opts *fs_infop;
> > diff --git a/fs/stat.c b/fs/stat.c
> > index 6e60389d6a15..2f18dd5de18b 100644
> > --- a/fs/stat.c
> > +++ b/fs/stat.c
> > @@ -90,7 +90,7 @@ void generic_fillattr(struct mnt_idmap *idmap, u32 re=
quest_mask,
> >  =09stat->size =3D i_size_read(inode);
> >  =09stat->atime =3D inode->i_atime;
> > =20
> > -=09if (is_mgtime(inode)) {
> > +=09if (IS_MGTIME(inode)) {
> >  =09=09fill_mg_cmtime(stat, request_mask, inode);
> >  =09} else {
> >  =09=09stat->mtime =3D inode->i_mtime;
> > diff --git a/include/linux/fs.h b/include/linux/fs.h
> > index 4aeb3fa11927..03e415fb3a7c 100644
> > --- a/include/linux/fs.h
> > +++ b/include/linux/fs.h
> > @@ -1114,6 +1114,7 @@ extern int send_sigurg(struct fown_struct *fown);
> >  #define SB_NODEV        BIT(2)=09/* Disallow access to device special =
files */
> >  #define SB_NOEXEC       BIT(3)=09/* Disallow program execution */
> >  #define SB_SYNCHRONOUS  BIT(4)=09/* Writes are synced at once */
> > +#define SB_MGTIME=09BIT(5)=09/* Use multi-grain timestamps */
> >  #define SB_MANDLOCK     BIT(6)=09/* Allow mandatory locks on an FS */
> >  #define SB_DIRSYNC      BIT(7)=09/* Directory modifications are synchr=
onous */
> >  #define SB_NOATIME      BIT(10)=09/* Do not update access times. */
> > @@ -2105,6 +2106,7 @@ static inline bool sb_rdonly(const struct super_b=
lock *sb) { return sb->s_flags
> >  =09=09=09=09=09((inode)->i_flags & (S_SYNC|S_DIRSYNC)))
> >  #define IS_MANDLOCK(inode)=09__IS_FLG(inode, SB_MANDLOCK)
> >  #define IS_NOATIME(inode)=09__IS_FLG(inode, SB_RDONLY|SB_NOATIME)
> > +#define IS_MGTIME(inode)=09__IS_FLG(inode, SB_MGTIME)
> >  #define IS_I_VERSION(inode)=09__IS_FLG(inode, SB_I_VERSION)
> > =20
> >  #define IS_NOQUOTA(inode)=09((inode)->i_flags & S_NOQUOTA)
> > @@ -2366,7 +2368,7 @@ struct file_system_type {
> >   */
> >  static inline bool is_mgtime(const struct inode *inode)
> >  {
> > -=09return inode->i_sb->s_type->fs_flags & FS_MGTIME;
> > +=09return inode->i_sb->s_flags & SB_MGTIME;
> >  }
> > =20
> >  extern struct dentry *mount_bdev(struct file_system_type *fs_type,
> > --=20
> > 2.34.1
> >=20

--=20
Jeff Layton <[email protected]>