bug#80106: Don't sort dirents (in du) for Lustre

"Day, Timothy" via GNU coreutils Bug Reports <[email protected]>
writes:

> The gnulib function dirent_inode_sort_may_be_useful() should return
> false for Lustre (i.e. #define S_MAGIC_LUSTRE 0x0BD00BD0 as seen
> in lustre/include/uapi/linux/lustre/lustre_user.h in the Lustre source
> tree as LL_SUPER_MAGIC [1]). Sorting dirents negatively impacts du
> performance on Lustre because it interferes with Lustre's ability to
> prefetch file metadata (via statahead).

Thanks for the investigation.

I think the inode sorting is done any time there is no sort function
given to fts_open and there are 10000 or more entries. So it probably
affects these other programs as well:

     $ grep 'fts_open (.*, nullptr);' src/*.c
     src/chcon.c:  FTS *fts = xfts_open (files, bit_flags, nullptr);
     src/chmod.c:  FTS *fts = xfts_open (files, bit_flags, nullptr);
     src/chown-core.c:  FTS *fts = xfts_open (files, bit_flags | stat_flags, nullptr);
     src/du.c:      FTS *fts = xfts_open (files, bit_flags, nullptr);
     src/remove.c:      FTS *fts = xfts_open (file, bit_flags, nullptr);
     src/selinux.c:  FTS *fts = xfts_open ((char *const *) ftspath, FTS_PHYSICAL, nullptr);

> For context, Lustre is an open-source (GPLv2) out-of-tree Linux filesystem
> commonly used for HPC applications.
>
> Let me know if you need anymore context. Thanks!

Could you check coreutils built with this patch and see if it fixes the
performance? I can push it to Gnulib afterwards:

diff --git a/lib/fts.c b/lib/fts.c
index e7e0be5a98..a65e7ab771 100644
--- a/lib/fts.c
+++ b/lib/fts.c
@@ -652,6 +652,7 @@ enum leaf_optimization
 /* Linux-specific constants from coreutils' src/fs.h */
 # define S_MAGIC_AFS 0x5346414F
 # define S_MAGIC_CIFS 0xFF534D42
+# define S_MAGIC_LUSTRE 0x0BD00BD0
 # define S_MAGIC_NFS 0x6969
 # define S_MAGIC_PROC 0x9FA0
 # define S_MAGIC_TMPFS 0x1021994
@@ -760,6 +761,7 @@ dirent_inode_sort_may_be_useful (FTSENT const *p, int dir_fd)
     {
     case S_MAGIC_CIFS:
     case S_MAGIC_NFS:
+    case S_MAGIC_LUSTRE:
     case S_MAGIC_TMPFS:
       /* On a file system of any of these types, sorting
          is unnecessary, and hence wasteful.  */