bug#80106: Don't sort dirents (in du) for Lustre
Collin Funk <[email protected]> Thu, 01 Jan 2026 21:07:06 -0800
| Newsgroups | gmane.comp.gnu.core-utils.bugs |
|---|---|
| Message-ID | <[email protected]> |
"Day, Timothy" <[email protected]> writes: >>> For context, Lustre is an open-source (GPLv2) out-of-tree Linux filesystem >>> commonly used for HPC applications. >>> >>> Let me know if you need anymore context. Thanks! >> >>Could you check coreutils built with this patch and see if it fixes the >>performance? I can push it to Gnulib afterwards: > > Works perfectly: > > $ echo 3 | sudo tee /proc/sys/vm/drop_caches > 3 > $ time coreutils-9.9-unpatched/src/du /fsx/test_100K > /dev/null > > real 1m54.268s > user 0m0.229s > sys 0m14.136s > $ echo 3 | sudo tee /proc/sys/vm/drop_caches > 3 > $ time coreutils-9.9-patched/src/du /fsx/test_100K > /dev/null > > real 0m13.150s > user 0m0.075s > sys 0m1.956s > > That's around a 9x speedup. I haven't tested any other commands. > Lustre statahead (which is giving the performance boost) is designed > for "stat everything in a directory" type workloads. The other commands > wouldn't do that, I think. Perhaps chown would? Either way, thanks for > the help! Great, thanks for your help investigating! I think 'chown -R' and 'rm -r' would probably be affected as well: $ mkdir -p dir && (cd dir && mkdir $(seq 10000)) $ strace chown -R collin:collin dir 2>&1 > /dev/null | grep -Fc stat 20022 $ strace rm -r dir 2>&1 > /dev/null | grep -Fc stat 40011 Anyways, I pushed the Gnulib patch and updated the submodule in coreutils with the attached patch. Pádraig, I assume this warrants a NEWS entry under "Bug Fixes"? Collin
0001-build-update-gnulib-submodule-to-latest.patch
(text/x-patch, 1.9 KB)
From 3424ae3efa8b487f6327773e6f0554d1bfb33191 Mon Sep 17 00:00:00 2001 Message-ID: <3424ae3efa8b487f6327773e6f0554d1bfb33191.1767329553.git.collin.funk1@gmail.com> From: Collin Funk <[email protected]> Date: Thu, 1 Jan 2026 20:43:17 -0800 Subject: [PATCH] build: update gnulib submodule to latest * bootstrap: Update using './bootstrap --bootstrap-sync'. * tests/sample-test: Adjust to use the single most recent year. --- bootstrap | 4 ++-- gnulib | 2 +- tests/sample-test | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/bootstrap b/bootstrap index 29070675b..19f9424bb 100755 --- a/bootstrap +++ b/bootstrap @@ -5,7 +5,7 @@ scriptversion=2025-06-10.02; # UTC -# Copyright (C) 2003-2025 Free Software Foundation, Inc. +# Copyright (C) 2003-2026 Free Software Foundation, Inc. # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -39,7 +39,7 @@ medir=`dirname "$me"` scriptlibversion=2025-12-04.19; # UTC -# Copyright (C) 2003-2025 Free Software Foundation, Inc. +# Copyright (C) 2003-2026 Free Software Foundation, Inc. # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by diff --git a/gnulib b/gnulib index 594d98768..8e8991c41 160000 --- a/gnulib +++ b/gnulib @@ -1 +1 @@ -Subproject commit 594d9876806a796e669d08730dab03b6bfedb29a +Subproject commit 8e8991c4115ae0274d481144ae6e6c0754642fb4 diff --git a/tests/sample-test b/tests/sample-test index b9014a850..1a74bbd56 100644 --- a/tests/sample-test +++ b/tests/sample-test @@ -1,7 +1,7 @@ #!/bin/sh # FIXME -# Copyright (C) 2025-2026 Free Software Foundation, Inc. +# Copyright (C) 2026 Free Software Foundation, Inc. # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by -- 2.52.0