[PATCH] erofs: weaken full memory barriers to r/w barriers

From: Bingwu Zhang <[email protected]>

fs/erofs/{xattr.c,zmap.c} both implemented one-time initialization locks
with general memory barriers.

Weaken these unnecessarily-strong general memory barriers to SMP
(ordering) read/write barriers for better performance on weakly-ordered
architectures.

Signed-off-by: Bingwu Zhang <[email protected]>
---
(All test results below are done using QEMU with KVM; guest VM root FS
are mounted with v9fs; test target is an EROFS mounted on a loop device
backed by a file in tmpfs, in attempt to reduce error caused by physical
disks; mkfs flags: --zD=1 --all-root -zzstd -E48bit)

On x86-64, with x86-TSO, a full barrier lowers to a fence instruction and
read/write barriers are just compiler barriers. This patch removes
several fence instructions although they don't affect performance much.

On weakly-ordered architectures, a full memory barrier could be very
expensive. Thus, on these platforms, this patch saves more performance
than x86-64.

Test result on x86-64:
(host: AMD Ryzen 7 255 (@ 3.3GHz))
(guest VM: ArchLinux, test data: linux.git/drivers @ commit fc02acf6ac0c)
  Original:
    read: IOPS=1684, BW=6737KiB/s (6899kB/s)(1006MiB/152838msec)
    clat (usec): min=25, max=26293, avg=578.58, stdev=426.55
     lat (usec): min=25, max=26293, avg=578.70, stdev=426.56
   bw (  KiB/s): min= 1976, max=10904, per=100.00%, avg=6749.91,
      stdev=1883.80, samples=305
   iops        : min=  494, max= 2726, avg=1687.48, stdev=470.95, samples=305

  New:
   read: IOPS=1687, BW=6751KiB/s (6913kB/s)(1006MiB/152513msec)
    clat (usec): min=25, max=19136, avg=577.00, stdev=425.66
     lat (usec): min=25, max=19136, avg=577.09, stdev=425.67
   bw (  KiB/s): min= 2192, max=11088, per=100.00%, avg=6770.26
      stdev=1870.55, samples=304
   iops        : min=  548, max= 2772, avg=1692.44, stdev=467.65, samples=304

IOPS +0.17%, bandwidth +0.20%, max completion latency -27.22%

Test result on ARM64:
(host: HUAWEI Kunpeng 920 @ 2.6GHz, both host & VM are AOSC OS)
(test data:
1. AOSC OS: aosc-os_base_20260621_arm64.squashfs, /usr/bin directory is
   repacked as EROFS.
2. linux.git at commit fc02acf6ac0c, drivers/ subdirectory
)
  Original:
   read: IOPS=12.3k, BW=47.9MiB/s (50.2MB/s)(1291MiB/26950msec)
    clat (nsec): min=910, max=37500k, avg=56026.68, stdev=198628.59
     lat (nsec): min=960, max=37500k, avg=56167.70, stdev=198639.09
   bw (  KiB/s): min= 3888, max=94328, per=100.00%, avg=49790.64,
      stdev=31594.09, samples=53
   iops        : min=  972, max=23582, avg=12447.57, stdev=7898.51, samples=53

  New:
   read: IOPS=13.8k, BW=54.0MiB/s (56.6MB/s)(1315MiB/24375msec)
    clat (nsec): min=930, max=34271k, avg=53069.97, stdev=145651.76
     lat (nsec): min=980, max=34271k, avg=53221.68, stdev=145661.24
   bw (  KiB/s): min= 5843, max=95054, per=100.00%, avg=55988.58,
      stdev=30477.44, samples=48
   iops        : min= 1460, max=23763, avg=13997.00, stdev=7619.37, samples=48

IOPS min +50.20%, avg +12.44%
bandwidth min +50.28%, max +0.76%, avg +12.44%
completion latency max -8.61%, avg -5.27%

I am not pretty sure why the minimum completion latencies vary so much on
AArch64. Even when I repeat the test with the same kernel without
rebooting/re-mounting, clat min fluctuates randomly, sometimes spiking to
over 1000 and sometimes dropping to as low as 2, but I think this patch
still improves the overall throughput performance.
---
 fs/erofs/xattr.c | 8 ++++----
 fs/erofs/zmap.c  | 8 ++++----
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/fs/erofs/xattr.c b/fs/erofs/xattr.c
index df7ea019526d..0b252cfe616a 100644
--- a/fs/erofs/xattr.c
+++ b/fs/erofs/xattr.c
@@ -44,10 +44,10 @@ static int erofs_init_inode_xattrs(struct inode *inode)
 	/* the most case is that xattrs of this inode are initialized. */
 	if (test_bit(EROFS_I_EA_INITED_BIT, &vi->flags)) {
 		/*
-		 * paired with smp_mb() at the end of the function to ensure
+		 * paired with smp_wmb() at the end of the function to ensure
 		 * fields will only be observed after the bit is set.
 		 */
-		smp_mb();
+		smp_rmb();
 		return 0;
 	}
 	if (wait_on_bit_lock(&vi->flags, EROFS_I_BL_XATTR_BIT, TASK_KILLABLE))
@@ -111,8 +111,8 @@ static int erofs_init_inode_xattrs(struct inode *inode)
 		vi->xattr_shared_xattrs[i] = le32_to_cpu(*xattr_id);
 	}
 
-	/* paired with smp_mb() at the beginning of the function. */
-	smp_mb();
+	/* paired with smp_rmb() at the beginning of the function. */
+	smp_wmb();
 	set_bit(EROFS_I_EA_INITED_BIT, &vi->flags);
 out_unlock:
 	erofs_put_metabuf(&buf);
diff --git a/fs/erofs/zmap.c b/fs/erofs/zmap.c
index 5811556a7b71..d5b001a537f4 100644
--- a/fs/erofs/zmap.c
+++ b/fs/erofs/zmap.c
@@ -628,10 +628,10 @@ static int z_erofs_fill_inode(struct inode *inode, struct erofs_map_blocks *map)
 
 	if (test_bit(EROFS_I_Z_INITED_BIT, &vi->flags)) {
 		/*
-		 * paired with smp_mb() at the end of the function to ensure
+		 * paired with smp_wmb() at the end of the function to ensure
 		 * fields will only be observed after the bit is set.
 		 */
-		smp_mb();
+		smp_rmb();
 		return 0;
 	}
 
@@ -704,8 +704,8 @@ static int z_erofs_fill_inode(struct inode *inode, struct erofs_map_blocks *map)
 			goto out_unlock;
 	}
 done:
-	/* paired with smp_mb() at the beginning of the function */
-	smp_mb();
+	/* paired with smp_rmb() at the beginning of the function */
+	smp_wmb();
 	set_bit(EROFS_I_Z_INITED_BIT, &vi->flags);
 out_unlock:
 	clear_and_wake_up_bit(EROFS_I_BL_Z_BIT, &vi->flags);

---
base-commit: 11028ab62899e4191e074ee364c712b77823a9c4
change-id: 20260729-erofs-weaken-dbar-9fd2edff90e5

Best regards,
--  
Bingwu Zhang <[email protected]>