[ANNOUNCE] oans: duperemove fork; FIDEDUPERANGE cost under memory pressure

Hi,

I've been running offline dedup on a large btrfs tree on a schedule, and
while profiling duperemove there I found something I didn't expect: once
the working set exceeds RAM, hashing is no longer the bottleneck. The
dedupe phase is, and by a wide margin.

FIDEDUPERANGE byte-compares every candidate range before sharing it, so
the data is read twice: once by userspace to hash it, once by the kernel
to verify it. When the tree fits in the page cache the second read is
free. When it doesn't, the compare falls back to cold reads and ends up
dominating the run.

Measured on two non-reflinked copies of a Linux kernel tree (189546
files, ~10.5 GiB), deduped inside a hard 4 GiB cgroup limit so eviction
actually happens, 10 interleaved rounds, median wall time:

  hash + dedupe   13.8 s     vs   179.7 s
  peak RSS         121 MiB   vs    243 MiB

The second column is duperemove v0.15.2, the first is my fork. Both read
the same 16.7 GiB, and "btrfs filesystem du -s" reports an identical
on-disk layout afterwards, so the work is the same and only the read
pattern differs.

What the fork does is read each dedupe round's source and destination
ranges (at most 32 MiB each) sequentially in userspace immediately
before issuing the ioctl, so the kernel's compare hits warm page cache.
It also stops calling POSIX_FADV_DONTNEED on data it just hashed when a
dedupe is going to follow.

My hypothesis for why this helps at all, given the kernel would read the
same pages anyway, is that one sequential userspace read of a 32 MiB
range is much friendlier to the device than the compare loop faulting
pages in as it walks two files in lockstep. I have not verified that
against the kernel side, and I would welcome a correction if the real
mechanism is something else.

Two questions I would genuinely like input on:

1. Is the cold read path in the in-kernel compare something that could
   reasonably be improved in btrfs, e.g. with readahead over the two
   ranges before comparing? Doing it in userspace works but feels like
   the wrong layer.

2. Are there workloads where prefetching like this would hurt, for
   instance on a system already under memory pressure from something
   else, where pulling 64 MiB into the page cache per round evicts
   something more useful?

The fork is at https://github.com/martinus/oans (GPL, same as
duperemove). It carries the benchmark methodology, the raw rounds and a
script to reproduce the numbers above. Full credit to Mark Fasheh and
the duperemove contributors; it is still their engine underneath, and
the fixes I found along the way have been sent upstream as PRs.

For completeness: much of the fork was written with AI assistance
reviewed and benchmarked on real data before landing. Dedup
still goes through FIDEDUPERANGE, so a userspace bug there can waste
work or miss a dedup but cannot make the kernel share data that is not
identical.

Thanks,
Martin