Re: [PATCH] fuse: disable default bdi strictlimiting

On Sat, May 30, 2026 at 4:04 AM Jan Kara <[email protected]> wrote:
>
> On Thu 28-05-26 15:11:18, Joanne Koong wrote:
> > On Thu, May 28, 2026 at 5:34 AM Jan Kara <[email protected]> wrote:
> > > > I think this is also going to be a problem for cgroups with large
> > > > folios since they also, as I understand it, are constrained with a
> > > > limited / tight dirty budget. I ran some initial benchmarks with
> > > > cgroup memory constraints on NVMe and saw similar instability (a
> > > > single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
> > > > 15 ms without the cgroup, with the balance_dirty_pages() throttling
> > > > oscillating rather than settling near the set point).
> > >
> > > Ok, so this is more folios (819) than my 512 gut feeling estimate :) What
> > > was the write throughput of the NVMe drive? The high drive throughput also
> > > requires more dirty data to keep the drive saturated so that writeback
> > > throughput doesn't oscilate too much.
> >
> > The write throughput of the NVMe drive I was using was around ~1.1
> > Gb/s (measured by running direct I/O). I think with the 1.6GB dirty
> > budget, the math for that ends up being that the device drains it in
> > ~1.5 secs. The performance I was seeing with the initial cgroup
> > benchmarks was with 4k pages (no large folios enabled) on btrfs.
>
> OK, you might want to experiment with some other filesystem (I suggest xfs
> or ext4) as well. Btrfs writeback behavior is a bit special with its data
> checksum computations etc. and thus latency of starting writeback. It could
> contribute to the erratic behavior with the relatively low dirty limits.

I reran the benchmarks on xfs and ext4 and saw similiar results. The
max write latency under the memcg were
xfs        5.1s
ext4      5.2 s
btrfs      6.2 s

compared to the non-memcg case (0.4 to 2.0 ms). This didn't affect
throughput though, just the tail latency. When I ran xfs with large
folios disabled, I still saw ~4.1s.

From the balance_dirty_pages() ftrace tracepoints, the dirty_ratelimit
value is consistently stable / accurate but it looks like what's
happening is that during freerun, the writer essentially dirties at
memcpy speed until the freerun ceiling, and then the soft/proportional
throttling kicks in but doesn't kick in fast or hard enough, which
allows the number of dirty pages to exceed the hard limit by ~2x, and
then at that point the writer is forced into the loop where it sleeps
max_pause (200ms) each iteration until writeback has drained the
number of dirty pages back under the limit.

If we wanted to fix this, it seems like we could for
memory-constrained memcgs either start soft/proportional throttling
earlier or make it throttle more aggressively as dirty approaches the
limit, or start writeback sooner so it finishes draining the dirty
pages faster. When I experimented with lowering
dirty_background_ratio, the worst-case latency dropped around 2-3x
(~6s to 2s) but there was still multi-second tail latencies. Maybe
this indicates more that the throttling needs to happen earlier/harder
than that writeback needs to happen earlier.

I'm not sure how much this matters in practice though, given that no
one has complained about it.

Ironically, for fuse there is not this problem because with
strictlimiting, it makes the dirty budget so small that even with
exceeding the hard limit by ~2x, there's not that many bytes to write
back so there's not that much latency waiting for it to finish, and
strictlimiting also inherently starts the soft/proportional throttling
more early/aggressively, which prevents too much overrun exceeding the
limit.

Thanks,
Joanne