Re: [PATCH] fuse: disable default bdi strictlimiting

On Mon 13-07-26 18:13:29, Joanne Koong wrote:
> On Sat, May 30, 2026 at 4:04 AM Jan Kara <[email protected]> wrote:
> >
> > On Thu 28-05-26 15:11:18, Joanne Koong wrote:
> > > On Thu, May 28, 2026 at 5:34 AM Jan Kara <[email protected]> wrote:
> > > > > I think this is also going to be a problem for cgroups with large
> > > > > folios since they also, as I understand it, are constrained with a
> > > > > limited / tight dirty budget. I ran some initial benchmarks with
> > > > > cgroup memory constraints on NVMe and saw similar instability (a
> > > > > single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
> > > > > 15 ms without the cgroup, with the balance_dirty_pages() throttling
> > > > > oscillating rather than settling near the set point).
> > > >
> > > > Ok, so this is more folios (819) than my 512 gut feeling estimate :) What
> > > > was the write throughput of the NVMe drive? The high drive throughput also
> > > > requires more dirty data to keep the drive saturated so that writeback
> > > > throughput doesn't oscilate too much.
> > >
> > > The write throughput of the NVMe drive I was using was around ~1.1
> > > Gb/s (measured by running direct I/O). I think with the 1.6GB dirty
> > > budget, the math for that ends up being that the device drains it in
> > > ~1.5 secs. The performance I was seeing with the initial cgroup
> > > benchmarks was with 4k pages (no large folios enabled) on btrfs.
> >
> > OK, you might want to experiment with some other filesystem (I suggest xfs
> > or ext4) as well. Btrfs writeback behavior is a bit special with its data
> > checksum computations etc. and thus latency of starting writeback. It could
> > contribute to the erratic behavior with the relatively low dirty limits.
> 
> I reran the benchmarks on xfs and ext4 and saw similiar results. The
> max write latency under the memcg were
> xfs        5.1s
> ext4      5.2 s
> btrfs      6.2 s
> 
> compared to the non-memcg case (0.4 to 2.0 ms). This didn't affect
> throughput though, just the tail latency. When I ran xfs with large
> folios disabled, I still saw ~4.1s.

OK, interesting. Thanks for running these tests! Also it is good to know
this is not really related to large folios, that somewhat simplifies
matters.

> From the balance_dirty_pages() ftrace tracepoints, the dirty_ratelimit
> value is consistently stable / accurate but it looks like what's
> happening is that during freerun, the writer essentially dirties at
> memcpy speed until the freerun ceiling, and then the soft/proportional
> throttling kicks in but doesn't kick in fast or hard enough, which
> allows the number of dirty pages to exceed the hard limit by ~2x, and
> then at that point the writer is forced into the loop where it sleeps
> max_pause (200ms) each iteration until writeback has drained the
> number of dirty pages back under the limit.

OK, above you mentioned that the dirty limit for the memcgs is set at
1.6GB. Does that mean that dirty throttling allows memcg to dirty up to
~3.2GB of pages? I wouldn't have expected that...

Also you mentioned that the disk you are using has writeback throughput of
~1.1GB/s. Even with 3.2GB of pages we should get below the dirty limit in
2s and not 5. So there's something going on I don't understand yet :)

BTW how much memory does the machine have in total?

> If we wanted to fix this, it seems like we could for
> memory-constrained memcgs either start soft/proportional throttling
> earlier or make it throttle more aggressively as dirty approaches the
> limit, or start writeback sooner so it finishes draining the dirty
> pages faster. When I experimented with lowering
> dirty_background_ratio, the worst-case latency dropped around 2-3x
> (~6s to 2s) but there was still multi-second tail latencies. Maybe
> this indicates more that the throttling needs to happen earlier/harder
> than that writeback needs to happen earlier.

Hum, can you share the script you are using for the benchmark? I can try
running in on my machine and see what happens. If I'm able to reproduce
these long waits, I want to do a bit of tracing to understand what's
happening with various writeback statistics. Because these large tail
latencies aren't really expected. The original design goal was to keep
maximum write latency (for fast enough disks) around 0.2s.

								Honza
-- 
Jan Kara <[email protected]>
SUSE Labs, CR