Re: [PATCH] fuse: disable default bdi strictlimiting

On Thu, Jul 16, 2026 at 8:12=E2=80=AFAM Shakeel Butt <[email protected]=
v> wrote:
>
> Hi Jan & Joanne,
>
> On Wed, Jul 15, 2026 at 06:28:16PM -0700, Joanne Koong wrote:
> > On Wed, Jul 15, 2026 at 3:32=E2=80=AFAM Jan Kara <[email protected]> wrote:
> > >
> > > [resending once more, this time with correct Shakeel's address]
> > >
> > > On Tue 14-07-26 18:21:57, Joanne Koong wrote:
> > > > On Tue, Jul 14, 2026 at 10:10=E2=80=AFAM Jan Kara <[email protected]> wr=
ote:
> > > > > On Mon 13-07-26 18:13:29, Joanne Koong wrote:
> > > > > > On Sat, May 30, 2026 at 4:04=E2=80=AFAM Jan Kara <[email protected]>=
 wrote:
> > > > > > > On Thu 28-05-26 15:11:18, Joanne Koong wrote:
> > > > > > > > On Thu, May 28, 2026 at 5:34=E2=80=AFAM Jan Kara <jack@suse=
.cz> wrote:
> > > > > > > > > > I think this is also going to be a problem for cgroups =
with large
> > > > > > > > > > folios since they also, as I understand it, are constra=
ined with a
> > > > > > > > > > limited / tight dirty budget. I ran some initial benchm=
arks with
> > > > > > > > > > cgroup memory constraints on NVMe and saw similar insta=
bility (a
> > > > > > > > > > single writer in a 8 GB cgroup had max write latencies =
of 6 seconds vs
> > > > > > > > > > 15 ms without the cgroup, with the balance_dirty_pages(=
) throttling
> > > > > > > > > > oscillating rather than settling near the set point).
> > > > > > > > >
> > > > > > > > > Ok, so this is more folios (819) than my 512 gut feeling =
estimate :) What
> > > > > > > > > was the write throughput of the NVMe drive? The high driv=
e throughput also
> > > > > > > > > requires more dirty data to keep the drive saturated so t=
hat writeback
> > > > > > > > > throughput doesn't oscilate too much.
> > > > > > > >
> > > > > > > > The write throughput of the NVMe drive I was using was arou=
nd ~1.1
> > > > > > > > Gb/s (measured by running direct I/O). I think with the 1.6=
GB dirty
> > > > > > > > budget, the math for that ends up being that the device dra=
ins it in
> > > > > > > > ~1.5 secs. The performance I was seeing with the initial cg=
roup
> > > > > > > > benchmarks was with 4k pages (no large folios enabled) on b=
trfs.
> > > > > > >
> > > > > > > OK, you might want to experiment with some other filesystem (=
I suggest xfs
> > > > > > > or ext4) as well. Btrfs writeback behavior is a bit special w=
ith its data
> > > > > > > checksum computations etc. and thus latency of starting write=
back. It could
> > > > > > > contribute to the erratic behavior with the relatively low di=
rty limits.
> > > > > >
> > > > > > I reran the benchmarks on xfs and ext4 and saw similiar results=
. The
> > > > > > max write latency under the memcg were
> > > > > > xfs        5.1s
> > > > > > ext4      5.2 s
> > > > > > btrfs      6.2 s
> > > > > >
> > > > > > compared to the non-memcg case (0.4 to 2.0 ms). This didn't aff=
ect
> > > > > > throughput though, just the tail latency. When I ran xfs with l=
arge
> > > > > > folios disabled, I still saw ~4.1s.
> > > > >
> > > > > OK, interesting. Thanks for running these tests! Also it is good =
to know
> > > > > this is not really related to large folios, that somewhat simplif=
ies
> > > > > matters.
> > > > >
> > > > > > From the balance_dirty_pages() ftrace tracepoints, the dirty_ra=
telimit
> > > > > > value is consistently stable / accurate but it looks like what'=
s
> > > > > > happening is that during freerun, the writer essentially dirtie=
s at
> > > > > > memcpy speed until the freerun ceiling, and then the soft/propo=
rtional
> > > > > > throttling kicks in but doesn't kick in fast or hard enough, wh=
ich
> > > > > > allows the number of dirty pages to exceed the hard limit by ~2=
x, and
> > > > > > then at that point the writer is forced into the loop where it =
sleeps
> > > > > > max_pause (200ms) each iteration until writeback has drained th=
e
> > > > > > number of dirty pages back under the limit.
> > > > >
> > > > > OK, above you mentioned that the dirty limit for the memcgs is se=
t at
> > > > > 1.6GB. Does that mean that dirty throttling allows memcg to dirty=
 up to
> > > > > ~3.2GB of pages? I wouldn't have expected that...
> > > >
> > > > On a couple of the runs, I saw it get as high as up to 5GB.
> > > >
> > > > I think this is because the balance_dirty_pages() code uses the mem=
cg
> > > > stats (NR_FILE_DIRTY) but these stats are only periodically
> > > > flushed/refreshed when reading it, so the balance dirty code is
> > > > seeing/using lagging/non-uptodate values.
> > > >
> > > > I'm seeing this in
> > > >   balance_dirty_pages()
> > > >       balance_domain_limits(mdtc,...)
> > > >           domain_dirty_avail()
> > > >                 mem_cgroup_wb_stats()
> > > >                        mem_cgroup_flush_stats_ratelimited()
> > > >
> > > > where mem_cgroup_flush_stats_ratelimted() has this logic:
> > > >
> > > > void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg)
> > > > {
> > > >         /* Only flush if the periodic flusher is one full cycle lat=
e */
> > > >         if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2=
*FLUSH_TIME))
> > > >                 mem_cgroup_flush_stats(memcg);
> > > > }
> > > >
> > > > where FLUSH_TIME is defined as 2UL * HZ, which afaict means the
> > > > flushing can be around 2 seconds stale (accounting for the periodic
> > > > flusher in flush_memcg_stats_dwork() that flushes every FLUSH_TIME
> > > > interval).
> > > >
> > > > From what I see, the freerun and hard limit checks use this stale
> > > > under-reported value (in domain_dirty_freerun() and
> > > > wb_position_ratio()), which means the writer gets to keep dirtying =
and
> > > > blow past the limit for up to 2 extra seconds before the throttling
> > > > limits it.
> > >
> > > I see. That would indeed explain why we can overrun the dirty limit s=
o
> > > much.
> > >
> > > > Maybe one idea is to in the memcg throttle path do the flush based =
on
> > > > how many pages are getting dirtied instead of on how much time has
> > > > elapsed? It looks like flushing grabs the rstat lock though, so may=
be
> > > > only doing it in the throttled case where it's past freerun would b=
e
> > > > best. It looks like the logic in the regular non-ratelimited
> > > > mem_cgroup_flush_stats() call does something similar. I can run som=
e
> > > > experiments this week with this if you think it could be promising.
> > >
> > > Checking some git history, this actually seems to be a relatively rec=
ent
> > > change from Shakeel (added to CC) - d9b3ce8769e3 ("mm: writeback: rat=
elimit
> > > stat flush from mem_cgroup_wb_stats"). So you can check whether rever=
ting
> > > that change makes dirty throttling not overshoot so much.
> > >
> >
> > Reverting d9b3ce8769e3 fixes the issue. On xfs, I'm now seeing the
> > number of dirty pages stays bounded by the limit instead of going 2x+
> > over, and the max latency is now around ~0.33s instead of ~4s.
> >
> > > Shakeel, your change to use mem_cgroup_flush_stats_ratelimited() in
> > > mem_cgroup_wb_stats() results in applications overshooting memcg dirt=
y
> > > limits by several gigabytes which for small memcgs (with 8GB memory i=
n this
> > > case, which means a dirty limit of ~1.5GB) is a considerable issue. I=
 guess
> > > the ratelimiting should take into account the size of memcg (or ideal=
ly the
> > > dirty limit in the case of mem_cgroup_wb_stats()) as well. Using
> > > somewhat outdated numbers for dirty throttling is fine but when we ca=
n
> > > overshoot the dirty limit more than three times, it leads to rather l=
ong
> > > stalls in dirty throttling and similar issues...
>
> Yeah I agree this is an issue.
>
> >
> > It looks like commit d9b3ce8769e3 was landed in Feb 2024 when rstat
> > was still using the single global lock. I see commit 748922dcfabd
> > ("cgroup: use subsystem-specific rstat locks to avoid contention")
> > later in the tree, which was landed around May 2025 and reduces that
> > global lock contention. That might make the cost of the flush a lot
> > lower than what it was when the ratelimit was needed in d9b3ce8769e3,
> > so maybe reverting d9b3ce8769e3 wouldn't make much of a difference for
> > write-heavy workloads and would be the simplest fix?
>
> Unfortunetely global lock was just part of the cost and even per-subsyste=
m
> lock reduces contention across subsystems but flushing is still expensive=
 as the
> kernel may have to traverse the cgroup update tree for each cpu on the sy=
stem.

Ah I see, thanks for the context.

>
> CCing Yosry who is looking into similar issue but for zswap stats. Maybe =
a
> general solution for such specific stats is to move them from rstat to a
> separate mechanism where update can be a bit more expensive but the read =
side
> can be cheap.
>
> > If it does still
> > regress, I'd be happy to look into making the flush based on how many
> > pages have been dirtied and scaled to the dirty limit, if that sounds
> > like the best way forward.
>
> First let's see if the general solution is doable and simple otherwise we=
 can
> explore use-case specific solutions. In extreme case revert is an options=
 as
> well but I would prefer a more sophisticated ratelimiting (that Jan sugge=
sted)
> than a revert.
>

Sounds good. I'll keep an eye out for the work Yosry's doing on this
and look into hooking it up to writeback if/when his general solution
becomes available.

Thanks,
Joanne

> > I tried running the sysbench
> > oltp_write_only benchmark mentioned in d9b3ce8769e3 on my setup but
> > couldn't get the database to dirty pages fast enough to trigger the
> > balance dirty pages logic to see how it performs without the
> > ratelimiting. If you still have the script for how you ran the test
> > back in 2024, I'm happy to give it a go on my local machine.
>
> Unfortunately I don't have those scripts or setup.