Re: [PATCH] fuse: disable default bdi strictlimiting

On Tue, Jul 14, 2026 at 10:10 AM Jan Kara <[email protected]> wrote:
>
> On Mon 13-07-26 18:13:29, Joanne Koong wrote:
> > On Sat, May 30, 2026 at 4:04 AM Jan Kara <[email protected]> wrote:
> > >
> > > On Thu 28-05-26 15:11:18, Joanne Koong wrote:
> > > > On Thu, May 28, 2026 at 5:34 AM Jan Kara <[email protected]> wrote:
> > > > > > I think this is also going to be a problem for cgroups with large
> > > > > > folios since they also, as I understand it, are constrained with a
> > > > > > limited / tight dirty budget. I ran some initial benchmarks with
> > > > > > cgroup memory constraints on NVMe and saw similar instability (a
> > > > > > single writer in a 8 GB cgroup had max write latencies of 6 seconds vs
> > > > > > 15 ms without the cgroup, with the balance_dirty_pages() throttling
> > > > > > oscillating rather than settling near the set point).
> > > > >
> > > > > Ok, so this is more folios (819) than my 512 gut feeling estimate :) What
> > > > > was the write throughput of the NVMe drive? The high drive throughput also
> > > > > requires more dirty data to keep the drive saturated so that writeback
> > > > > throughput doesn't oscilate too much.
> > > >
> > > > The write throughput of the NVMe drive I was using was around ~1.1
> > > > Gb/s (measured by running direct I/O). I think with the 1.6GB dirty
> > > > budget, the math for that ends up being that the device drains it in
> > > > ~1.5 secs. The performance I was seeing with the initial cgroup
> > > > benchmarks was with 4k pages (no large folios enabled) on btrfs.
> > >
> > > OK, you might want to experiment with some other filesystem (I suggest xfs
> > > or ext4) as well. Btrfs writeback behavior is a bit special with its data
> > > checksum computations etc. and thus latency of starting writeback. It could
> > > contribute to the erratic behavior with the relatively low dirty limits.
> >
> > I reran the benchmarks on xfs and ext4 and saw similiar results. The
> > max write latency under the memcg were
> > xfs        5.1s
> > ext4      5.2 s
> > btrfs      6.2 s
> >
> > compared to the non-memcg case (0.4 to 2.0 ms). This didn't affect
> > throughput though, just the tail latency. When I ran xfs with large
> > folios disabled, I still saw ~4.1s.
>
> OK, interesting. Thanks for running these tests! Also it is good to know
> this is not really related to large folios, that somewhat simplifies
> matters.
>
> > From the balance_dirty_pages() ftrace tracepoints, the dirty_ratelimit
> > value is consistently stable / accurate but it looks like what's
> > happening is that during freerun, the writer essentially dirties at
> > memcpy speed until the freerun ceiling, and then the soft/proportional
> > throttling kicks in but doesn't kick in fast or hard enough, which
> > allows the number of dirty pages to exceed the hard limit by ~2x, and
> > then at that point the writer is forced into the loop where it sleeps
> > max_pause (200ms) each iteration until writeback has drained the
> > number of dirty pages back under the limit.
>
> OK, above you mentioned that the dirty limit for the memcgs is set at
> 1.6GB. Does that mean that dirty throttling allows memcg to dirty up to
> ~3.2GB of pages? I wouldn't have expected that...

On a couple of the runs, I saw it get as high as up to 5GB.

I think this is because the balance_dirty_pages() code uses the memcg
stats (NR_FILE_DIRTY) but these stats are only periodically
flushed/refreshed when reading it, so the balance dirty code is
seeing/using lagging/non-uptodate values.

I'm seeing this in
  balance_dirty_pages()
      balance_domain_limits(mdtc,...)
          domain_dirty_avail()
                mem_cgroup_wb_stats()
                       mem_cgroup_flush_stats_ratelimited()

where mem_cgroup_flush_stats_ratelimted() has this logic:

void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg)
{
        /* Only flush if the periodic flusher is one full cycle late */
        if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME))
                mem_cgroup_flush_stats(memcg);
}

where FLUSH_TIME is defined as 2UL * HZ, which afaict means the
flushing can be around 2 seconds stale (accounting for the periodic
flusher in flush_memcg_stats_dwork() that flushes every FLUSH_TIME
interval).

From what I see, the freerun and hard limit checks use this stale
under-reported value (in domain_dirty_freerun() and
wb_position_ratio()), which means the writer gets to keep dirtying and
blow past the limit for up to 2 extra seconds before the throttling
limits it.

Maybe one idea is to in the memcg throttle path do the flush based on
how many pages are getting dirtied instead of on how much time has
elapsed? It looks like flushing grabs the rstat lock though, so maybe
only doing it in the throttled case where it's past freerun would be
best. It looks like the logic in the regular non-ratelimited
mem_cgroup_flush_stats() call does something similar. I can run some
experiments this week with this if you think it could be promising.

>
> Also you mentioned that the disk you are using has writeback throughput of
> ~1.1GB/s. Even with 3.2GB of pages we should get below the dirty limit in
> 2s and not 5. So there's something going on I don't understand yet :)

I'm not sure how the 5 got there either :) I reran it a couple of
times today and saw some variance, with the average of the runs
landing closer to around ~4s for xfs. From the traces, the writeback
throughput looks fine, so I don't think it's that the device is
writing back too slowly. I saw a trace where dirty went from ~4.7GB to
1.3 GB over 2.42 seconds which is about ~1.4 GB/s, which matches the
device's direct-io speed I saw. I think it's more that the number of
dirty pages is exceeding the limit by more than 2x and then it writes
back even below the hard limit (eg it keeps writing back for the
entire 200 ms hard throttle pause)

>
> BTW how much memory does the machine have in total?

The machine has 250 GB in total.

>
> > If we wanted to fix this, it seems like we could for
> > memory-constrained memcgs either start soft/proportional throttling
> > earlier or make it throttle more aggressively as dirty approaches the
> > limit, or start writeback sooner so it finishes draining the dirty
> > pages faster. When I experimented with lowering
> > dirty_background_ratio, the worst-case latency dropped around 2-3x
> > (~6s to 2s) but there was still multi-second tail latencies. Maybe
> > this indicates more that the throttling needs to happen earlier/harder
> > than that writeback needs to happen earlier.
>
> Hum, can you share the script you are using for the benchmark? I can try
> running in on my machine and see what happens. If I'm able to reproduce
> these long waits, I want to do a bit of tracing to understand what's
> happening with various writeback statistics. Because these large tail
> latencies aren't really expected. The original design goal was to keep
> maximum write latency (for fast enough disks) around 0.2s.

This is how I am running the benchmarks:

DEV=/dev/nvme5n1
# get device bandwidth (for reference)
fio --name=raw --filename=$DEV --rw=write --bs=1M --direct=1 --size=4G
--runtime=15 --time_based
mkfs.xfs -f $DEV && mount $DEV /mnt

mkdir -p /sys/fs/cgroup/balancedirtytest
echo 8G > /sys/fs/cgroup/balancedirtytest/memory.max

cd /sys/kernel/tracing
echo 131072 > buffer_size_kb
echo 1 > events/writeback/balance_dirty_pages/enable
echo 1 > events/writeback/global_dirty_state/enable
echo 1 > events/memcg/memcg_flush_stats/enable
echo > trace
echo 1 > tracing_on

echo 3 > /proc/sys/vm/drop_caches
( echo $BASHPID > /sys/fs/cgroup/balancedirtytest/cgroup.procs
fio --name=w --directory=/mnt --rw=write --bs=1M --ioengine=psync
--numjobs=1 --size=16G --time_based --runtime=60 --end_fsync=1 )

echo 0 > tracing_on
cp trace /tmp/xfs-data.trace

Thanks,
Joanne