Re: [PATCH] fuse: disable default bdi strictlimiting

> > > First let's see if the general solution is doable and simple otherwise we can
> > > explore use-case specific solutions. In extreme case revert is an options as
> > > well but I would prefer a more sophisticated ratelimiting (that Jan suggested)
> > > than a revert.
> > >
> >
> > Sounds good. I'll keep an eye out for the work Yosry's doing on this
> > and look into hooking it up to writeback if/when his general solution
> > becomes available.
>
> I am not sure if I had in mind would generalize well. For zswap, I was
> just going to replace rstat with per-memcg atomic counters, and simply
> walk the cgroup parents in the update path, as the zswap load/store path
> shouldn't be too hot.
>
> However, looking at other in-kernel flushers (including
> mem_cgroup_wb_stats()), I see they consume other stats like
> NR_INACTIVE_FILE/NR_ACTIVE_FILE, which are updated in the page
> allocation path. I am not sure if a cgroup parent walk with atomic
> updates would fly there.
>
> A more problematic one is count_shadow_nodes(), which consumes slab
> object stats. I think we definitely cannot do atomic updates in the slab
> allocation path. Although it seems like count_shadow_nodes() is a rough
> estimate and perhaps we can forgoe using the stats there.
>
> If we want a generic solution for in-kernel flushers to improve stats
> accuracy without killing performance, I think we need a heavier lift to
> rework rstat or move away from it completely.

(Adding a few more people and mailing lists)

So I experimented with a per-memcg atomics in zswap (see [1] for
context), and it appears to be too expensive even for updating the
zswap stats during reclaim / swap faults. So I think zswap will need a
similar approach to writeback and other in-kernel users.

One idea is to keep per-CPU per-cgroup counters, but on the update
side we update all the parents' counters, instead of just updating the
current cgroup's counters. Flushing the stats then only needs to
iterate the per-CPU counters and doesn't need to walk the cgroup tree
at all. There is added work on the update side, but hopefully it's
cheap enough to update per-CPU counters, and cgroup hierarchies aren't
usually too deep.

A more radical approach is to completely drop the per-CPU rstat trees.
So instead of walking the parents to add them to the per-CPU tree, we
just walk the parents and directly update the stat. The flush side
will then always iterate the per-CPU counters to accumulate them. With
this approach, the update cost roughly remains the same, but the flush
cost changes. Instead of walking the trees and flushing all cgroups,
but only doing it on CPUs/cgroups that actually have updates, we
always iterate and sum the per-CPU counters. The average flush cost
may become more expensive, but it should be more consistent and won't
scale with the cgroup tree size. I am not sure if this would be a net
win or not.

We can also choose to do the above for all the stats or only the ones
used by in-kernel flushers, the latter would probably perform better
but at the cost of more complexity.

I honestly don't have any time to pursue any of these approaches, so
this is mostly just a brain dump. But I do think we should try to more
systematically solve the stats flushing problem as it has given us a
lot of pain over the years.

[1]https://lore.kernel.org/linux-mm/CAO9r8zP6JrZcBnzb8A3A9V8so4gJMWLvrJQ7VwPyyU417GikQg@mail.gmail.com/