Re: [RFC PATCH] fsnotify: raise default fs.inotify.max_user_instances now that memcg accounts for it

On Tue, Aug 18, 2026 at 3:47 PM Jan Kara <[email protected]> wrote:
>
> Hello!
>
> On Wed 12-08-26 20:51:36, Harsh Upadhayay wrote:
> > I hit this running a 3-node k3s homelab cluster: every container on a
> > node shares one 128-slot fs.inotify.max_user_instances pool (no user
> > namespace remapping), kubelet alone was holding 25-40+ instances just
> > from ConfigMap/Secret volume watching, and the pool emptied under
> > normal load -- surfaced as an unhelpful, inotify-silent "too many open
> > files" in whichever container's turn it was to ask next. Filing as
> > RFC since I couldn't find prior discussion of raising this default.
>
> Not sure what Amir thinks about this but I think default is just a default
> and it is natural it is not good for everybody. I can see how 128 inotify
> instances may not be enough for Kubernetes setups but then the answer is
> for Kubernetes installation to configure maximum inotify instances to a
> higher value. And as you show below, different container runtimes tune the
> value differently which just proves my point that whatever value we choose
> people will need to tune it - which is the point of this limit being tunable
> after all.

I agree that I don't see a strong enough reason to change an upstream default
to another arbitrarily bigger value that could be easily changed by the distro
or by the kubernetes host.

>
> So I don't see a strong enough reason to bump up the default limit because
> there can be other users for which increased limit could allow a potential
> DoS in their setups. The concern isn't as much the memory consumed by
> inotify instances themselves as you write below but more all the events
> that can get queued to each instance. OTOH those are accounted to the
> instance's memcg so your overall argument that in memcg enabled setups the
> limits aren't that much needed is still somewhat valid.
>

Not only the memory of marks nor events but also processing time and
arbitrary long delays to other users in the system.
I'll explain this with an extreme example below...

> > fs.inotify.max_user_instances has been a flat 128 since inotify's
> > introduction in 2005. That number is not per-container or per-workload
> > -- it is a single pool shared by every process running under a given
> > real UID on the host. On a multi-tenant Linux box, that means every
> > container's root process shares one 128-slot pool with every other
> > unremapped container's root, plus the host's own kubelet/containerd
> > (or equivalent) bookkeeping, since user-namespace UID remapping is
> > off by default in essentially every mainstream container runtime.
> >

...let's do a thought experiment, why stop at one multi-tenant level
why not nest it several times as userns is designed to do to allow for
containers inside containers.

In this hypothetical environment I am not sure it makes sense for
users in container A to pay for the cost of subscribers in nested
containers B.*.*.

It may make more sense in nested environment for container B to have a
single group in container B getting the events for all the subscribers in B.*
and forwarding the events to sub-subscribers and so forth in B.*.*.

Maybe the group in container B does not have inode marks but fs marks
with bpf inode maps as filter.

IOW, if kubernetes needs to increase max_user_instances for modern day
workloads - it may do that, but I don't think this design extends well into
a future of many more containers on the same host.
Even if userns were used by kubernetes, I am not sure that the current
publish subscribe model of inode marks would be able to handle thousands
of userns well.

Thanks,
Amir.