[PATCH slab/for-next v4 3/8] mm/slab: use call_rcu() in unknown context if irqs are enabled
"Harry Yoo (Oracle)" <[email protected]>
| Newsgroups | dev.linux.lists.linux-rt-devel,org.kernel.vger.bpf,org.kernel.vger.linux-kernel,org.kernel.vger.rcu,org.kvack.linux-mm |
|---|---|
| Message-ID | <[email protected]> |
call_rcu() disables IRQs with local_irq_save() to protect its per-cpu data structures. Therefore, if IRQs are not disabled, they cannot be corrupted by reentrance into call_rcu(). So fall back to the deferred path only when !allow_spin && irqs_disabled(). The RCU subsystem does not guarantee this contractually, and this optimization relies on RCU's implementation details. Ideally, it should be removed once call_rcu_nolock() is supported by the RCU subsystem. Link: https://lore.kernel.org/linux-mm/CAADnVQKRVD5ZSnEKbZZU7w86gHbGHUug2pvzpgZTngNS+fg4rw@mail.gmail.com Suggested-by: Alexei Starovoitov <[email protected]> Signed-off-by: Harry Yoo (Oracle) <[email protected]> --- mm/slub.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 0c350274fbff..680fe64aff72 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -6136,8 +6136,12 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj, unsigned int free_flags) if (likely(rcu_sheaf->size < s->sheaf_capacity)) { rcu_sheaf = NULL; } else { - if (unlikely(!allow_spin)) { - /* call_rcu() cannot be called in an unknown context */ + /* + * With !allow_spin, we might have interrupted call_rcu()'s + * IRQ-disabled critical section. If IRQs are not disabled, + * we know that's not the case. + */ + if (unlikely(!allow_spin && irqs_disabled())) { rcu_sheaf->size--; local_unlock(&s->cpu_sheaves->lock); goto fail; -- 2.53.0