Re: 6.18.37 has problems with nfs4 (server), 6.18.36 works

Am 2026-07-03 22:59, schrieb Chuck Lever:
> On Fri, Jul 3, 2026, at 2:30 PM, Wolfgang Walter wrote:
>> Hello Chuck,
>> 
>> Am 2026-07-03 18:03, schrieb Chuck Lever:
>>> Hi Wolfgang, and stable@ --
>>> 
>>> Short version for stable@: 6.18.37 does not need a revert of
>>> 95f9eb19d5e6 ("Revert 'NFSD: Defer sub-object cleanup in export
>>> put callbacks'").  That commit is correct for 6.18, and it is
>>> not the cause of Wolfgang's crash.  Please leave it in place.
>> 
>> Ok. I run v6.18.37 with the patch reverted since about a day (just for
>> the record). But according to your analysis, that's just a 
>> coincidence.
>> 
>>> 
>>> The reasoning: 95f9eb19d5e6 touches only fs/nfsd/export.c,
>>> export.h, and nfsctl.c.  Wolfgang's oops is in
>>> remove_blocked_locks() -> __destroy_client() ->
>>> nfsd4_destroy_clientid(), entirely within fs/nfsd/nfs4state.c,
>>> which the revert does not modify.  That path is byte-for-byte
>>> identical across 6.18.36, 6.18.37, and current mainline, so the
>>> revert cannot have introduced the bug and no missing backport
>>> repairs it.  The 6.18.36-good / 6.18.37-bad split is a timing
>>> coincidence; I believe the same latent bug is present in both.
>>> 
>>> Because the defect is present upstream as well, the fix belongs
>>> in mainline first and is then backported to 6.18.y and the other
>>> affected trees.
>>> 
>>> Wolfgang - to confirm this and capture the allocation and free
>>> stacks, a KASAN-enabled kernel would settle it.  On a v6.18.37
>>> tree:
>>> 
>>>   1. Add to your .config (keep your usual CONFIG_DEBUG_INFO so
>>>      symbols resolve):
>>> 
>>>        CONFIG_KASAN=y
>>>        CONFIG_KASAN_GENERIC=y
>>>        CONFIG_KASAN_INLINE=y
>>>        CONFIG_STACKTRACE=y
>>> 
>>>   2. Build and boot that kernel.  Stay on 6.18.37 -- you do not
>>>      need the revert-the-revert build I suggested earlier; that
>>>      experiment no longer tells us anything.
>>> 
>>>   3. When it trips, KASAN prints a "BUG: KASAN: use-after-free"
>>>      report with "Allocated by" and "Freed by" call stacks.
>>>      That report, in full, is what I need -- it should land in
>>>      /var/log/messages just as the last oops did.
>>> 
>>> One caveat: KASAN roughly doubles memory use and adds CPU cost,
>>> so weigh that before running it on the production server.  If
>>> that is not practical, a full log from the first stall line
>>> onward, with all CPU backtraces, captured over netconsole or
>>> serial, is a useful second best.
>>> 
>>> I will draft a candidate upstream fix from the analysis so far
>>> and send it separately.  If KASAN on the production box is not
>>> an option, testing that patch may be the least disruptive way
>>> to confirm.
>>> 
>> 
>> I think the memory usage should not be a problem, higher cpu usage
>> neither.
>> 
>> But as it is a coincidence the probability to catch that error is
>> probably very low. We use v6.18 kernels since v6.18.1 on that 
>> fileserver
>> and this error never occured before.
>> 
>> Or do you think it happens more often, but without symptoms, and KASAN
>> would detect it?
>> 
>> So I will try running a v3.18.37 + your patch applied. This of course
>> can not prove that it fixes the problem because it almost never 
>> happens,
>> but probably this would detect if if the patch had side effects.
> 
> Correct: your reproduction of the crash does not appear to
> be strongly correlated with any particular kernel release. I
> based my analysis strictly on the additional stack trace data
> you sent earlier today.
> 
> I think it's more likely that your 50 client workload hit a
> particular race that exposed a pre-existing UAF. KASAN will
> change execution timing, certainly, but I can't predict
> whether it will make the race window bigger.
> 
> So you can only test whether my patch causes new regressions,
> not whether it prevents your crasher. :-(

6.18.37 + your patch is running here since saturday with no problems.

Regards,
-- 
Wolfgang Walter
Studierendenwerk München Oberbayern
Anstalt des öffentlichen Rechts