[Bug 296835] kern: coredump of a process with many EVFILT_VNODE knotes hangs for hours, process unkillable (SIGKILL pending), holds fd table lock (NT_PROCSTAT_KQUEUES note generation)

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296835

--- Comment #2 from [email protected] ---
Follow-up, same machine (dal, 15.1-RELEASE
releng/15.1-n283562-96841ea08dcf, amd64, root and /home on ZFS).

The original report speculated that the NT_PROCSTAT_FILES note "has the same
shape" as NT_PROCSTAT_KQUEUES and had probably already consumed part of the
elapsed time. I can now confirm that directly, and with a worse consequence
than expected: **the per-fd path-resolution path is reachable without any
coredump at all**, through the plain `kern.proc.filedesc` sysctl, and it is
equally uninterruptible.

### What happened

Same trigger as before: a `codex` process (codex-0.144.1, Rust, notify-rs
kqueue backend) accumulated ~650,000 open fds, each with an EVFILT_VNODE
knote. `kern.openfiles` was 668,507 and the KNOTE UMA zone showed 657,555
items in use. The process itself was idle (state `I+`, 0.0% CPU) and never
crashed — no signal, no core dump.

The victim was Konsole (konsole-26.04.3), which polls the foreground
process's current directory to render the `%d` field of its default tab
title. It does so via `KERN_PROC_FILEDESC`. One such call never completed:

```
  PID %CPU %MEM     TIME STAT COMMAND
37819 99.1  5.7 99:19.69 R    /usr/local/bin/konsole
```

99 minutes of CPU, of which 17 seconds user and 1h30m system. The GUI thread
was wedged in the kernel the entire time, so the terminal window was frozen
and unresponsive. procstat -kk, sampled repeatedly, was always in the same
place:

```
37819 123940 konsole  -
  fzap_cursor_retrieve+0x8d zap_cursor_retrieve+0x1e7 zap_value_search+0x8f
  zfs_znode_parent_and_name+0xbf zfs_vptocnp+0x1cd vn_vptocnp+0x17f
  vn_fullpath_dir+0x102 vn_fullpath_any+0x59 vn_fullpath+0xd2
  vn_fill_kinfo_vnode+0x41 vn_fill_kinfo+0x45 export_file_to_kinfo+0x144
  kern_proc_filedesc_out+0x387 sysctl_kern_proc_filedesc+0x8d
  sysctl_root_handler_locked+0x91 sysctl_root+0x268 userland_sysctl+0x1a6
  sys___sysctl+0x65
```

The other two konsole threads (QDBusConnectionManager, QXcbEventQueue) were
idle in `poll`. Nothing was looping in userspace.

`procstat -f <codex-pid>` reproduced it exactly, and **`timeout 15 procstat
-f` failed to kill it** — the command ran for ~105 seconds until the sysctl
returned on its own. Every other process on the box returned instantly.

Killing codex resolved it: `kern.openfiles` dropped 668,507 -> 2,720 and
konsole returned to `S` at 0.5% CPU.

### Analysis

In `sys/kern/kern_descrip.c` (checked against main, __FreeBSD_version
1600019), `kern_proc_filedesc_out()` walks the whole table:

```c
        FILEDESC_FOREACH_FP(fdp, i, fp) {
                ...
                error = export_file_to_sb(fp, i, &rights, efbuf);
                if (error != 0 || refcount_load(&fdp->fd_refcnt) == 0)
                        break;
        }
```

Two properties make this pathological at 650k fds:

1. **No cancellation point.** The loop terminates only on sbuf error or on
   the fd table going away. There is no check for pending signals on
   `curthread`, so the calling process cannot be interrupted or killed for
   as long as the walk runs — minutes here, and it scales linearly with the
   victim's fd count. This is the same "SIGKILL pending but never delivered"
   property as the coredump case, reached by a much cheaper route.

2. **`vn_fullpath()` runs under `FILEDESC_SLOCK`.** `export_file_to_sb()`
   drops the lock only around `export_kinfo_to_sb()`; the preceding
   `export_file_to_kinfo()` -> `vn_fill_kinfo()` -> `vn_fullpath()` is
   called with the shared filedesc lock held. On ZFS a namecache miss falls
   back to `zfs_vptocnp()` -> `zap_value_search()`, a linear ZAP scan of the
   parent directory, so per-fd cost is unbounded and the lock hold time
   inherits it.

Worth noting that no directory on this system is unusually large — the
biggest is ~9,600 entries. The cost is entirely the fd count multiplied by
an ordinary ZAP scan.

### Why this widens the impact of the original report

The coredump variant requires the fd-heavy process to crash. This variant
requires nothing of the victim at all — it just has to exist. Any process
that reads `kern.proc.filedesc` on it inherits a multi-minute uninterruptible
syscall, and at least one very common desktop application does this
automatically, on its GUI thread, on a timer.

This is not exotic. Any kqueue-based file watcher (notify-rs, watchman) is
one fd per watched vnode by construction, and LLM coding agents watching
large source trees are making fd counts in the 10^5-10^6 range routine.

### Suggested directions

- A cancellation point in the `FILEDESC_FOREACH_FP` loop (and in the
  equivalent knote walk) so a pending signal aborts the walk with EINTR.
  This alone would turn both bugs from "unkillable for hours" into
  "interruptible and slow", which is a large practical difference.
- Avoid holding `FILEDESC_SLOCK` across `vn_fullpath()`, or resolve paths in
  a second pass outside the lock.
- Optionally a flag or sysctl to skip path resolution for consumers that do
  not need it, mirroring `kern.coredump_pack_fileinfo`.

-- 
You are receiving this mail because:
You are the assignee for the bug.