https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=297307
Bug ID: 297307
Summary: if_pfsync: pfsync_sendout() silently and permanently
fails to transmit when pfsync MTU exceeds MJUMPAGESIZE
Product: Base System
Version: CURRENT
Hardware: Any
OS: Any
Status: New
Severity: Affects Some People
Priority: ---
Component: kern
Assignee: [email protected]
Reporter: [email protected]
# if_pfsync: pfsync_sendout() silently and permanently fails to transmit when
pfsync MTU exceeds MJUMPAGESIZE
## Synopsis
If a `pfsync` interface's MTU is set above `MJUMPAGESIZE` (4096 bytes on
amd64), `pfsync_sendout()` fails every packet allocation, never transmits
again, and cannot recover without destroying and recreating the interface
(i.e., a reboot, since `pfsync0` cannot be destroyed at runtime).
## Environment
- FreeBSD 14.3-RELEASE-p16 (`stable/26.1-n272152-9b6eef552f24`, via OPNsense
26.1.11_10, amd64)
- Believed applicable to any FreeBSD version with the current
`pfsync_sendout()`/`m_get2()` implementation
- Two-node CARP/pfsync pair, `pfsync0` MTU set to 8930 to match a jumbo-framed
sync interface
## Description
`pfsync_sendout()` (`sys/netpfil/pf/if_pfsync.c`) allocates its outgoing packet
as:
```c
m = m_get2(max_linkhdr + b->b_len, M_NOWAIT, MT_DATA, M_PKTHDR);
if (m == NULL) {
if_inc_counter(sc->sc_ifp, IFCOUNTER_OERRORS, 1);
V_pfsyncstats.pfsyncs_onomem++;
return;
}
```
`m_get2()` (`sys/kern/kern_mbuf.c`) returns NULL for any request above
`MJUMPAGESIZE` (`PAGE_SIZE`, 4096 on amd64):
```c
if (size <= MCLBYTES)
return (uma_zalloc_arg(zone_pack, &args, how));
if (size > MJUMPAGESIZE)
return (NULL);
```
`pfsync_q_ins()` accumulates `b->b_len` up to the pfsync interface's configured
MTU before calling `pfsync_sendout()`. If that MTU is above ~4080 bytes
(`MJUMPAGESIZE - max_linkhdr`), every flush requests an allocation `m_get2()`
will always refuse. **No pfsync packet is ever constructed**, silently — the
failure is counted (`pfsyncs_onomem`) but nothing is logged, and `bpf_mtap()`
(which would make the failure visible to `tcpdump -i pfsync0`) is never
reached, since it only runs after a successful allocation.
**The failure is permanent, not transient.** The `m_get2() == NULL` branch
returns without resetting `b->b_len`. The only reset (`b->b_len =
PFSYNC_MINPKT`) happens after a successful send. The only other path that
clears it is `pfsync_drop()`, reachable only from `pfsync_clone_destroy()`.
Once `b_len` exceeds the threshold, every subsequent insert pushes it further
past it, and every subsequent flush fails identically — indefinitely, surviving
config reloads and even, in our case, a full VM restart with the interface
still misconfigured (recreating the interface with a *corrected* MTU does
resolve it, since that resets `b_len`; changing the MTU on the live,
already-latched instance does not).
**Symptom chain observed:** with pfsync effectively unable to transmit
anything, the pfsync bulk-update mechanism (used e.g. at boot, or when CARP
failover occurs) also fails every attempt (times out after ~65–90s), and
because `pfsync bulk start` self-demotes the requesting node's CARP priority
(`carp_demote_adj_p`) until a successful `pfsync bulk done` clears it, a bulk
transfer that can never succeed causes a full, repeating CARP mastership
handoff to the peer every time it's attempted.
## Steps to reproduce
1. Configure `pfsync0` (via `ifconfig pfsync0 mtu <N>` or by setting it to
match a syncdev interface with `N > ~4080`) with an MTU above `MJUMPAGESIZE -
max_linkhdr`.
2. Generate enough pf state churn that a pfsync send bucket's `b_len` exceeds
~4080 bytes before a flush (in practice: any nontrivial amount of real
traffic).
3. Observe `netstat -s -p pfsync`: `failures due to mbuf memory error` climbs
continuously; `state inserts sent` and related send counters stay frozen.
4. Observe `pfsync bulk start` always followed by `pfsync bulk fail`, never
`pfsync bulk done`, if CARP is in use.
5. Confirmed via DTrace (`fbt::pfsync_sendout:entry`, `fbt::pfsync_tx:entry`,
`fbt::m_get2:entry`): `pfsync_sendout` executes continuously (tens of thousands
of calls per few seconds under load), `pfsync_tx` is never reached, and
`m_get2` is repeatedly called with a size in the 8192–16384 range, all
returning NULL.
## Expected result
In descending order of desirability:
1. **Best:** a pfsync MTU above `MJUMPAGESIZE` actually works — the allocator
path supports it, and jumbo pfsync frames are transmitted as configured.
2. **Acceptable:** the interface self-clamps to the largest MTU the allocator
can actually satisfy, so an operator's oversized request degrades to a working
configuration rather than a broken one.
3. **Minimum acceptable:** an MTU the allocator cannot satisfy is rejected at
set time (`SIOCSIFMTU` / clone-create) with a visible error, rather than
accepted and left to fail silently at every subsequent send.
Independent of which of the above is implemented, `pfsync_sendout()` should not
permanently latch on an allocation failure — the queue-reset/drop behavior
described below is defense-in-depth that's worth having regardless of whether
(1), (2), or (3) is also fixed, since it protects against any other path that
could produce an oversized `b_len`.
## Actual result
Silent, total, permanent loss of pfsync transmission with no log output and no
visible indication beyond the `pfsyncs_onomem` counter, until the interface is
destroyed and recreated (reboot, in practice).
## Suggested fixes
In order matching the cascade above:
1. **Support the configured MTU.** Use an allocation path capable of satisfying
requests above `MJUMPAGESIZE` (e.g. `m_getjcl`/`m_getm2`) so a large,
deliberately-configured pfsync MTU actually works rather than being silently
rejected. This is the best outcome and removes the bug's precondition entirely.
2. **Self-clamp if (1) isn't feasible.** If pfsync's send path is intentionally
bounded by `MJUMPAGESIZE`, have the MTU-setting path clamp any requested value
down to `MJUMPAGESIZE - max_linkhdr` rather than accepting a larger value it
cannot serve.
3. **Reject loudly, at minimum.** If clamping is undesirable (e.g. because a
silently-lowered MTU could itself be confusing), reject an MTU above the
allocator's ceiling at `SIOCSIFMTU`/clone-create time with a clear error, so
the misconfiguration fails at the moment it's introduced instead of the first
time traffic volume triggers a flush.
4. **Defense-in-depth, regardless of (1)–(3):** reset or drop the accumulated
queue state on the `m_get2() == NULL` path in `pfsync_sendout()`, so an
allocation failure — from this cause or any other — degrades gracefully (drops
this batch, recovers next cycle) instead of permanently wedging the send path
until the interface is recreated.
## Additional notes
- This was reached downstream of an OPNsense-specific issue where `pfsync0`'s
MTU is unconditionally copied from the sync interface with no upper bound
(`interfaces_pfsync_configure()`, OPNsense `src/etc/inc/interfaces.inc`,
referencing an earlier fix at commit `15acbad935` for the reverse problem).
That OPNsense-side issue is what makes this reachable via normal configuration
(any jumbo-framed interface assigned to pfsync duty) rather than requiring a
deliberately unusual manual MTU. It's reported separately to OPNsense; this
report is about the underlying FreeBSD kernel behavior, which affects any
FreeBSD/pfsync consumer that can end up with an oversized pfsync MTU by any
means.
- `netstat -m` will show no mbuf/cluster exhaustion during this failure —
`m_get2()`'s size check rejects the request before it reaches UMA, so the usual
mbuf-pressure diagnostics look completely clean while the failure is ongoing.
This is worth calling out since it's an easy place for a diagnosis to go astray
(as happened in our own investigation before the `pfsyncs_onomem` counter and
the source were correlated).
## How this was found (AI transparency)
This root cause was identified through a collaborative debugging session
between the reporter and two Claude models (Anthropic): Claude Sonnet for
interactive troubleshooting, and Claude Opus for the source-level analysis once
the fault was narrowed to a single function. All commands, DTrace invocations,
and configuration changes were run by the reporter on live hardware; the AI
proposed hypotheses and interpreted output but never had direct system access.
The cause was located by: (1) using DTrace (`fbt` provider) to trace
`pfsync_sendout`, `pfsync_tx`, and related functions by call count, showing
`pfsync_sendout` executing continuously while `pfsync_tx` was never reached
despite a nonempty, actively-filling send queue; (2) fetching and reading the
actual `if_pfsync.c`/`kern_mbuf.c` source for the exact FreeBSD version in use,
which located the `m_get2()` call and the `MJUMPAGESIZE` ceiling; and (3)
confirming `pfsyncs_onomem` incremented at a rate matching the DTrace call
volume. The fix was then confirmed empirically: lowering the sync interface's
MTU and rebooting produced the three outcomes predicted from the source — send
counters resuming, the failure counter reaching zero, and bulk transfers
completing — observed together on the reporter's production hardware.
The source excerpts, `MJUMPAGESIZE` value, DTrace results, and before/after
counters in this report reflect real system output and real source code, not
paraphrase or invention.
--
You are receiving this mail because:
You are the assignee for the bug.
lmpx.com only provides a reader for public news (NNTP) servers. It is not
affiliated with the servers or forums shown here and is not responsible for
the content of articles, which is written by their respective authors.