Re: Asymmetric forwarded TCP TX collapse over one GRE path on FreeBSD 14.3-RELEASE-p8 (PF independent, offloads disabled)

On Tue, Feb 24, 2026 at 3:02 PM Ivan K. <[email protected]> wrote:

> Each real life - 1500, gre 1472.
> I checked this. I tried setting mtu 1200 on the client device. I have mss
> clamping at 1400 in pf.
> If it's mtu, it doesn't explain why the identical tunnel on the same
> machine functions normally.
>
>
> 24 февр. 2026 г., в 21:10, Mark Saad <[email protected]> написал(а):
>
>
>
> On Tue, Feb 24, 2026 at 12:19 PM Ivan K. <[email protected]> wrote:
>
>> Hello, everyone. The text below was written by me and “polished” by a
>> neural network, which I have been using to research this problem for three
>> days now.
>>
>> I am investigating a reproducible GRE issue and would appreciate advice
>> on further debugging.
>>
>> Versions / hosts
>> - Router: FreeBSD 14.3-RELEASE-p8
>> - VPS0: FreeBSD 14.3-RELEASE-p8
>> - VPS2: FreeBSD 14.3-RELEASE-p8
>>
>> Topology
>> - Router has WAN on VLAN interface `onp` (parent `igb0`) and LAN on
>> `igb0`.
>> - Two GRE tunnels from the same router:
>>   - GRE to VPS0: healthy
>>   - GRE to VPS2: problematic
>> - There is also an ng tunnel to VPS2 (works better than bad GRE, but not
>> full speed).
>> - Clients behind router tested via both Wi-Fi and wired Ethernet (same
>> problem).
>> - Also tested traffic from a VM on the router: VM path does not reproduce
>> the severe collapse.
>>
>> Core symptom (asymmetric)
>> - Only TX direction is affected (client uploads through GRE->VPS2).
>> - RX/download direction is not affected.
>> - Through GRE->VPS2, TCP upload collapses to about ~100 Kbit/s per flow.
>> - 5 parallel TCP flows produce roughly 5 x ~100 Kbit/s.
>> - iperf3 TCP shows many retransmissions and very small cwnd (~2.7 KB).
>> - Same client via GRE->VPS0 is fast (normal high throughput).
>> - Traffic generated locally on router through GRE->VPS2 is also fast.
>>
>> Firewall / policy checks
>> - PF disabled completely (`pfctl -d`) for tests: problem remains.
>> - Therefore this is not PF rule/policy dependent in current testing.
>> - Also checked for ipfw/dummynet:
>>   - `kldstat | egrep 'ipfw|dummynet|pf'` shows only `pf.ko`.
>>   - `ipfw show` returns “Protocol not available”.
>>   - `net.inet.ip.fw.enable` OID not present.
>>
>> Offload / NIC checks performed
>> - Explicitly disabled offload features (multiple attempts), including:
>>   - `-lro`
>>   - `-tso` / tso-related toggles
>>   - `-rxcsum` / `-txcsum` / v6 checksum toggles
>>   - VLAN hardware offload related toggles where supported
>> - No improvement.
>> - Current interface snapshot during testing:
>>   - `ifconfig igb0` shows options: `VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS`
>> (no LRO/TSO flags active)
>>   - `ifconfig onp` shows `options=0`
>> - Reboots of router and VPS2 done after changes: no improvement.
>> - Also tested with minimal manual GRE setup (without tunnel-management
>> scripts, only one GRE up): no improvement.
>>
>> Queue/counters/sysctl checks
>> - `netstat -Q` shows no queue drops (`QDrops=0`).
>> - `netstat -iW -I ...` during tests shows no obvious interface error
>> growth.
>> - `sysctl net.inet.ip.fastforwarding` does not exist on this release.
>>
>> UDP vs TCP behavior
>> - UDP over problematic GRE path can run at high throughput (similar order
>> from client and from router), i.e., no analogous dramatic collapse.
>> - TCP over same path collapses hard in TX direction from forwarded
>> clients.
>>
>> Packet capture findings
>> - Collected captures at multiple points (router and VPS side).
>> - In bad runs, client-side TCP packets are seen, but many segments are
>> retransmitted repeatedly.
>> - Comparing capture points suggests segments often appear on VPS GRE side
>> only much later / after retransmission, consistent with loss on the GRE
>> path for this forwarded TCP case.
>> - In control runs (local router-generated traffic), packet timing/count
>> alignment is normal.
>>
>> What makes this puzzling
>> - Same router, same OS version, same general config:
>>   - GRE to VPS0 works well.
>>   - GRE to VPS2 fails for forwarded TCP TX only.
>> - PF disabled does not change behavior.
>> - Offloads disabled does not change behavior.
>> - No obvious drops in standard queue/interface counters.
>>
>> Request
>> Any suggestions on what to inspect next for FreeBSD 14.3-p8 in this
>> scenario:
>> - netisr / forwarding path diagnostics
>> - GRE-specific kernel instrumentation
>> - known igb/VLAN/GRE forwarding edge cases
>> - any tunables or debug knobs that could explain “forwarded TCP TX
>> collapse only on one GRE peer/path”
>>
>> Thanks.
>>
>>
> Ivan
>  Just a guess but it sounds like you may have a MTU issue . What is the
> MTU settings on each of the real interfaces and the gre tunnels ?
> --
> mark saad | [email protected]
>
>
> Ivan
  What sort of network cards are you using? This is another long shot but
do you have anything that does GRE offload? Some of the intel
and mellanox cards support some sort of checksum offloads.

-- 
mark saad | [email protected]