[Bug 296598] inbound ipsec packets get tracked via outer ip header when the tunnel runs over if_wg

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296598

            Bug ID: 296598
           Summary: inbound ipsec packets get tracked via outer ip header
                    when the tunnel runs over if_wg
           Product: Base System
           Version: Unspecified
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: kern
          Assignee: [email protected]
          Reporter: [email protected]

- Summary
With a policy-based IPSec tunnel over an in-kernel wireguard tunnel (if_wg.ko),
it seems like the outer ip-ip header is not stripped at the correct time for
state tracking to work.
This results in TCP sessions (UDP and ICMP are unaffected) dropping after an
initial window of approximately 64k.

- Reproducing
Create a wireguard tunnel using the kernel module.
Create an IPSec tunnel in policy mode, with the outer layer routed over the
wireguard tunnel.
Test TCP connectivity over the IPSec tunnel with some traffic (I've been using
iperf, but any TCP-based protocol shows similar results)
Observe that the session drops after a small amount of traffic.

- Troubleshooting steps taken
A few different hints I've found that made me conclude this is likely a kernel
issue:
1. Creating firewall rules for the enc0 interface with state tracking disabled
fixes the issue.
2. Using the (old) go implementation of wireguard fixes the issue.
3. The following dtrace program:

    dtrace -x nolibs -qn '
    struct pf_addr { uint8_t addr8[16]; };
    struct pf_state_key_cmp { 
        struct pf_addr addr[2]; 
        uint16_t port[2];
        uint8_t af; 
        uint8_t proto; 
        uint8_t pad[2]; 
    };
    pf:ip:state:lookup
    /arg2 == 1 && stringof((char *)arg0) == "enc0"/
    {
      this->k = (struct pf_state_key_cmp *)arg1;
      printf("enc0 IN proto=%d  %d.%d.%d.%d <-> %d.%d.%d.%d  ports=%d,%d 
%s\n",
        this->k->proto,
        this->k->addr[0].addr8[0], this->k->addr[0].addr8[1],
        this->k->addr[0].addr8[2], this->k->addr[0].addr8[3],
        this->k->addr[1].addr8[0], this->k->addr[1].addr8[1],
        this->k->addr[1].addr8[2], this->k->addr[1].addr8[3],
        ((this->k->port[0]&0xff)<<8)|((this->k->port[0]>>8)&0xff),
        ((this->k->port[1]&0xff)<<8)|((this->k->port[1]>>8)&0xff),
        arg4 ? "HIT" : "MISS");
    }'

produces the following log line when run with if_wg:

    enc0 IN proto=4  <wg peer IP> <-> <local wg IP>  ports=0,0  HIT

and the following when run with the old wireguard-go:

    enc0 IN proto=6 192.168.1.10 <-> 192.168.20.60 ports=5201,49970 HIT

4. While an iperf test is running, pf logs the following lines:

    <13><...> pf: loose state match: TCP out wire: 192.168.1.10:5201
192.168.20.60:42200 stack: - [lo=4067707952 high=4067644560 win=502
modulator=0] [lo=0 high=502 win=1 modulator=0] 2:0 A seq=4067707952
(4067707952) ack=0 len=1348 ackskew=0 pkts=50:0 dir=out,fwd
    <13><...> pf: loose state match: TCP out wire: 192.168.1.10:5201
192.168.20.60:42200 stack: - [lo=4067709300 high=4067644560 win=502
modulator=0] [lo=0 high=502 win=1 modulator=0] 2:0 A seq=4067709300
(4067709300) ack=0 len=1348 ackskew=0 pkts=51:0 dir=out,fwd
    <13><...> pf: BAD state: TCP out wire: 192.168.1.10:5201
192.168.20.60:42200 stack: - [lo=4067710648 high=4067644560 win=502
modulator=0] [lo=0 high=502 win=1 modulator=0] 2:0 A seq=4067710648
(4067710648) ack=0 len=1348 ackskew=0 pkts=52:0 dir=out,fwd
    <13><...> pf: State failure on: 1       | 5

5. No change in hardware, system version, firewall settings (except those that
disable stateful filtering on enc0), tunnel settings, etc. has made any
detectable difference to the issue.

- Environment
This issue has surfaced on an OPNSense installation, an issue with OPNSense
unfortunately could not be solved
(https://github.com/opnsense/core/issues/8680).
I have reproduced this bug on OPNSense, Version 23.7.12 (FreeBSD
OPNsense.localdomain 13.2-RELEASE-p7 FreeBSD 13.2-RELEASE-p7
stable/23.7-n254871-d5ec322cffc SMP amd64) since that is the last version that
also supported the wireguard-go implementation, which lets me swap between
implementations quickly.
I have also reproduced the issue on OPSense Version 26.1 (FreeBSD
OPNSense.localdomain 14.3-RELEASE-p15 FreeBSD 14.3-RELEASE-p15
stable/26.1-n272132-0101c21cac77 SMP amd64) with no change in behaviour at all.
I can try to reproduce the issue on vanilla FreeBSD 15.1 for example, though I
don't expect OPNSense or the FreeBSD version to be the issue here, given the
symptoms.

If you'd like me to try and produce any further debug logs or try a modified
setup, let me know. I know the setup for reproduction is a bit special, and I
have a working setup I can modify easily.

-- 
You are receiving this mail because:
You are the assignee for the bug.