problem with ixl(4) and vlans

Hi List,

just recently I discovered a problem with the ixl(4) driver.
Hopefully someone here
can help me. my setup is as follows:

Network ----- ixl3 interface ----- ixl3.15 vlan interface -----
bridge0 ----- vnet0.1 to jail

the problem now is that the jail can send data out (arp requests),
i do see the responses on the ixl3 interface of the host, but they
never make their way up to the ixl3.15 vlan interface (even though
they are tagged correctly). To rule out that my config or the
network is the cruel pit i did test the same setup with a cheap
usb-ethernet adapter and there everything works as expected. I'm
on FreeBSD 13.2-RELEASE-p8 and I did test both, the in kernel
driver and the driver from ports intel-ixl-kmod-1.13.4_1.

Here is a bit of information on my environment:

# uname -a

FreeBSD mimir 13.2-RELEASE-p8 FreeBSD 13.2-RELEASE-p8 GENERIC
amd64

# pciconf -lBbcevV pci0:25:0:3

ixl3@pci0:25:0:3: class=0x020000 rev=0x09 hdr=0x00
vendor=0x8086 device=0x37d3 subvendor=0x8086 subdevice=0x35d5

vendor = 'Intel Corporation'

device = 'Ethernet Connection X722 for 10GbE SFP+'

class = network

subclass = ethernet

bar [10] = type Prefetchable Memory, range 64, base
0xb0000000, size 16777216, enabled

bar [1c] = type Prefetchable Memory, range 64, base
0xb5000000, size 32768, enabled

cap 01[40] = powerspec 3 supports D0 D3 current D0

cap 05[50] = MSI supports 1 message, 64 bit, vector masks

cap 11[70] = MSI-X supports 129 messages, enabled

Table in map 0x1c[0x0], PBA in map 0x1c[0x1000]

cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO

max read 512

link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)

cap 03[e0] = VPD

ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected

ecap 0003[140] = Serial 1 9aef76ffff01bfa4

ecap 000e[150] = ARI 1

ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled,
ARI disabled

0 VFs configured out of 32 supported

First VF RID Offset 0x006d, VF RID Stride
0x0001

VF Device ID 0x37cd

Page Sizes: 4096 (enabled), 8192, 65536,
262144, 1048576, 4194304

ecap 0017[1a0] = TPH Requester 1

ecap 000d[1b0] = ACS 1 Source Validation unavailable,
Translation Blocking unavailable

P2P Req Redirect unavailable, P2P Cmpl
Redirect unavailable

P2P Upstream Forwarding unavailable, P2P
Egress Control unavailable

P2P Direct Translated unavailable, Enhanced
Capability unavailable

PCI-e errors = Correctable Error Detected

Unsupported Request Detected

Corrected = Advisory Non-Fatal Error

VPD ident = 'Example VPD'

# ifconfig

[...]

ixl3:
flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric 0 mtu 1500

options=4a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP>

ether a4:bf:01:76:ef:9d

media: Ethernet autoselect (10Gbase-SR
<full-duplex>)

status: active

nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

ixl3.15:
flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric 0 mtu 1500

options=4200001<RXCSUM,RXCSUM_IPV6,NOMAP>

ether a4:bf:01:76:ef:9d

groups: vlan

vlan: 15 vlanproto: 802.1q vlanpcp: 0 parent interface:
ixl3

media: Ethernet autoselect (10Gbase-SR
<full-duplex>)

status: active

nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>
metric 0 mtu 1500

ether 58:9c:fc:10:dd:05

inet 192.168.55.20 netmask 0xffffff00 broadcast
192.168.55.255

id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay
15

maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200

root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0

member: vnet0.1
flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

ifmaxaddr 0 port 9 priority 128 path cost 2000

member: ixl3.15
flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

ifmaxaddr 0 port 8 priority 128 path cost 55

groups: bridge

nd6 options=9<PERFORMNUD,IFDISABLED>

[...]

# cat /etc/rc.conf

[...]

ifconfig_ixl3="up"

vlans_ixl3="15"

cloned_interfaces="bridge0"

ifconfig_bridge0="addm ixl3.15 up"

[...]

# dmesg | grep ixl

ixl0: <Intel(R) Ethernet Connection X722 for 10GBASE-T -
2.3.3-k> mem 0xb3000000-0xb3ffffff,0xb5018000-0xb501ffff at
device 0.0 numa-domain 0 on pci6

ixl0: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0

ixl0: PF-ID[0]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, MDIO
shared

ixl0: Using 1024 TX descriptors and 1024 RX descriptors

ixl0: Using 12 RX queues 12 TX queues

ixl0: Using MSI-X interrupts with 13 vectors

ixl0: Ethernet address: a4:bf:01:76:ef:9a

ixl0: Allocating 16 queues for PF LAN VSI; 12 queues active

ixl0: SR-IOV ready

ixl0: netmap queues/slots: TX 12/1024, RX 12/1024

ixl1: <Intel(R) Ethernet Connection X722 for 10GBASE-T -
2.3.3-k> mem 0xb2000000-0xb2ffffff,0xb5010000-0xb5017fff at
device 0.1 numa-domain 0 on pci6

ixl1: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0

ixl1: PF-ID[1]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, MDIO
shared

ixl1: Using 1024 TX descriptors and 1024 RX descriptors

ixl1: Using 12 RX queues 12 TX queues

ixl1: Using MSI-X interrupts with 13 vectors

ixl1: Ethernet address: a4:bf:01:76:ef:9b

ixl1: Allocating 16 queues for PF LAN VSI; 12 queues active

ixl1: SR-IOV ready

ixl1: netmap queues/slots: TX 12/1024, RX 12/1024

ixl2: <Intel(R) Ethernet Connection X722 for 10GbE SFP+ -
2.3.3-k> mem 0xb1000000-0xb1ffffff,0xb5008000-0xb500ffff at
device 0.2 numa-domain 0 on pci6

ixl2: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0

ixl2: PF-ID[2]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C

ixl2: Using 1024 TX descriptors and 1024 RX descriptors

ixl2: Using 12 RX queues 12 TX queues

ixl2: Using MSI-X interrupts with 13 vectors

ixl2: Ethernet address: a4:bf:01:76:ef:9c

ixl2: Allocating 16 queues for PF LAN VSI; 12 queues active

ixl2: ixl_set_link: Error getting phy capabilities -7, aq error: 5

ixl2: SR-IOV ready

ixl2: netmap queues/slots: TX 12/1024, RX 12/1024

ixl3: <Intel(R) Ethernet Connection X722 for 10GbE SFP+ -
2.3.3-k> mem 0xb0000000-0xb0ffffff,0xb5000000-0xb5007fff at
device 0.3 numa-domain 0 on pci6

ixl3: fw 3.1.55727 api 1.5 nvm 3.31 etid 80000d32 oem 1.262.0

ixl3: PF-ID[3]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C

ixl3: Using 1024 TX descriptors and 1024 RX descriptors

ixl3: Using 12 RX queues 12 TX queues

ixl3: Using MSI-X interrupts with 13 vectors

ixl3: Ethernet address: a4:bf:01:76:ef:9d

ixl3: Allocating 16 queues for PF LAN VSI; 12 queues active

ixl3: ixl_set_link: Error getting phy capabilities -7, aq error: 5

ixl3: SR-IOV ready

ixl3: netmap queues/slots: TX 12/1024, RX 12/1024

ixl2: Link is up, 10 Gbps Full Duplex, Requested FEC: None,
Negotiated FEC: None, Autoneg: False, Flow Control: None

ixl2: link state changed to UP

ixl3: Link is up, 10 Gbps Full Duplex, Requested FEC: None,
Negotiated FEC: None, Autoneg: False, Flow Control: None

ixl3: link state changed to UP

bridge0: can't disable some capabilities on ixl3.15: 0x400

ixl3: promiscuous mode enabled

ixl3.15: promiscuous mode enabled

from my packet traces:

# tcpdump -vvv -i ixl3

tcpdump: listening on ixl3, link-type EN10MB (Ethernet), capture
size 262144 bytes

[...]

13:36:20.155843 ARP, Ethernet (len 6), IPv4 (len 4), Request
who-has 192.168.55.1 tell 192.168.55.10, length 28

13:36:20.156285 ARP, Ethernet (len 6), IPv4 (len 4), Reply
192.168.55.1 is-at b8:27:eb:47:8f:43 (oui Unknown), length 46

13:36:21.169003 ARP, Ethernet (len 6), IPv4 (len 4), Request
who-has 192.168.55.1 tell 192.168.55.10, length 28

13:36:21.169538 ARP, Ethernet (len 6), IPv4 (len 4), Reply
192.168.55.1 is-at b8:27:eb:47:8f:43 (oui Unknown), length 46

Here the answer can be see, its tagged with 802.1q tag 15

# tcpdump -vvv -i ixl3.15

tcpdump: listening on ixl3.15, link-type EN10MB (Ethernet),
capture size 262144 bytes

14:14:37.255429 ARP, Ethernet (len 6), IPv4 (len 4), Request
who-has 192.168.55.1 tell 192.168.55.10, length 28

14:14:42.263475 ARP, Ethernet (len 6), IPv4 (len 4), Request
who-has 192.168.55.1 tell 192.168.55.10, length 28

14:15:02.556311 ARP, Ethernet (len 6), IPv4 (len 4), Request
who-has 192.168.55.1 tell 192.168.55.10, length 28

14:15:07.557644 ARP, Ethernet (len 6), IPv4 (len 4), Request
who-has 192.168.55.1 tell 192.168.55.10, length 28

The answer cannot be seen on the VLAN interface ):

I hope the list can help me out here, as I am lost.

Thanks & best

Daniel