https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=291696
Bug ID: 291696
Summary: dwc unable to forward epair bridge packets with
checksum offloading
Product: Base System
Version: 15.0-RELEASE
Hardware: arm64
OS: Any
Status: New
Severity: Affects Only Me
Priority: ---
Component: arm
Assignee: [email protected]
Reporter: [email protected]
This issue appeared for me when updating my RockPro64 from FreeBSD 14.3-RELEASE
to 15.0-RELEASE. The initial symptoms appeared to be a lack of access to the
external network from VNET jails, which on further inspection was limited to
outgoing TCP traffic from the jails.
Running tcpdump revealed that the TCP checksums of packets originating from
VNET jails was incorrect, here is an example of trying to connect to the jail
with SSH:
```
root@central-rock:~ # tcpdump -vvv -i epair111a
tcpdump: listening on epair111a, link-type EN10MB (Ethernet), snapshot length
262144 bytes
17:36:18.943373 IP (tos 0x10, ttl 64, id 3737, offset 0, flags [DF], proto TCP
(6), length 60)
192.168.100.101.34048 > irc.central-rock.home.ssh: Flags [S], cksum 0x5f54
(correct), seq 644269107, win 64240, options [mss 1460,sackOK,TS val 2246290942
ecr 0,nop,wscale 10], length 0
17:36:18.943555 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6),
length 60)
irc.central-rock.home.ssh > 192.168.100.101.34048: Flags [S.], cksum 0x4a54
(incorrect -> 0x735d), seq 4098743306, ack 644269108, win 65535, options [mss
1460,nop,wscale 8,sackOK,TS val 3678485310 ecr 2246290942], length 0
```
ifconfig on the host:
```
root@central-rock:~ # ifconfig -a
dwc0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP>
metric 0 mtu 1500
options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE>
ether 82:16:d0:7b:f5:0b
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric
0 mtu 1500
options=10<VLAN_HWTAGGING>
ether 58:9c:fc:10:77:67
inet 192.168.100.100 netmask 0xffffff00 broadcast 192.168.100.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
bridge flags=0<>
member: epair111a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
port 9 priority 128 path cost 2000 vlan protocol 802.1q
member: epair110a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
port 5 priority 128 path cost 2000 vlan protocol 802.1q
member: dwc0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
port 1 priority 128 path cost 55 vlan protocol 802.1q
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
pflog0: flags=1000141<UP,RUNNING,PROMISC,LOWER_UP> metric 0 mtu 33152
options=0
groups: pflog
epair110a:
flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0
mtu 1500
description: jail:www
options=20000b<RXCSUM,TXCSUM,VLAN_MTU,RXCSUM_IPV6>
ether 58:9c:fc:10:24:0b
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair111a:
flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0
mtu 1500
description: jail:irc
options=20000b<RXCSUM,TXCSUM,VLAN_MTU,RXCSUM_IPV6>
ether 58:9c:fc:10:bf:1d
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```
ifconfig in the jail:
```
root@central-rock:~ # jexec -u root irc ifconfig -a
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0 metric 0 mtu 33152
options=0
groups: pflog
epair111b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP>
metric 0 mtu 1500
options=20000b<RXCSUM,TXCSUM,VLAN_MTU,RXCSUM_IPV6>
ether 58:9c:fc:10:98:82
inet 192.168.100.111 netmask 0xffffff00 broadcast 192.168.100.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```
The bad checksum likely isn't an issue in in of itself, as it's documented in
the epair manpage:
```
The epair has RXCSUM and RXCSUM6 enabled because it may receive a packet
where the checksum has already been validated by a physical interface.
The epair supports TXCSUM and TXCSUM6 for TCP and UDP, but only by
forwarding the order to compute the checksum. Thus, when using an epair
interface, a TCP or UDP sender can offload checksum computation to a
physical interface. Note that, in case the packet does not leave the
host, the checksum is unnecessary and will be ignored if offloaded. Such
packets contain an incorrect checksum, since it is not computed yet.
```
And indeed, disabling TXCSUM by using the following prestart options in the
jail works around the issue (with a presumed performance penalty):
```
exec.prestart = "ifconfig ${epair} create up";
exec.prestart += "ifconfig ${epair}a up descr jail:${name} -txcsum";
exec.prestart += "ifconfig ${bridge} addm ${epair}a up";
```
Full jail configuration without the above workaround:
```
irc {
# STARTUP/LOGGING
exec.consolelog = "/var/log/jail_console_${name}.log";
# PERMISSIONS
allow.raw_sockets;
exec.clean;
mount.devfs;
devfs_ruleset = 5;
# PATH/HOSTNAME
path = "/usr/local/jails/containers/${name}";
host.hostname = "${name}.central-rock.home";
# VNET/VIMAGE
vnet;
vnet.interface = "${epair}b";
# NETWORKS/INTERFACES
$id = "111";
$ip = "192.168.100.${id}/24";
$gateway = "192.168.100.1";
$bridge = "bridge0";
$epair = "epair${id}";
# ADD TO bridge INTERFACE
exec.prestart = "ifconfig ${epair} create up";
exec.prestart += "ifconfig ${epair}a up descr jail:${name}";
exec.prestart += "ifconfig ${bridge} addm ${epair}a up";
exec.start = "ifconfig ${epair}b ${ip} up";
exec.start += "route add default ${gateway}";
exec.start += "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.poststop = "ifconfig ${bridge} deletem ${epair}a";
exec.poststop += "ifconfig ${epair}a destroy";
}
```
And the corresponding network rules from /etc/rc.conf:
```
hostname="central-rock.home"
defaultrouter="192.168.100.1"
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 192.168.100.100/24 addm dwc0 up"
ifconfig_dwc0="up"
```
It's quite likely that this issue is related to bug #235607, the symptoms
appear similar, but in the case of this bug, it doesn't appear to be caused by
the firewall, as I'm not using it for NAT, I'm using a bridge / virtual switch
for level 2 switching. I've tried disabling pf (my firewall) both in the host
and guest machines, and it didn't change the behavior.
I've tried reproducing the issue in a virtual machine, both in
qemu-system-x86_64 and qemu-system-aarch64, and I was unable to reproduce it,
network traffic for VNET jails with TCP checksum offloading worked correctly.
The checksum of the packets coming from the epair was initially bad, and it was
presumably corrected by the NIC. Here's an example of what tcpdump shows when
connecting to FreeBSD.org from the jail in a QEMU VM:
```
tcpdump: listening on epair16a, link-type EN10MB (Ethernet), snapshot length
262144 bytes
19:13:59.056947 IP (tos 0x0, ttl 64, id 63837, offset 0, flags [none], proto
UDP (17), length 61)
10.0.2.16.37700 > 10.0.2.3.domain: [bad udp cksum 0x184d -> 0x8e11!] 32678+
A? www.freebsd.org. (33)
19:13:59.209415 IP (tos 0x0, ttl 64, id 30, offset 0, flags [none], proto UDP
(17), length 110)
10.0.2.3.domain > 10.0.2.16.37700: [udp sum ok] 32678 q: A?
www.freebsd.org. 2/0/0 www.freebsd.org. [10s] CNAME web.geo.freebsd.org.,
web.geo.freebsd.org. [2m30s] A 85.30.190.141 (82)
19:13:59.210656 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6),
length 60)
10.0.2.16.10688 > wfe0.sjb.freebsd.org.https: Flags [S], cksum 0x1fea
(incorrect -> 0xc108), seq 4277776097, win 65535, options [mss 1460,nop,wscale
8,sackOK,TS val 3808209639 ecr 0], length 0
19:13:59.248539 IP (tos 0x0, ttl 64, id 31, offset 0, flags [none], proto TCP
(6), length 44)
wfe0.sjb.freebsd.org.https > 10.0.2.16.10688: Flags [S.], cksum 0x34ea
(correct), seq 1600001, ack 4277776098, win 65535, options [mss 1460], length 0
```
This suggests to me that FreeBSD 15 included changes to the dwc driver that
broke the checksum offloading support on the RockPro64 SBC. Taking a quick
glance at the code, it appears to have been re-factored, split up into multiple
code files from one, so it's possible that something slipped during that
process.
--
You are receiving this mail because:
You are the assignee for the bug.
lmpx.com only provides a reader for public news (NNTP) servers. It is not
affiliated with the servers or forums shown here and is not responsible for
the content of articles, which is written by their respective authors.