[Bug 280648] Traffic leak between fibs

[email protected]
Newsgroups gmane.os.freebsd.devel.net
Message-ID <[email protected]/bugzilla/> 
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=280648

--- Comment #7 from Egor <[email protected]> ---
(In reply to Tatsuki Makino from comment #2)

I reproduced this problem in my lab. Config is:

1) Asus 750 hypervisor with proxmox 8.4:

pve-fw01:~$ uname -a
Linux pve-fw01 6.8.8-4-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.8-4
(2024-07-26T11:15Z) x86_64 GNU/Linux

2) Freebsd virtual machine with linked mellanox connectx6 pci card:

test-fw01:~$ uname -a
FreeBSD test-fw01 14.1-RELEASE-p4 FreeBSD 14.1-RELEASE-p4 GENERIC amd64

I catched traffic for all interfaces with enabled pf and with disabled pf.
There is no difference.

Tcpdump with enabled pf

test-fw01:~$ sudo tcpdump -nei mce1.1280 host 172.16.188.194 and port 22
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on mce1.1280, link-type EN10MB (Ethernet), snapshot length 262144
bytes
16:50:53.753073 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 74: 172.16.179.42.57836 > 172.16.188.194.22: Flags [S], seq 3922742222,
win 64240, options [mss 1460,sackOK,TS val 3411125639 ecr 0,nop,wscale 7],
length 0
16:50:54.816302 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 74: 172.16.179.42.57836 > 172.16.188.194.22: Flags [S], seq 3922742222,
win 64240, options [mss 1460,sackOK,TS val 3411126703 ecr 0,nop,wscale 7],
length 0
16:50:55.840297 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 74: 172.16.179.42.57836 > 172.16.188.194.22: Flags [S], seq 3922742222,
win 64240, options [mss 1460,sackOK,TS val 3411127727 ecr 0,nop,wscale 7],
length 0
16:50:56.864293 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 74: 172.16.179.42.57836 > 172.16.188.194.22: Flags [S], seq 3922742222,
win 64240, options [mss 1460,sackOK,TS val 3411128751 ecr 0,nop,wscale 7],
length 0
16:50:57.888290 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 74: 172.16.179.42.57836 > 172.16.188.194.22: Flags [S], seq 3922742222,
win 64240, options [mss 1460,sackOK,TS val 3411129775 ecr 0,nop,wscale 7],
length 0

test-fw01:~$ sudo tcpdump -nei mce1.3101 host 172.16.188.194 and port 22
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on mce1.3101, link-type EN10MB (Ethernet), snapshot length 262144
bytes
16:50:53.753130 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 74: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446,
ack 3922742223, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val
4184867862 ecr 3411125639], length 0
16:50:54.760769 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 74: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446,
ack 3922742223, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val
4184868868 ecr 3411125639], length 0
16:50:54.816333 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 74: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446,
ack 3922742223, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val
4184868920 ecr 3411126703], length 0
16:50:55.820324 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 74: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446,
ack 3922742223, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val
4184869929 ecr 3411126703], length 0
16:50:55.840332 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 74: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446,
ack 3922742223, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val
4184869949 ecr 3411127727], length 0
16:50:56.841091 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 74: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446,
ack 3922742223, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val
4184870950 ecr 3411127727], length 0
16:50:56.864323 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 74: 172.16.188.194.22 > 172.16.179.42.57836: Flags [S.], seq 1825143446,
ack 3922742223, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val
4184870969 ecr 3411128751], length 0

test-fw01:~$ sudo tcpdump -ner /var/log/pflog host 172.16.188.194 and port
57836
reading from file /var/log/pflog, link-type PFLOG (OpenBSD pflog file),
snapshot length 116
16:50:53.753073 rule 5/0(match): pass in on mce1.1280: 172.16.179.42.57836 >
172.16.188.194.22: Flags [S], seq 3922742222, win 64240, options [mss 1460,
[|tcp]
16:50:53.753130 rule 1/0(match): block in on mce1.3101: 172.16.188.194.22 >
172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535,
options [mss 1460, [|tcp]
16:50:54.760769 rule 1/0(match): block in on mce1.3101: 172.16.188.194.22 >
172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535,
options [mss 1460, [|tcp]
16:50:54.816333 rule 1/0(match): block in on mce1.3101: 172.16.188.194.22 >
172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535,
options [mss 1460, [|tcp]
16:50:55.820324 rule 1/0(match): block in on mce1.3101: 172.16.188.194.22 >
172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535,
options [mss 1460, [|tcp]
16:50:55.840332 rule 1/0(match): block in on mce1.3101: 172.16.188.194.22 >
172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535,
options [mss 1460, [|tcp]
16:50:56.841091 rule 1/0(match): block in on mce1.3101: 172.16.188.194.22 >
172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535,
options [mss 1460, [|tcp]
16:50:56.864323 rule 1/0(match): block in on mce1.3101: 172.16.188.194.22 >
172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535,
options [mss 1460, [|tcp]
16:50:57.863064 rule 1/0(match): block in on mce1.3101: 172.16.188.194.22 >
172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535,
options [mss 1460, [|tcp]
16:50:57.888331 rule 1/0(match): block in on mce1.3101: 172.16.188.194.22 >
172.16.179.42.57836: Flags [S.], seq 1825143446, ack 3922742223, win 65535,
options [mss 1460, [|tcp]

Tcpdump with disabled pf

test-fw01:~$ sudo tcpdump -nei mce1.1280 host 172.16.188.194 and port 22
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on mce1.1280, link-type EN10MB (Ethernet), snapshot length 262144
bytes

17:01:42.533010 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 74: 172.16.179.42.34620 > 172.16.188.194.22: Flags [S], seq 215584557,
win 64240, options [mss 1460,sackOK,TS val 3411775040 ecr 0,nop,wscale 7],
length 0
17:01:42.533054 b8:3f:d2:1c:e2:09 > d0:09:c8:ca:09:27, ethertype IPv4 (0x0800),
length 74: 172.16.188.194.22 > 172.16.179.42.34620: Flags [S.], seq 1105243323,
ack 215584558, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val
2313997153 ecr 3411775040], length 0
17:01:42.533144 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 66: 172.16.179.42.34620 > 172.16.188.194.22: Flags [.], ack 1, win 502,
options [nop,nop,TS val 3411775040 ecr 2313997153], length 0
17:01:42.533505 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 108: 172.16.179.42.34620 > 172.16.188.194.22: Flags [P.], seq 1:43, ack
1, win 502, options [nop,nop,TS val 3411775040 ecr 2313997153], length 42: SSH:
SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.5
17:01:42.566078 b8:3f:d2:1c:e2:09 > d0:09:c8:ca:09:27, ethertype IPv4 (0x0800),
length 66: 172.16.188.194.22 > 172.16.179.42.34620: Flags [.], ack 43, win 129,
options [nop,nop,TS val 2313997190 ecr 3411775040], length 0
17:02:04.656826 b8:3f:d2:1c:e2:09 > d0:09:c8:ca:09:27, ethertype IPv4 (0x0800),
length 104: 172.16.188.194.22 > 172.16.179.42.34620: Flags [P.], seq 1:39, ack
43, win 129, options [nop,nop,TS val 2314019278 ecr 3411775040], length 38:
SSH: SSH-2.0-OpenSSH_9.7 FreeBSD-20240806
17:02:04.656940 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 66: 172.16.179.42.34620 > 172.16.188.194.22: Flags [.], ack 39, win 502,
options [nop,nop,TS val 3411797164 ecr 2314019278], length 0
17:02:04.657554 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 1514: 172.16.179.42.34620 > 172.16.188.194.22: Flags [.], seq 43:1491,
ack 39, win 502, options [nop,nop,TS val 3411797165 ecr 2314019278], length
1448
17:02:04.657554 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 154: 172.16.179.42.34620 > 172.16.188.194.22: Flags [P.], seq 1491:1579,
ack 39, win 502, options [nop,nop,TS val 3411797165 ecr 2314019278], length 88
17:02:04.657604 b8:3f:d2:1c:e2:09 > d0:09:c8:ca:09:27, ethertype IPv4 (0x0800),
length 66: 172.16.188.194.22 > 172.16.179.42.34620: Flags [.], ack 1579, win
126, options [nop,nop,TS val 2314019278 ecr 3411797165], length 0
17:02:04.657843 b8:3f:d2:1c:e2:09 > d0:09:c8:ca:09:27, ethertype IPv4 (0x0800),
length 1186: 172.16.188.194.22 > 172.16.179.42.34620: Flags [P.], seq 39:1159,
ack 1579, win 126, options [nop,nop,TS val 2314019278 ecr 3411797165], length
1120
17:02:04.698103 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 66: 172.16.179.42.34620 > 172.16.188.194.22: Flags [.], ack 1159, win
501, options [nop,nop,TS val 3411797206 ecr 2314019278], length 0
17:02:04.792136 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 1274: 172.16.179.42.34620 > 172.16.188.194.22: Flags [P.], seq
1579:2787, ack 1159, win 501, options [nop,nop,TS val 3411797300 ecr
2314019278], length 1208
17:02:04.802961 b8:3f:d2:1c:e2:09 > d0:09:c8:ca:09:27, ethertype IPv4 (0x0800),
length 1514: 172.16.188.194.22 > 172.16.179.42.34620: Flags [.], seq 1159:2607,
ack 2787, win 129, options [nop,nop,TS val 2314019418 ecr 3411797300], length
1448
17:02:04.802963 b8:3f:d2:1c:e2:09 > d0:09:c8:ca:09:27, ethertype IPv4 (0x0800),
length 150: 172.16.188.194.22 > 172.16.179.42.34620: Flags [P.], seq 2607:2691,
ack 2787, win 129, options [nop,nop,TS val 2314019418 ecr 3411797300], length
84

test-fw01:~$ sudo tcpdump -nei mce1.3101 host 172.16.188.194 and port 22
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on mce1.3101, link-type EN10MB (Ethernet), snapshot length 262144
bytes
17:01:42.533054 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 74: 172.16.188.194.22 > 172.16.179.42.34620: Flags [S.], seq 1105243323,
ack 215584558, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val
2313997153 ecr 3411775040], length 0
17:01:42.566078 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 66: 172.16.188.194.22 > 172.16.179.42.34620: Flags [.], ack 43, win 129,
options [nop,nop,TS val 2313997190 ecr 3411775040], length 0
17:02:04.656826 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 104: 172.16.188.194.22 > 172.16.179.42.34620: Flags [P.], seq 1:39, ack
43, win 129, options [nop,nop,TS val 2314019278 ecr 3411775040], length 38:
SSH: SSH-2.0-OpenSSH_9.7 FreeBSD-20240806
17:02:04.657604 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 66: 172.16.188.194.22 > 172.16.179.42.34620: Flags [.], ack 1579, win
126, options [nop,nop,TS val 2314019278 ecr 3411797165], length 0
17:02:04.657843 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 1186: 172.16.188.194.22 > 172.16.179.42.34620: Flags [P.], seq 39:1159,
ack 1579, win 126, options [nop,nop,TS val 2314019278 ecr 3411797165], length
1120
17:02:04.802961 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 1514: 172.16.188.194.22 > 172.16.179.42.34620: Flags [.], seq 1159:2607,
ack 2787, win 129, options [nop,nop,TS val 2314019418 ecr 3411797300], length
1448
17:02:04.802963 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 150: 172.16.188.194.22 > 172.16.179.42.34620: Flags [P.], seq 2607:2691,
ack 2787, win 129, options [nop,nop,TS val 2314019418 ecr 3411797300], length
84
17:02:05.514898 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 66: 172.16.188.194.22 > 172.16.179.42.34620: Flags [.], ack 2871, win
129, options [nop,nop,TS val 2314019519 ecr 3411797357], length 0
17:02:05.515063 d0:09:c8:ca:09:27 > b8:3f:d2:1c:e2:09, ethertype IPv4 (0x0800),
length 110: 172.16.188.194.22 > 172.16.179.42.34620: Flags [P.], seq 2691:2735,
ack 2915, win 129, options [nop,nop,TS val 2314019519 ecr 3411797408], length
44

routing tables info

test-fw01:~$ sudo netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            172.16.188.193     UGS      vtnet0
10.222.254.254     link#3             UHS         lo0
10.222.254.254/31  link#2             U        vtnet1
127.0.0.1          link#3             UH          lo0
172.16.188.192/26  link#1             U        vtnet0
172.16.188.194     link#3             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif
Expire
::/96                             link#3                        URS         lo0
::1                               link#3                        UHS         lo0
::ffff:0.0.0.0/96                 link#3                        URS         lo0
fe80::%lo0/10                     link#3                        URS         lo0
fe80::%lo0/64                     link#3                        U           lo0
fe80::1%lo0                       link#3                        UHS         lo0
ff02::/16                         link#3                        URS         lo0

test-fw01:~$ sudo setfib 1 netstat -rn
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
127.0.0.1          link#3             UHS         lo0
172.16.0.0/16      link#3             U1B         lo0
172.16.176.0/26    192.168.255.0      UG1    mce0.124
172.16.176.0/26    192.168.255.2      UG1    mce1.124
172.16.176.64/26   192.168.255.0      UG1    mce0.124
172.16.176.64/26   192.168.255.2      UG1    mce1.124
172.16.176.128/26  192.168.255.0      UG1    mce0.124
172.16.176.128/26  192.168.255.2      UG1    mce1.124
172.16.176.192/27  192.168.255.0      UG1    mce0.124
172.16.176.192/27  192.168.255.2      UG1    mce1.124
172.16.177.0/26    192.168.255.16     UG1    mce0.125
172.16.177.0/26    192.168.255.18     UG1    mce1.125
172.16.177.64/27   192.168.255.16     UG1    mce0.125
172.16.177.64/27   192.168.255.18     UG1    mce1.125
172.16.177.96/27   192.168.255.16     UG1    mce0.125
172.16.177.96/27   192.168.255.18     UG1    mce1.125
172.16.177.128/27  192.168.255.16     UG1    mce0.125
172.16.177.128/27  192.168.255.18     UG1    mce1.125
172.16.178.0/28    192.168.255.32     UG1    mce0.126
172.16.178.0/28    192.168.255.34     UG1    mce1.126
172.16.178.32/27   192.168.255.32     UG1    mce0.126
172.16.178.32/27   192.168.255.34     UG1    mce1.126
172.16.179.0/28    192.168.255.64     UG1    mce0.128
172.16.179.0/28    192.168.255.66     UG1    mce1.128
172.16.179.16/29   192.168.255.64     UG1    mce0.128
172.16.179.16/29   192.168.255.66     UG1    mce1.128
172.16.179.24/30   192.168.255.64     UG1    mce0.128
172.16.179.24/30   192.168.255.66     UG1    mce1.128
172.16.179.28/30   192.168.255.64     UG1    mce0.128
172.16.179.28/30   192.168.255.66     UG1    mce1.128
172.16.179.32/30   192.168.255.64     UG1    mce0.128
172.16.179.32/30   192.168.255.66     UG1    mce1.128
172.16.179.36/30   192.168.255.64     UG1    mce0.128
172.16.179.36/30   192.168.255.66     UG1    mce1.128
172.16.179.40/30   192.168.255.64     UG1    mce0.128
172.16.179.40/30   192.168.255.66     UG1    mce1.128
172.16.179.48/30   192.168.255.64     UG1    mce0.128
172.16.179.48/30   192.168.255.66     UG1    mce1.128
172.16.179.64/28   192.168.255.64     UG1    mce0.128
172.16.179.64/28   192.168.255.66     UG1    mce1.128
172.16.180.0/27    192.168.255.48     UG1    mce0.127
172.16.180.0/27    192.168.255.50     UG1    mce1.127
172.16.180.32/27   192.168.255.48     UG1    mce0.127
172.16.180.32/27   192.168.255.50     UG1    mce1.127
172.16.181.0/28    192.168.255.80     UG1    mce0.131
172.16.181.0/28    192.168.255.82     UG1    mce1.131
172.17.0.0/16      link#3             U1B         lo0
172.17.160.0/27    192.168.255.96     UG1    mce0.149
172.17.160.0/27    192.168.255.98     UG1    mce1.149
172.17.160.32/27   192.168.255.96     UG1    mce0.149
172.17.160.32/27   192.168.255.98     UG1    mce1.149
172.17.160.64/27   192.168.255.96     UG1    mce0.149
172.17.160.64/27   192.168.255.98     UG1    mce1.149
172.17.161.0/24    192.168.255.96     UG1    mce0.149
172.17.161.0/24    192.168.255.98     UG1    mce1.149
192.168.255.0/31   link#6             U      mce0.124
192.168.255.1      link#3             UHS         lo0
192.168.255.2/31   link#7             U      mce1.124
192.168.255.3      link#3             UHS         lo0
192.168.255.16/31  link#8             U      mce0.125
192.168.255.17     link#3             UHS         lo0
192.168.255.18/31  link#9             U      mce1.125
192.168.255.19     link#3             UHS         lo0
192.168.255.32/31  link#10            U      mce0.126
192.168.255.33     link#3             UHS         lo0
192.168.255.34/31  link#11            U      mce1.126
192.168.255.35     link#3             UHS         lo0
192.168.255.48/31  link#12            U      mce0.127
192.168.255.49     link#3             UHS         lo0
192.168.255.50/31  link#13            U      mce1.127
192.168.255.51     link#3             UHS         lo0
192.168.255.64/31  link#14            U      mce0.128
192.168.255.65     link#3             UHS         lo0
192.168.255.66/31  link#15            U      mce1.128
192.168.255.67     link#3             UHS         lo0
192.168.255.80/31  link#16            U      mce0.131
192.168.255.81     link#3             UHS         lo0
192.168.255.82/31  link#17            U      mce1.131
192.168.255.83     link#3             UHS         lo0
192.168.255.96/31  link#18            U      mce0.149
192.168.255.97     link#3             UHS         lo0
192.168.255.98/31  link#19            U      mce1.149
192.168.255.99     link#3             UHS         lo0
192.168.255.112/31 link#22            U      mce0.310
192.168.255.113    link#3             UHS         lo0
192.168.255.114/31 link#23            U      mce1.310
192.168.255.115    link#3             UHS         lo0
192.168.255.144/31 link#20            U      mce0.310
192.168.255.145    link#3             UHS         lo0
192.168.255.146/31 link#21            U      mce1.310
192.168.255.147    link#3             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif
Expire
::/96                             link#3                        URS         lo0
::1                               link#3                        UHS         lo0
::ffff:0.0.0.0/96                 link#3                        URS         lo0
fe80::%lo0/10                     link#3                        URS         lo0
ff02::/16                         link#3                        URS         lo0

-- 
You are receiving this mail because:
You are the assignee for the bug.
lmpx.com only provides a reader for public news (NNTP) servers. It is not affiliated with the servers or forums shown here and is not responsible for the content of articles, which is written by their respective authors.