Re: Chelsio / cxlv(4): strange messages, SR-IOV interface does not work

On Tue, Oct 22, 2024 at 11:21 PM Lexi Winter <[email protected]> wrote:

> hello,
>
> i'm trying to configure a cxlv(4) device, which is a VF of a Chelsio
> T540-CR on a host running bhyve.
>
> host: FreeBSD 15.0-CURRENT #3 lf/main-n269068-2cff93ced1d: Wed Oct 23
> 02:48:20 BST 2024
> guest: FreeBSD 15.0-CURRENT #2 lf/main-n269067-56dd459904b: Sat Oct 19
> 18:36:40 BST 2024
>
> the VF appears correctly in the VM:
>
> root@lily:~ # kldload if_cxlv
> t5vf0: <Chelsio T540-CR VF> mem
> 0xc000e000-0xc000efff,0xc0000000-0xc0007fff,0xc0008000-0xc0009fff at
> device 6.0 on pci0
> t5vf0: 1 ports, 2 MSI-X interrupts, 4 eq, 2 iq
> cxlv0: <port 0> on t5vf0
> cxlv0: 2 txq, 1 rxq (NIC)
>
> and after bringing the interface 'up' everything seems fine:
>
> root@lily:~ # ifconfig cxlv0
> cxlv0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP>
> metric 0 mtu 1500
>
>
> options=6ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,HWRXTSTMP,MEXTPG>
>          ether 06:44:3f:e7:60:30
>          media: Ethernet 10Gbase-Twinax <full-duplex> (10Gbase-Twinax
> <full-duplex,rxpause,txpause>)
>          status: active
>          nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>
> however, trying to assign an IP address causes immediate problems:
>
> root@lily:~ # ifconfig cxlv0 inet6 2001:8b0:aab5:7::10/64
> root@lily:~ # Oct 23 06:16:07 lily kernel: cxlv0: a looped back NS
> message is detected during DAD for fe80:3::444:3fff:fee7:6030.  Another
> DAD probes are being sent.
> root@lily:~ # dmesg|grep loop
> cxlv0: a looped back NS message is detected during DAD for
> fe80:3::444:3fff:fee7:6030.  Another DAD probes are being sent.
> cxlv0: a looped back NS message is detected during DAD for
> fe80:3::444:3fff:fee7:6030.  Another DAD probes are being sent.
> cxlv0: a looped back NS message is detected during DAD for
> fe80:3::444:3fff:fee7:6030.  Another DAD probes are being sent.
> cxlv0: a looped back NS message is detected during DAD for
> fe80:3::444:3fff:fee7:6030.  Another DAD probes are being sent.
> cxlv0: a looped back NS message is detected during DAD for
> fe80:3::444:3fff:fee7:6030.  Another DAD probes are being sent.
> cxlv0: a looped back NS message is detected during DAD for
> fe80:3::444:3fff:fee7:6030.  Another DAD probes are being sent.
> cxlv0: a looped back NS message is detected during DAD for
> fe80:3::444:3fff:fee7:6030.  Another DAD probes are being sent.
>
> i find this strange because the link local IP address in the kernel
> error is not even configured on the interface:
>
> root@lily:~ # ifconfig cxlv0
> cxlv0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP>
> metric 0 mtu 1500
>
>
> options=6ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,HWRXTSTMP,MEXTPG>
>          ether 06:44:3f:e7:60:30
>          inet6 2001:8b0:aab5:7::10/64
>          inet6 fe80::444:3fff:fee7:6030%cxlv0/64 tentative scopeid 0x3
>          media: Ethernet 10Gbase-Twinax <full-duplex> (10Gbase-Twinax
> <full-duplex,rxpause,txpause>)
>          status: active
>          nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
>
> am i doing something wrong here?
>

You can disable IPv6 DAD as a workaround for this issue.  The problem is
that the VF's multicast tx is getting reflected back to it by the internal
switch when it shouldn't.

# sysctl net.inet6.ip6.dad_count=0

Regards,
Navdeep


>
>         thanks, lexi.
>
>