Oracle S7-2 and ConnextX4-LX, ConnectX3 (VPI), and i40e

For those with an Oracle S7-2, but this may be valid for other Sparc 
systems as well: (just a heads-up)

1)  The "ConnectX-4 lx" NIC appears to work with Solaris 11.4 and Debian 
12.  However, on Debian 12, the "ConnectX-4 lx" will generate many 
tainted kernel messages (related to mlx5 module).  Tried latest 
firmwares for this card, (from Dell and Nvidia), all have the same 
issues.  In my case, the tainted messages appear when configured to use 
systemd-networkd and during system boot/reboot/poweroff.

2) As such, decided to downgrade and use "ConnectX-3 CX354A-FCBT 
(VPI)".  This particular card works with Solaris 11.4 and Debian 12.  It 
also works in Ethernet and Infiniband modes.  Both Solaris 11.4 and 
Debian 12 see this nic in either mode.  It uses the mlx4 module and 
appears to avoid generating tainted kernel messages.


Note(S):

2A) For infiniband verbs, the (rdma-core package) a.k.a. 
(infiniband-diags package replacement) will need to add a specific build 
option to create the ibverbs executables.   I did so by building from 
source. Might be helpful to modify the rdma-core build so it will 
automatically generate all ibverbs such as ibstat, ibstatus, etc.

2B) opensm package has a dependency with infiniband-diags package.  
However, the infiniband-diags package will not install (possibly because 
of rdma-core).  Guessing opensm package will require an update to use 
rdma-core and possibly dropping infiniband-diags.  Didn't attempt to 
build opensm from scratch, so I can't say for sure yet. Unsure if 
infiniband-diags is required by other packages at this point.

2C) On another note, the "ConnectX-3 CX314A-BCCT" will work with Debian 
12 in Ethernet only mode, but it is not detected/recognized by Solaris 
11.4.  If Solaris 11.4, Debian 12, Ethernet, and Infiniband 
compatibility are required, make sure to use the "ConnectX-3 CX354A-FCBT 
(VPI)" NIC.


3) The built-in i40e 10GBE ports on the S7-2 generate the following 
messages within Debian 12. It works and doesn't appear to be a major 
issue. Just unsure if this unaligned access may impact performance on 
some level?  iperf3 didn't show any noticeable performance drop, but it 
may be something to keep an eye on.

For Example:
i40e 0000:03:00.1: DCB init failed -16, disabled
Kernel unaligned access at TPC[101a7f28] i40e_config_netdev+0x1e8/0x2e0 
[i40e]

Regards,

Tony