[lustre-devel] Request of Lustre with Linux kernel 5.10.X and MLNX_OFED 5.3.X

Greetings,

Recently we received new machines, which would join our existing cluster
to be Lustre clients. However, due to its pretty new hardware specs
(Intel 40G ethernet chip, Broadcom Megaraid RAID, ConnectX-5 Infiniband ....
etc), we are force to install somewhat newer versions of system software:

- Linux vallina kernel 5.10.153
  https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.10.153.tar.xz

- MLNX_OFED_SRC-5.6-2.0.9.0
  https://www.mellanox.com/downloads/ofed/MLNX_OFED-5.6-2.0.9.0/MLNX_OFED_SRC-debian-5.6-2.0.9.0.tgz

- Debian/GNU Linux 10.13.

The choice of Linux vallina kernel 5.10.153 is necessary to support the
ethernet and RAID chips. The choice of MLNX_OFED_SRC-5.6-2.0.9.0 is to
be successfully built together with Linux vallina kernel 5.10.153.

However, we faced a problem of building Lustre-2.15.1. Without
MLNX_OFED_SRC-5.6-2.0.9.0, Lustre-2.15.1 can be successfully built. But
adding MLNX_OFED_SRC-5.6-2.0.9.0 there are errors.

Here are our procedures:

1. We build vanilla Linux kernel 5.10.153, with Linux kernel options
   shown in the attachment (config-5.10.153).

2. We compiled and installed the whole MLNX_OFED_SRC-5.6-2.0.9.0 packages
   using the following command:

        ./install.pl --hpc

   After that, we confirmed that our Infiniband network was active.

3. We manually extract:

   MLNX_OFED_SRC-5.6-2.0.9.0/SOURCES/mlnx-ofed-kernel_5.6.orig.tar.gz

   and configured it, compiled it, in order to provide the configured
   source for Lustre-2.15.1 for building. The configure command is:

   ./configure --kernel-version=5.10.153 \
               --kernel-sources=/usr/src/linux-5.10.153 \
               --with-core-mod --with-ipoib-mod --with-ipoib-cm \
               --with-user_mad-mod --with-srp-mod --with-rxe-mod \
               --with-user_access-mod --with-addr_trans-mod \
               --with-mlx5-mod --with-mlx5_core-mod \
               --with-mlx5-core-and-ib-and-en-mod \
               --with-mlx5_inf-mod --with-madeye-mod
   make

   The whole process runs successfully.

4. Finally we tried to compile Lustre-2.15.1. Our command is:

   ./configure --prefix=/opt/lustre --with-linux=/usr/src/linux-5.10.153 \
      --with-o2ib=/usr/src/MLNX_OFED_SRC-5.6-2.0.9.0/work/mlnx-ofed-kernel_5.6 \
      --disable-server --enable-mpitests=no

   However, the configure command failed with:
   (please see the attached config.log for details)

   configure:19909: error: can't compile with OpenIB gen2 headers under /usr/src/MLNX_OFED_SRC-5.6-2.0.9.0/work/mlnx-ofed-kernel-5.6


We think that currently Lustre does not support MLNX_OFED to such a new
version (5.6-2.0.9.0). Unfortunately, for our new machines this is needed.
So we would like to make a request to support MLNX_OFED at least up to
version 5.6-2.0.9.0, for future releases of Lustre. We understand that
this is not a trivial work. We would be highly appreciated if one day it
could come true.


Thank you very much.

T.H.Hsieh

_______________________________________________
lustre-devel mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org