CM5如何安装网卡驱动？

Jake · Jul 20, 2023, 1:08 AM

我在装mellanox的时候遇到无法识别linux内核头文件的问题，这个该怎么解决呢？希望可以解答。

george · Jul 20, 2023, 3:06 AM

@Jake 这个需要把内核的头文件编译生成DEB包，然后系统上安装。

Jake · Jul 20, 2023, 7:05 AM

@george 好的，感谢，我去试试

Jachyn · Jul 21, 2023, 3:28 AM

This post is deleted!

Jachyn · Jul 21, 2023, 3:30 AM

@george

目前的错误提示如上图所示，问一下怎么把内核的头文件编译生成deb包呢？

大法师 · Jul 21, 2023, 3:51 AM

@Jachyn 我们提供的内核源码，你下载一份，先编译一次。

Jachyn · Jul 24, 2023, 4:30 AM

@大法师大师内核已经编译了编译之后进行安装又报如下的错

jack_admin · Jul 24, 2023, 4:35 AM

@Jachyn 你用的网卡驱动源码提供一个链接

大法师 · Jul 24, 2023, 5:43 AM

@jack_admin https://network.nvidia.com/products/ethernet-drivers/linux/mlnx_en/


The NVIDIA ConnectX adapter family can run Ethernet traffic simultaneously on two ports
Single software stack that operates across all available NVIDIA Ethernet devices and configurations such as mem-free, up to 400Gb/s, and PCI Express modes 3.0, 4.0 and 5.0
Support for HPC applications for scientific research and AI
Support for data center applications such as Oracle 11g/10g RAC, IBM DB2, Financial services applications such as IBM WebSphere LLM, Red Hat MRG, NYSE Data Fabric

大法师 · Jul 24, 2023, 8:58 AM

@Jachyn said in CM5如何安装网卡驱动？:

@大法师大师内核已经编译了编译之后进行安装又报如下的错

更新内核源码编译目录链接（假定源码目录在/home/coolpi/kernel）

sudo ln -sf /home/coolpi/kernel /lib/modules/5.10.110/build
sudo ln -sf /home/coolpi/kernel /lib/modules/5.10.110/source

MLNX_EN_SRC-debian-23.04-1.1.3.0.tgz

MLNX_EN_SRC-debian-5.8-3.0.7.0.tgz
解压网卡驱动源码，编译

coolpi@coolpi:~/MLNX_EN_SRC-23.04-1.1.3.0$ sudo ./install.pl 
Logs dir: /tmp/OFED.80654.logs
General log file: /tmp/OFED.80654.logs/general.log

Below is the list of OFED packages that you have chosen
(some may have been added by the installer due to package dependencies):

ofed-scripts
mlnx-tools
mlnx-ofed-kernel-utils
mlnx-ofed-kernel-modules

Checking SW Requirements...
One or more required packages for installing OFED-internal are missing.
Attempting to install the following missing packages:
swig  make python3 chrpath gcc quilt debhelper dh-python python3-distutils
This program will install the OFED package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with OFED, do not reinstall them.

Removing old packages
get_requires: libibverbs1 is required by qemu-system-arm, but qemu-system-arm won't be removed.
get_requires: librdmacm1 is required by qemu-system-arm, but qemu-system-arm won't be removed.
Building packages
Building DEB for ofed-scripts-23.04 (ofed-scripts)...
Running  /usr/bin/dpkg-buildpackage -us -uc 
Installing ofed-scripts-23.04...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/MLNX_EN_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/ofed-scripts_23.04-OFED.23.04.1.1.3_arm64.deb'
Building DEB for mlnx-tools-23.04 (mlnx-tools)...
Running  /usr/bin/dpkg-buildpackage -us -uc 
Installing mlnx-tools-23.04...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/MLNX_EN_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/mlnx-tools_23.04-0.2304053_arm64.deb'
Building DEB for mlnx-ofed-kernel-utils-23.04 (mlnx-ofed-kernel)...

-W- --with-mlx5-ipsec is enabled
Running  /usr/bin/dpkg-buildpackage -us -uc  -Pmodules
...

Jachyn · Jul 24, 2023, 6:42 AM

@大法师更新源码目录链接显示如下错误

ln: /lib/modules/5.10.110/build/kernel: cannot overwrite directory
ln: /lib/modules/5.10.110/source/kernel: cannot overwrite directory

大法师 · Jul 24, 2023, 8:00 AM

@Jachyn said in CM5如何安装网卡驱动？:

@大法师更新源码目录链接显示如下错误

ln: /lib/modules/5.10.110/build/kernel: cannot overwrite directory
ln: /lib/modules/5.10.110/source/kernel: cannot overwrite directory

sudo unlink /lib/modules/5.10.110/build
sudo unlink /lib/modules/5.10.110/source

然后再操作。编译可能有错误，你先排查下

Jachyn · Jul 24, 2023, 9:11 AM

@大法师编译内核使用命令make Image -j4，然后编译模块make modules -j4，并没有报错，内核 make modules_install 也是成功的
当前日志显示

coolpi@coolpi:~/Downloads/mlnx-en-23.04-0.5.3.3-ubuntu22.04-aarch64$ sudo ./install --force-dkms
Logs dir: /tmp/mlnx-en.6425.logs
General log file: /tmp/mlnx-en.6425.logs/general.log

Below is the list of mlnx-en packages that you have chosen
(some may have been added by the installer due to package dependencies):

ofed-scripts
mlnx-tools
mlnx-en-utils
mlnx-en-dkms
mstflint

This program will install the mlnx-en package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with mlnx-en, do not reinstall them.

Do you want to continue?[y/N]:y

Checking SW Requirements...
Removing old packages...
Uninstalling the previous version of mlnx-en
Installing new packages
Installing ofed-scripts-23.04...
Installing mlnx-tools-23.04...
Installing mlnx-en-utils-23.04...
Installing mlnx-en-dkms-23.04...
Installing mstflint-4.16.1...
Selecting previously unselected package mlnx-fw-updater.
(Reading database ... 196397 files and directories currently installed.)
Preparing to unpack .../mlnx-fw-updater_23.04-0.5.3.3_arm64.deb ...
Unpacking mlnx-fw-updater (23.04-0.5.3.3) ...
Setting up mlnx-fw-updater (23.04-0.5.3.3) ...
Initializing...
Attempting to perform Firmware update...
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX4LX
  Part Number:      MCX4121A-ACA_Ax
  Description:      ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
  PSID:             MT_2420110034
  PCI Device Name:  0001:11:00.0
  Base MAC:         98039b2ba640
  Versions:         Current        Available     
     FW             14.22.1002     14.32.1010    
     PXE            3.5.0403       3.6.0502      
     UEFI           14.15.0019     14.25.0017    

  Status:           Update required

---------
Found 1 device(s) requiring firmware update...

Device #1: Updating FW ...     
Done

Restart needed for updates to take effect.
Log File: /tmp/C3m3Iz3_P3
Real log file: /tmp/mlnx-en.6425.logs/fw_update.log
Device (0001:11:00.0):
	0001:11:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
	Link Width: x2 ( WARNING - device supports x8 )
	PCI Link Speed: 2.5GT/s

Device (0001:11:00.1):
	0001:11:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
	Link Width: x2 ( WARNING - device supports x8 )
	PCI Link Speed: 2.5GT/s

Installation passed successfully
To load the new driver, run:
/etc/init.d/mlnx-en.d restart

大法师 · Jul 24, 2023, 9:15 AM

@Jachyn 好

Jachyn · Jul 24, 2023, 9:33 AM

@大法师

coolpi@coolpi:~/Desktop$ /etc/init.d/mlnx-en.d restart
Unloading NIC driver:                                      [  OK  ]
Loading Mellanox MLX5 NIC driver:                          [FAILED]
Loading NIC driver:                                        [FAILED]

Please run /usr/sbin/sysinfo-snapshot.py to collect the debug information
and open an issue in the http://support.mellanox.com/SupportWeb/service_center/SelfService

大法师 · Jul 24, 2023, 9:34 AM

@Jachyn 加 sudo 执行试下

Jachyn · Jul 24, 2023, 9:42 AM

@大法师

coolpi@coolpi:~/Desktop$ sudo /etc/init.d/mlnx-en.d restart
Unloading NIC driver:                                      [  OK  ]
Loading Mellanox MLX5 NIC driver:                          [FAILED]
Loading NIC driver:                                        [FAILED]

Please run /usr/sbin/sysinfo-snapshot.py to collect the debug information
and open an issue in the http://support.mellanox.com/SupportWeb/service_center/SelfService

Jachyn · Jul 25, 2023, 2:34 AM

@大法师

coolpi@coolpi:~/Downloads/mlnx-en-5.9-0.5.6.0-ubuntu22.04-aarch64$ sudo ./install --add-kernel-support
Note: This program will create mlnx-en TGZ for ubuntu22.04 under /tmp/mlnx-en-5.9-0.5.6.0-5.10.110 directory.
See log file /tmp/mlnx-en-5.9-0.5.6.0-5.10.110/mlnx_iso.109096_logs/mlnx_ofed_iso.109096.log

Checking if all needed packages are installed...
Building mlnx-en DEBS . Please wait...
          
ERROR: Failed executing "MLNX_EN_SRC-5.9-0.5.6.0/install.pl --tmpdir /tmp/mlnx-en-5.9-0.5.6.0-5.10.110/mlnx_iso.109096_logs --eth-only --kernel 5.10.110 --kernel-sources /lib/modules/5.10.110/build --builddir /tmp/mlnx-en-5.9-0.5.6.0-5.10.110/mlnx_iso.109096 --without-dkms --without-debug-symbols --build-only --distro ubuntu22.04"
ERROR: See /tmp/mlnx-en-5.9-0.5.6.0-5.10.110/mlnx_iso.109096_logs/mlnx_ofed_iso.109096.log
Failed to build mlnx-en for 5.10.110

是不是这系统就不支持这个驱动呢

jack_admin · Jul 25, 2023, 6:51 AM

@Jachyn said in CM5如何安装网卡驱动？:

@大法师

coolpi@coolpi:~/Desktop$ sudo /etc/init.d/mlnx-en.d restart
Unloading NIC driver:                                      [  OK  ]
Loading Mellanox MLX5 NIC driver:                          [FAILED]
Loading NIC driver:                                        [FAILED]

Please run /usr/sbin/sysinfo-snapshot.py to collect the debug information
and open an issue in the http://support.mellanox.com/SupportWeb/service_center/SelfService

执行这个之后，系统日志输出(dmesg)分析下错误原因。

Jake · Jul 25, 2023, 7:22 AM

@george 解决了这个驱动的问题，很感谢，但是我现在遇到了一个新的问题
我发现使用网卡后仍是pci进行操作的网卡，这个我该如何解决？

Jake · Jul 25, 2023, 7:23 AM

我是不是需要去内核修改一些相关的配置？

jack_admin · Jul 25, 2023, 7:29 AM

@Jake 有什么问题？网口节点出来了没

Jake · Jul 25, 2023, 7:42 AM

@jack_admin 用ifconfig/ip addr等命令查看并没有网卡，只有lspci | grep Eth可以看到有相关网卡pcie信息，但是查看pcie日志，并没有更换为网卡驱动

Jachyn · Jul 25, 2023, 8:43 AM

@jack_admin

CMD: mlxup -u --log-on-update --ssl-certificate /tmp/SGKRmQP1We/mlxfwmanager_sriov_dis_aarch64_4115-dir/ca-bundle.crt --current-dir /opt/mellanox/mlnx-fw-updater/  -L /tmp/Nr5DC9bSSF -y -d 0001:11:00.0 
Querying Mellanox devices firmware ...

Device #1:
----------

 Device Type:      ConnectX4LX
 Part Number:      --
 Description:      
 PSID:             
 PCI Device Name:  0001:11:00.0
 Base MAC:         N/A
 Versions:         Current        Available     
    FW             --                           

 Status:           Failed to open device

---------
-E- Failed to query 0001:11:00.0 device, error : No such file or directory. MFE_ICMD_NOT_SUPPORTED

EXIT_STATUS: 0

安装网卡驱动后显示如上的日志

Jachyn · Jul 27, 2023, 7:26 AM

coolpi@coolpi:~/Downloads/mlnx-en-5.8-1.0.1.1-ubuntu22.04-aarch64$ sudo ./install --force-dkms
[sudo] password for coolpi: 
Logs dir: /tmp/mlnx-en.738979.logs
General log file: /tmp/mlnx-en.738979.logs/general.log

Below is the list of mlnx-en packages that you have chosen
(some may have been added by the installer due to package dependencies):

ofed-scripts
mlnx-tools
mlnx-en-utils
mlnx-en-dkms
mstflint

This program will install the mlnx-en package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with mlnx-en, do not reinstall them.

Do you want to continue?[y/N]:y

Checking SW Requirements...
Removing old packages...
Uninstalling the previous version of mlnx-en
Installing new packages
Installing ofed-scripts-5.8...
Installing mlnx-tools-5.2.0...
Installing mlnx-en-utils-5.8...
Installing mlnx-en-dkms-5.8...
Installing mstflint-4.16.1...
Selecting previously unselected package mlnx-fw-updater.
(Reading database ... 199866 files and directories currently installed.)
Preparing to unpack .../mlnx-fw-updater_5.8-1.0.1.1_arm64.deb ...
Unpacking mlnx-fw-updater (5.8-1.0.1.1) ...
Setting up mlnx-fw-updater (5.8-1.0.1.1) ...
Initializing...
Attempting to perform Firmware update...
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX4LX
  Part Number:      MCX4121A-ACA_Ax
  Description:      ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
  PSID:             MT_2420110034
  PCI Device Name:  0001:11:00.0
  Base MAC:         98039b2ba640
  Versions:         Current        Available     
     FW             14.32.1010     14.32.1010    
     PXE            3.6.0502       3.6.0502      
     UEFI           14.25.0017     14.25.0017    

  Status:           Up to date


Log File: /tmp/DR3i76jLzy
Real log file: /tmp/mlnx-en.738979.logs/fw_update.log
Device (0001:11:00.0):
	0001:11:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
	Link Width: x2 ( WARNING - device supports x8 )
	PCI Link Speed: 2.5GT/s

Device (0001:11:00.1):
	0001:11:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
	Link Width: x2 ( WARNING - device supports x8 )
	PCI Link Speed: 2.5GT/s

Installation passed successfully
To load the new driver, run:
/etc/init.d/mlnx-en.d restart
coolpi@coolpi:~/Downloads/mlnx-en-5.8-1.0.1.1-ubuntu22.04-aarch64$ /etc/init.d/mlnx-en.d restart
Unloading NIC driver:                                      [  OK  ]
Loading Mellanox MLX5 NIC driver:                          [FAILED]
Loading NIC driver:                                        [FAILED]

Please run /usr/sbin/sysinfo-snapshot.py to collect the debug information
and open an issue in the http://support.mellanox.com/SupportWeb/service_center/SelfService

Jachyn · Jul 27, 2023, 7:39 AM

coolpi@coolpi:~/Downloads/mlnx-en-5.8-2.0.3.0-ubuntu22.04-aarch64$ sudo ./install --force-dkms
[sudo] password for coolpi: 
Logs dir: /tmp/mlnx-en.3438.logs
General log file: /tmp/mlnx-en.3438.logs/general.log

Below is the list of mlnx-en packages that you have chosen
(some may have been added by the installer due to package dependencies):

ofed-scripts
mlnx-tools
mlnx-en-utils
mlnx-en-dkms
mstflint

This program will install the mlnx-en package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with mlnx-en, do not reinstall them.

Do you want to continue?[y/N]:y

Checking SW Requirements...
Removing old packages...
Uninstalling the previous version of mlnx-en
Installing new packages
Installing ofed-scripts-5.8...
Installing mlnx-tools-5.2.0...
Installing mlnx-en-utils-5.8...
Installing mlnx-en-dkms-5.8...
Installing mstflint-4.16.1...
Selecting previously unselected package mlnx-fw-updater.
(Reading database ... 199866 files and directories currently installed.)
Preparing to unpack .../mlnx-fw-updater_5.8-2.0.3.0_arm64.deb ...
Unpacking mlnx-fw-updater (5.8-2.0.3.0) ...
Setting up mlnx-fw-updater (5.8-2.0.3.0) ...
Initializing...
Attempting to perform Firmware update...
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX4LX
  Part Number:      MCX4121A-ACA_Ax
  Description:      ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
  PSID:             MT_2420110034
  PCI Device Name:  0001:11:00.0
  Base MAC:         98039b2ba640
  Versions:         Current        Available     
     FW             14.32.1010     14.32.1010    
     PXE            3.6.0502       3.6.0502      
     UEFI           14.25.0017     14.25.0017    

  Status:           Up to date


Log File: /tmp/05iafkfb8h
Real log file: /tmp/mlnx-en.3438.logs/fw_update.log
Device (0001:11:00.0):
	0001:11:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
	Link Width: x2 ( WARNING - device supports x8 )
	PCI Link Speed: 8GT/s

Device (0001:11:00.1):
	0001:11:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
	Link Width: x2 ( WARNING - device supports x8 )
	PCI Link Speed: 8GT/s

Installation passed successfully
To load the new driver, run:
/etc/init.d/mlnx-en.d restart
coolpi@coolpi:~/Downloads/mlnx-en-5.8-2.0.3.0-ubuntu22.04-aarch64$ /etc/init.d/mlnx-en.d restart
Unloading NIC driver:                                      [  OK  ]
Loading Mellanox MLX5 NIC driver:                          [FAILED]
Loading NIC driver:                                        [FAILED]

Please run /usr/sbin/sysinfo-snapshot.py to collect the debug information
and open an issue in the http://support.mellanox.com/SupportWeb/service_center/SelfService

coolpi@coolpi:~/Downloads/mlnx-en-5.8-2.0.3.0-ubuntu22.04-aarch64$

Jachyn · Jul 27, 2023, 7:59 AM

coolpi@coolpi:~/Downloads/mlnx-en-5.6-2.0.9.0-ubuntu22.04-aarch64$ sudo ./install --force-dkms
[sudo] password for coolpi: 
Logs dir: /tmp/mlnx-en.514532.logs
General log file: /tmp/mlnx-en.514532.logs/general.log

Below is the list of mlnx-en packages that you have chosen
(some may have been added by the installer due to package dependencies):

ofed-scripts
mlnx-tools
mlnx-en-utils
mlnx-en-dkms
mstflint

This program will install the mlnx-en package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with mlnx-en, do not reinstall them.

Do you want to continue?[y/N]:y

Checking SW Requirements...
Removing old packages...
Uninstalling the previous version of mlnx-en
Installing new packages
Installing ofed-scripts-5.6...
Installing mlnx-tools-5.2.0...
Installing mlnx-en-utils-5.6...
Installing mlnx-en-dkms-5.6...
Installing mstflint-4.16.1...
Selecting previously unselected package mlnx-fw-updater.
(Reading database ... 199769 files and directories currently installed.)
Preparing to unpack .../mlnx-fw-updater_5.6-2.0.9.0_arm64.deb ...
Unpacking mlnx-fw-updater (5.6-2.0.9.0) ...
Setting up mlnx-fw-updater (5.6-2.0.9.0) ...
Initializing...
Attempting to perform Firmware update...
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX4LX
  Part Number:      MCX4121A-ACA_Ax
  Description:      ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
  PSID:             MT_2420110034
  PCI Device Name:  0001:11:00.0
  Base MAC:         98039b2ba640
  Versions:         Current        Available     
     FW             14.32.1010     14.32.1010    
     PXE            3.6.0502       3.6.0502      
     UEFI           14.25.0017     14.25.0017    

  Status:           Up to date


Log File: /tmp/07OcqgbJK0
Real log file: /tmp/mlnx-en.514532.logs/fw_update.log
Device (0001:11:00.0):
	0001:11:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
	Link Width: x2 ( WARNING - device supports x8 )
	PCI Link Speed: 8GT/s

Device (0001:11:00.1):
	0001:11:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
	Link Width: x2 ( WARNING - device supports x8 )
	PCI Link Speed: 8GT/s

Installation passed successfully
To load the new driver, run:
/etc/init.d/mlnx-en.d restart
coolpi@coolpi:~/Downloads/mlnx-en-5.6-2.0.9.0-ubuntu22.04-aarch64$ /etc/init.d/mlnx-en.d restart
Unloading NIC driver:                                      [  OK  ]
Loading Mellanox MLX5 NIC driver:                          [FAILED]
Loading NIC driver:                                        [FAILED]

Please run /usr/sbin/sysinfo-snapshot.py to collect the debug information
and open an issue in the http://support.mellanox.com/SupportWeb/service_center/SelfService

Jachyn · Jul 27, 2023, 8:09 AM

coolpi@coolpi:~/Downloads/mlnx-en-5.7-1.0.2.0-ubuntu22.04-aarch64$ sudo ./install --force-dkms
[sudo] password for coolpi: 
Logs dir: /tmp/mlnx-en.759187.logs
General log file: /tmp/mlnx-en.759187.logs/general.log

Below is the list of mlnx-en packages that you have chosen
(some may have been added by the installer due to package dependencies):

ofed-scripts
mlnx-tools
mlnx-en-utils
mlnx-en-dkms
mstflint

This program will install the mlnx-en package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with mlnx-en, do not reinstall them.

Do you want to continue?[y/N]:y

Checking SW Requirements...
Removing old packages...
Uninstalling the previous version of mlnx-en
Installing new packages
Installing ofed-scripts-5.7...
Installing mlnx-tools-5.2.0...
Installing mlnx-en-utils-5.7...
Installing mlnx-en-dkms-5.7...
Installing mstflint-4.16.1...
Selecting previously unselected package mlnx-fw-updater.
(Reading database ... 199851 files and directories currently installed.)
Preparing to unpack .../mlnx-fw-updater_5.7-1.0.2.0_arm64.deb ...
Unpacking mlnx-fw-updater (5.7-1.0.2.0) ...
Setting up mlnx-fw-updater (5.7-1.0.2.0) ...
Initializing...
Attempting to perform Firmware update...
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX4LX
  Part Number:      MCX4121A-ACA_Ax
  Description:      ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
  PSID:             MT_2420110034
  PCI Device Name:  0001:11:00.0
  Base MAC:         98039b2ba640
  Versions:         Current        Available     
     FW             14.32.1010     14.32.1010    
     PXE            3.6.0502       3.6.0502      
     UEFI           14.25.0017     14.25.0017    

  Status:           Up to date


Log File: /tmp/FFuM6_xJPq
Real log file: /tmp/mlnx-en.759187.logs/fw_update.log
Device (0001:11:00.0):
	0001:11:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
	Link Width: x2 ( WARNING - device supports x8 )
	PCI Link Speed: 8GT/s

Device (0001:11:00.1):
	0001:11:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
	Link Width: x2 ( WARNING - device supports x8 )
	PCI Link Speed: 8GT/s

Installation passed successfully
To load the new driver, run:
/etc/init.d/mlnx-en.d restart
coolpi@coolpi:~/Downloads/mlnx-en-5.7-1.0.2.0-ubuntu22.04-aarch64$ /etc/init.d/mlnx-en.d restart
Unloading NIC driver:                                      [  OK  ]
Loading Mellanox MLX5 NIC driver:                          [FAILED]
Loading NIC driver:                                        [FAILED]

Please run /usr/sbin/sysinfo-snapshot.py to collect the debug information
and open an issue in the http://support.mellanox.com/SupportWeb/service_center/SelfService

jack_admin · Jul 28, 2023, 6:36 AM

@Jachyn 需要把驱动加载过程，内核打印日志输出看看错误在哪，可以用dmesg命令

Jachyn · Jul 28, 2023, 6:09 AM

root@coolpi:/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0# ./install.pl 
Logs dir: /tmp/OFED.127412.logs
General log file: /tmp/OFED.127412.logs/general.log

Below is the list of OFED packages that you have chosen
(some may have been added by the installer due to package dependencies):

ofed-scripts
mlnx-tools
mlnx-ofed-kernel-utils
mlnx-ofed-kernel-modules
srp-modules
mlnx-nvme-modules
rdma-core
libibverbs1
ibverbs-utils
ibverbs-providers
libibverbs-dev
libibverbs1-dbg
libibumad3
libibumad-dev
ibacm
librdmacm1
rdmacm-utils
librdmacm-dev
mstflint
libibmad5
libibmad-dev
libibnetdisc5
infiniband-diags
kernel-mft-modules
perftest
ibsim
ibsim-doc
ucx
openmpi
mpitests
knem-modules
srptools
mlnx-ethtool
mlnx-iproute2
rshim
ibarr

Checking SW Requirements...
One or more required packages for installing OFED-internal are missing.
Attempting to install the following missing packages:
build-essential autotools-dev pkg-config  python3-docutils debhelper libstdc++6 pandoc bison dh-autoreconf devscripts libmnl-dev fakeroot libgfortran5 cython3 libnuma-dev libnl-route-3-dev libssl-dev python3-dev libnl-3-dev autoconf quilt libfuse-dev python3-distutils flex libsystemd-dev libselinux1-dev libudev-dev dh-python cmake libpci-dev sudo make libdb-dev libusb-1.0-0-dev libelf-dev libtool valgrind python3 gcc bzip2
This program will install the OFED package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with OFED, do not reinstall them.

Removing old packages
get_requires: libibverbs1 is required by qemu-system-arm, but qemu-system-arm won't be removed.
get_requires: librdmacm1 is required by qemu-system-arm, but qemu-system-arm won't be removed.

Error: One or more packages depends on MLNX_OFED.
Those packages should be removed before uninstalling MLNX_OFED:

licensecheck libarray-intspan-perl

To force uninstallation use '--force' flag.
root@coolpi:/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0# ./install.pl --force
Logs dir: /tmp/OFED.129276.logs
General log file: /tmp/OFED.129276.logs/general.log

Below is the list of OFED packages that you have chosen
(some may have been added by the installer due to package dependencies):

ofed-scripts
mlnx-tools
mlnx-ofed-kernel-utils
mlnx-ofed-kernel-modules
srp-modules
mlnx-nvme-modules
rdma-core
libibverbs1
ibverbs-utils
ibverbs-providers
libibverbs-dev
libibverbs1-dbg
libibumad3
libibumad-dev
ibacm
librdmacm1
rdmacm-utils
librdmacm-dev
mstflint
libibmad5
libibmad-dev
libibnetdisc5
infiniband-diags
kernel-mft-modules
perftest
ibsim
ibsim-doc
ucx
openmpi
mpitests
knem-modules
srptools
mlnx-ethtool
mlnx-iproute2
rshim
ibarr

Checking SW Requirements...
One or more required packages for installing OFED-internal are missing.
Attempting to install the following missing packages:
flex libelf-dev fakeroot autoconf autotools-dev dh-python libtool libfuse-dev sudo debhelper python3-docutils libnl-route-3-dev pandoc bison libselinux1-dev quilt python3 libgfortran5 build-essential valgrind libsystemd-dev libssl-dev cmake libnl-3-dev libudev-dev cython3 python3-distutils libdb-dev libpci-dev python3-dev  devscripts bzip2 libmnl-dev gcc libstdc++6 pkg-config libnuma-dev libusb-1.0-0-dev make dh-autoreconf
This program will install the OFED package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with OFED, do not reinstall them.

Removing old packages
get_requires: libibverbs1 is required by qemu-system-arm, but qemu-system-arm won't be removed.
get_requires: librdmacm1 is required by qemu-system-arm, but qemu-system-arm won't be removed.
Building packages
Building DEB for ofed-scripts-23.04 (ofed-scripts)...
Running  /usr/bin/dpkg-buildpackage -us -uc 
Installing ofed-scripts-23.04...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/ofed-scripts_23.04-OFED.23.04.1.1.3_arm64.deb'
Building DEB for mlnx-tools-23.04 (mlnx-tools)...
Running  /usr/bin/dpkg-buildpackage -us -uc 
Installing mlnx-tools-23.04...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/mlnx-tools_23.04-0.2304053_arm64.deb'
Building DEB for mlnx-ofed-kernel-utils-23.04 (mlnx-ofed-kernel)...
Running  /usr/bin/dpkg-buildpackage -us -uc  -Pmodules
Installing mlnx-ofed-kernel-utils-23.04...
Running /usr/bin/dpkg -i --force-confnew --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/mlnx-ofed-kernel-utils_23.04-OFED.23.04.1.1.3.1.kver.5.10.110_arm64.deb'
Installing mlnx-ofed-kernel-modules-23.04...
Running /usr/bin/dpkg -i --force-confnew --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/mlnx-ofed-kernel-modules_23.04-OFED.23.04.1.1.3.1.kver.5.10.110_arm64.deb'
Building DEB for srp-modules-23.04 (srp)...
Running  /usr/bin/dpkg-buildpackage -us -uc  -Pmodules
Installing srp-modules-23.04...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/srp-modules_23.04-OFED.23.04.0.5.3.1.kver.5.10.110_arm64.deb'
Building DEB for mlnx-nvme-modules-23.04 (mlnx-nvme)...
Running  /usr/bin/dpkg-buildpackage -us -uc  -Pmodules
Installing mlnx-nvme-modules-23.04...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/mlnx-nvme-modules_23.04-OFED.23.04.0.5.3.1.kver.5.10.110_arm64.deb'
Building DEB for rdma-core-2304mlnx44 (rdma-core)...
Running  /usr/bin/dpkg-buildpackage -us -uc 
Installing rdma-core-2304mlnx44...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/rdma-core_2304mlnx44-1.2304113_arm64.deb'
Installing libibverbs1-2304mlnx44...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/libibverbs1_2304mlnx44-1.2304113_arm64.deb'
Installing ibverbs-utils-2304mlnx44...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/ibverbs-utils_2304mlnx44-1.2304113_arm64.deb'
Installing ibverbs-providers-2304mlnx44...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/ibverbs-providers_2304mlnx44-1.2304113_arm64.deb'
Installing libibverbs-dev-2304mlnx44...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/libibverbs-dev_2304mlnx44-1.2304113_arm64.deb'
Installing libibverbs1-dbg-2304mlnx44...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/libibverbs1-dbg_2304mlnx44-1.2304113_arm64.deb'
Installing libibumad3-2304mlnx44...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/libibumad3_2304mlnx44-1.2304113_arm64.deb'
Installing libibumad-dev-2304mlnx44...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/libibumad-dev_2304mlnx44-1.2304113_arm64.deb'
Installing ibacm-2304mlnx44...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/ibacm_2304mlnx44-1.2304113_arm64.deb'
Installing librdmacm1-2304mlnx44...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/librdmacm1_2304mlnx44-1.2304113_arm64.deb'
Installing rdmacm-utils-2304mlnx44...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/rdmacm-utils_2304mlnx44-1.2304113_arm64.deb'
Installing librdmacm-dev-2304mlnx44...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/librdmacm-dev_2304mlnx44-1.2304113_arm64.deb'
Building DEB for mstflint-4.16.1 (mstflint)...
Running  /usr/bin/dpkg-buildpackage -us -uc 
Installing mstflint-4.16.1...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/mstflint_4.16.1-2.2304053_arm64.deb'
Installing libibmad5-2304mlnx44...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/libibmad5_2304mlnx44-1.2304113_arm64.deb'
Installing libibmad-dev-2304mlnx44...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/libibmad-dev_2304mlnx44-1.2304113_arm64.deb'
Installing libibnetdisc5-2304mlnx44...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/libibnetdisc5_2304mlnx44-1.2304113_arm64.deb'
Installing infiniband-diags-2304mlnx44...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/infiniband-diags_2304mlnx44-1.2304113_arm64.deb'
Building DEB for kernel-mft-modules-4.24.0 (kernel-mft)...
Running  /usr/bin/dpkg-buildpackage -us -uc  -Pmodules
Installing kernel-mft-modules-4.24.0...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/kernel-mft-modules_4.24.0-72.kver.5.10.110_all.deb'
Building DEB for perftest-23.04.0 (perftest)...
Running  /usr/bin/dpkg-buildpackage -us -uc 
Installing perftest-23.04.0...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/perftest_23.04.0-0.23.g63e250f.2304053_arm64.deb'
Building DEB for ibsim-0.12 (ibsim)...
Running  /usr/bin/dpkg-buildpackage -us -uc 
Installing ibsim-0.12...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/ibsim_0.12-1.2304053_arm64.deb'
Installing ibsim-doc-0.12...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/ibsim-doc_0.12-1.2304053_all.deb'
Building DEB for ucx-1.15.0 (ucx)...
Running  /usr/bin/dpkg-buildpackage -us -uc 
Installing ucx-1.15.0...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/ucx_1.15.0-1.2304053_arm64.deb'
Building DEB for openmpi-4.1.5rc2 (openmpi)...
Running  mellanox_autodetect='yes' /usr/bin/dpkg-buildpackage -us -uc 







Installing openmpi-4.1.5rc2...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/openmpi_4.1.5rc2-1.2304053_all.deb'
Building DEB for mpitests-3.2.20 (mpitests)...
Running  /usr/bin/dpkg-buildpackage -us -uc 
Installing mpitests-3.2.20...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/mpitests_3.2.20-de56b6b.2304053_arm64.deb'
Building DEB for knem-modules-1.1.4.90mlnx2 (knem)...
Running  /usr/bin/dpkg-buildpackage -us -uc  -Pmodules
Installing knem-modules-1.1.4.90mlnx2...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/knem-modules_1.1.4.90mlnx2-OFED.23.04.0.5.2.1.kver.5.10.110_arm64.deb'
Installing srptools-2304mlnx44...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/srptools_2304mlnx44-1.2304113_arm64.deb'
Building DEB for mlnx-ethtool-6.0 (mlnx-ethtool)...
Running  /usr/bin/dpkg-buildpackage -us -uc 
Installing mlnx-ethtool-6.0...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/mlnx-ethtool_6.0-1.2304053_arm64.deb'
Building DEB for mlnx-iproute2-6.2.0 (mlnx-iproute2)...
Running  /usr/bin/dpkg-buildpackage -us -uc 
Installing mlnx-iproute2-6.2.0...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/mlnx-iproute2_6.2.0-1.2304053_arm64.deb'
Building DEB for rshim-2.0.6 (rshim)...
Running  /usr/bin/dpkg-buildpackage -us -uc 
Installing rshim-2.0.6...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/rshim_2.0.6-19.g0873acd.2304113_arm64.deb'
Building DEB for ibarr-0.1.3 (ibarr)...
Running  /usr/bin/dpkg-buildpackage -us -uc 
Installing ibarr-0.1.3...
Running /usr/bin/dpkg -i --force-confmiss '/home/coolpi/Downloads/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-aarch64/src/MLNX_OFED_SRC-23.04-1.1.3.0/DEBS/ubuntu22.04/aarch64/ibarr_0.1.3-1.2304053_arm64.deb'
Device (0001:11:00.0):
	0001:11:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
	Link Width: x2 ( WARNING - device supports x8 )
	PCI Link Speed: 2.5GT/s

Device (0001:11:00.1):
	0001:11:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
	Link Width: x2 ( WARNING - device supports x8 )
	PCI Link Speed: 2.5GT/s

Installation passed successfully