Roce ofed環境搭建與測試
一、安裝包下載：
mellanox驅動下載地址：
1、進入到網址：https://cn.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed
2、在打開的頁面上找到自己平台，如：Linux SW/Drivers，這里以centos 8為例；
3、在頁面的下方找到對應的版本進行下載；
這里以tar.gz的格式進行相應的說明使用；
二、安裝
1、將下載好的驅動包上傳到服務器，上傳的步驟這里不敘述；
2、解壓上傳好的驅動包，等待解壓完成；
[root@localhost ~]# tar xzvf MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64.tgz
./MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64/
./MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64/RPM-GPG-KEY-Mellanox
./MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64/uninstall.sh
./MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64/.mlnx
./MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64/.arch
./MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64/distro
………………………
3、安裝
[root@localhost ~]# cd MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64/
[root@localhost MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.1-x86_64]# ./mlnxofedinstall
Logs dir: /tmp/MLNX_OFED_LINUX.10300.logs
General log file: /tmp/MLNX_OFED_LINUX.10300.logs/general.log
Verifying KMP rpms compatibility with target kernel...
……………………….
Complete!
等待系統安裝完成即可；
三、常用檢查配置；
1、InfiniBand 狀態：
[root@centos222 ~]# ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 1
Firmware version: 2.42.5000
Hardware version: 1
Node GUID: 0xf452140300880760
System image GUID: 0xf452140300880760
Port 1:
State: Active
Physical state: LinkUp
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0xf65214fffe880760
Link layer: Ethernet
2、InfiniBand 狀態：
[root@centos222 ~]# ibstatus
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:f652:14ff:fe88:0760
base lid: 0x0
sm lid: 0x0
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 10 Gb/sec (1X QDR)
link_layer: Ethernet
3、網卡的對應關系：
[root@centos222 ~]# ibdev2netdev
mlx4_0 port 1 ==> enp2s0 (Up)
4、網卡協商相關信息：
[root@centos222 ~]# ethtool enp2s0
Settings for enp2s0:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseKX/Full
10000baseKR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 1000baseKX/Full
10000baseKR/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000014 (20)
link ifdown
Link detected: yes
5、網卡支持的gid等相關信息：
[root@centos222 ~]# show_gids
DEV PORT INDEX GID IPv4 VER DEV
--- ---- ----- --- ------------ --- ---
mlx4_0 1 0 fe80:0000:0000:0000:f652:14ff:fe88:0760 v1 enp2s0
n_gids_found=1
6、網卡工作模式：
[root@centos222 ~]# ibstatus
Infiniband device 'mlx4_0' port 1 status:
default gid: fe80:0000:0000:0000:f652:14ff:fe88:0760
base lid: 0x0
sm lid: 0x0
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 10 Gb/sec (1X QDR)
link_layer: Ethernet
查看網卡當前的link 工作模式：
[root@centos7221 ~]# connectx_port_config -s
--------------------------------
Port configuration for PCI device: 0000:86:00.0 is:
eth
--------------------------------
[root@centos7221 ~]# connectx_port_config
ConnectX PCI devices :
|----------------------------|
| 1 0000:86:00.0 |
|----------------------------|
Before port change:
eth
|----------------------------|
| Possible port modes: |
| 1: Infiniband |
| 2: Ethernet |
| 3: AutoSense |
|----------------------------|
Select mode for port 1 (1,2,3):
按照需要進行選擇；
Note:
Connectx-3只支持Ethernet模式
7、解釋：
$ ib_send_bw -d mlx5_4 -x 3 //在一邊服務器上啟動收包測試, 用index3, RoCEv2:這里要注意的是Index 值；
$ sudo ib_send_bw -d mlx5_4 192.168.1.1 --report_gbits -F -x 3 //另外一邊發包
Note:
# 記得給你的網卡綁定個IP, 兩邊能ping通
[root@centos222 ~]# nmcli c modify enp2s0 ipv4.addresses 10.10.10.222/24 autoconnect yes ipv4.method manual
[root@centos222 ~]# nmcli c down enp2s0
Connection 'enp2s0' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/439)
[root@centos222 ~]# nmcli c reload enp2s0
[root@centos222 ~]# nmcli c up enp2s0
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/440)
[root@centos222 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 6c:92:bf:70:97:cc brd ff:ff:ff:ff:ff:ff
inet 192.168.101.222/24 brd 192.168.101.255 scope global noprefixroute eno1
valid_lft forever preferred_lft forever
inet6 fe80::aad6:ae47:a954:b791/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: enp4s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 6c:92:bf:70:97:cd brd ff:ff:ff:ff:ff:ff
4: enp132s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 68:91:d0:61:57:2e brd ff:ff:ff:ff:ff:ff
5: enp132s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 68:91:d0:61:57:2f brd ff:ff:ff:ff:ff:ff
6: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether f4:52:14:88:07:60 brd ff:ff:ff:ff:ff:ff
inet 10.10.10.222/24 brd 10.10.10.255 scope global noprefixroute enp2s0
valid_lft forever preferred_lft forever
inet6 fe80::da78:33ac:bf32:8856/64 scope link noprefixroute
valid_lft forever preferred_lft forever
7: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:53:81:7e brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
8: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
link/ether 52:54:00:53:81:7e brd ff:ff:ff:ff:ff:ff
四、QA；
在安裝的過程中可能出現很多問題，最常見的就是缺少安裝包，可先安裝缺少的包，再次安裝驅動；
下面是安裝驅動必須的包：
tcl tcsh gcc-gfortran tk python36 perl
在centos 上直接聯網進行更新安裝即可；
yum install tcl tcsh gcc-gfortran tk python36 perl