rdma centos 7.3安裝
1、安裝依賴包
yum install epel-release -y yum install gcc gcc-c++ bc openssl-devel automake ncurses-devel libibverbs -y yum install libibverbs-devel libibverbs-utils librdmacm librdmacm-devel librdmacm-utils perl-Switch elfutils-libelf-devel -y
2、 librxe-dev 和 rxe-dev下載
下載地址
Github: https://github.com/SoftRoCE/rxe-dev.git Github: https://github.com/SoftRoCE/librxe-dev.git
備注：rxe-dev下載v18版本，即rxe-dev-rxesubmissionv18
3、安裝rxe-dev
unzip rxe-dev-rxe_submission_v18.zip cd rxe-dev-rxe_submission_v18/ cp /boot/config-3.10.0-514.el7.x86_64 .config
備注：使用root用戶，執行以下命令
make menuconfig
會出現選擇界面（如果沒出現，需要安裝 ncurse-devel）
輸入 "/" ，然后輸入 rxe，按下 enter，會查找有關 rxe 的選擇項。
輸入數字 1，就會選擇到“Software RDMA over Ethernet (ROCE) driver”的設置，輸入 "M" ，選中 RDMA 的配置，如果 輸不了 M，那就輸入空格。
移動到保存按鈕，回車，裝保存到.config中，退出安裝界面（exit）。
然后 vi .config 來確認
CONFIGRDMARXE 為 m
CONFIGINFINIBANDADDRTRANS 和 CONFIGINFINIBANDADDRTRANS_CONFIGFS 為 y
make -j 4 make modules_install ，可能執行中途 會提示 丟失一些 module，這個 沒關系，無關緊要。 make install make headers_install INSTALL_HDR_PATH=/usr
確認 新的內核是否在 grub 引導中。查看 /etc/grub.cfg 即可看見。在開機的時候可以選擇 新內核啟動
4、安裝 librxe-dev
cd librxe-dev ./configure --libdir=/usr/lib64/ --prefix= make make install
Here is the issue:
checking for ibv_get_device_list in -libverbs...
yes
checking infiniband/driver.h usability... no
checking infiniband/driver.h presence... no
checking for infiniband/driver.h... no
configure: error: <infiniband/driver.h> not found. librxe requires libibverbs.
How to fix?
重啟操作系統，在開機啟動時，選擇4.7.0-rc3內核
啟動后，查看內核版本
uname -r
5、驗證 rdma
[root@aboss ~]# rxe_cfg start Name Link Driver Speed NMTU IPv4_addr RDEV RMTU ens33 yes e1000 virbr0 no bridge virbr0-nic no tun [root@aboss ~]# rxe_cfg add ens33 [root@aboss ~]# rxe_cfg status Name Link Driver Speed NMTU IPv4_addr RDEV RMTU ens33 yes e1000 rxe0 1024 (3) virbr0 no bridge virbr0-nic no tun
查看rxe設備
ibvdevices 程序顯示該系統中目前所有設備，而 ibvdevinfo 命令會給出每個設備的具體信息
[root@aboss ~]# ibv_devices device node GUID ------ ---------------- rxe0 020c29fffe55c818 [root@aboss ~]# ibv_devinfo rxe0 hca_id: rxe0 transport: InfiniBand (0) fw_ver: 0.0.0 node_guid: 020c:29ff:fe55:c818 sys_p_w_picpath_guid: 0000:0000:0000:0000 vendor_id: 0x0000 vendor_part_id: 0 hw_ver: 0x0 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet
6、softRoCE連通性測試
服務端
rping -s -a 192.168.1.133 -v -C 10
客戶端
rping -c -a 192.168.1.133 -v -C 10
Test connectivity.
- On the server:
1 |
ibv_rc_pingpong -d rxe0 -g 0 |
- On the client:
1 |
ibv_rc_pingpong -d rxe0 -g 0 <server_management_ip> |
e.g Client:
1 |
kevin@ubuntu:~$ ibv_rc_pingpong -g 0 -d rxe0 -i 1 192.168.188.129 |
測試時需切換到root
Sever:
1 |
ib_send_bw –a |
Client:
1 |
ib_send_bw 192.168.46.132 –a |
7、關於librdmacm編譯說明
git clone https://github.com/ofiwg/librdmacm.git cd librdmacm yum install autoconf automake gettext gettext-devel libtool -y ./autogen.sh ./configure make make install
8、常見問題
(1)如果你克隆虛機，需要解決網卡問題
(2)使用rdma，請將防火牆與selinx關閉
How to build the development environment of Software RDMA over Converged Ethernet (RoCE) ?
Soft-RoCE (RXE)
In order to study the coding of RDMA, I need build the corresponding environment. Due to the lack of hardware, I found Soft-RoCE would be the first choice after some research work.
All the information focus on these two repositories: [rxe-dev] and [librxe-dev]. And others aim to help us how to build this virtual RDMA device which named RXE and how to use it.
Struggle Against RXE
Internet shows that the RXE need kernel support and user space codes.
Firstly, we git clone the [rxe-dev] to compile and install the new kernel to support RoCE. Then restart with the new kernel. Finally, compile the [librxe-dev] to get the utilities of Soft-Roce.
When I switched to the compiled new kernel, but it failed to start the system. When I configured the [librxe-dev], it showed “configure: error: <infiniband/driver.h> not found. librxe requires libibverbs”.
This page also shows the same issue, and several people also hangs on this error.
Sudden Inspiration
MosesAlexander’s comment of “I just found that the rxe functionality is all in rdma-core now.” gives me sudden inspiration. The rdma-core has already contained all the RXE utilities, and it looks like that there’s no maintenance for the two repositories [rxe-dev] and [librxe-dev].
Solution
Just “yum -y install libibverbs libibverbs-devel libibverbs-utils librdmacm librdmacm-devel librdmacm-utils” will be OK !
Notice: I blog this post at 2018/09/03, based on CentOS 7 (3.10.0-862.el7.x86_64), whose kernel support RDMA-related technologies originally.
Verification
Run some commands to verify:
[root@localhost ~]# rxe_cfg start Name Link Driver Speed NMTU IPv4_addr RDEV RMTU ens33 yes e1000 virbr0 no bridge virbr0-nic no tun [root@localhost ~]# rxe_cfg add ens33 [root@localhost ~]# rxe_cfg status Name Link Driver Speed NMTU IPv4_addr RDEV RMTU ens33 yes e1000 rxe0 1024 (3) virbr0 no bridge virbr0-nic no tun [root@localhost ~]# ibv_devices device node GUID ------ ---------------- rxe0 020c29fffe495c4d
Also, you could run example codes from the-geek-in-the-corner.
參考：
http://blog.sina.com.cn/s/blog_6de3aa8a0102wr14.html
http://www.unjeep.com/article/23742.html （rping測試，softroce/rdma安裝測試）
https://github.com/SoftRoCE/rxe-dev/wiki/Validate-that-RXE-is-working （驗證rxe是否工作）