[dpdk] dpdk編譯成動態庫使用 -- PCI port自動發現與pmd動態加載

 

1.  修改配置文件 .conf, 設置如下變量的值.

[root@D129 x86_64-native-linuxapp-gcc]# cat dpdk/x86_64-native-linuxapp-gcc/.config |grep SHARE
CONFIG_RTE_BUILD_SHARED_LIB=y

 

2.  這個時候, 再編譯的 dpdk app就會自動鏈接dpdk的動態庫. 如下:

[root@D129 app]# ldd testpmd
        linux-vdso.so.1 =>  (0x00007ffed89fd000)
        librte_kni.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_kni.so.2 (0x00007fe9c983f000)
        librte_pipeline.so.3 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_pipeline.so.3 (0x00007fe9c962b000)
        librte_table.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_table.so.2 (0x00007fe9c93ed000)
        librte_port.so.3 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_port.so.3 (0x00007fe9c9191000)
        librte_pdump.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_pdump.so.1 (0x00007fe9c8d77000)
        librte_distributor.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_distributor.so.1 (0x00007fe9c8b74000)
        librte_reorder.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_reorder.so.1 (0x00007fe9c8965000)
        librte_ip_frag.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_ip_frag.so.1 (0x00007fe9c8724000)
        librte_meter.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_meter.so.1 (0x00007fe9c8521000)
        librte_sched.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_sched.so.1 (0x00007fe9c830d000)
        librte_lpm.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_lpm.so.2 (0x00007fe9c80ff000)
        librte_acl.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_acl.so.2 (0x00007fe9c7ee5000)
        librte_jobstats.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_jobstats.so.1 (0x00007fe9c7ce2000)
        librte_power.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_power.so.1 (0x00007fe9c7acd000)
        librte_timer.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_timer.so.1 (0x00007fe9c78c3000)
        librte_hash.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_hash.so.2 (0x00007fe9c76af000)
        librte_vhost.so.3 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_vhost.so.3 (0x00007fe9c7477000)
        librte_kvargs.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_kvargs.so.1 (0x00007fe9c7274000)
        librte_mbuf.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_mbuf.so.2 (0x00007fe9c7071000)
        libethdev.so.4 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/libethdev.so.4 (0x00007fe9c6dd6000)
        librte_cryptodev.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_cryptodev.so.1 (0x00007fe9c6bc1000)
        librte_mempool.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_mempool.so.2 (0x00007fe9c69b9000)
        librte_ring.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_ring.so.1 (0x00007fe9c67b5000)
        librte_eal.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_eal.so.2 (0x00007fe9c6546000)
        librte_cmdline.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_cmdline.so.2 (0x00007fe9c633b000)
        librte_cfgfile.so.2 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_cfgfile.so.2 (0x00007fe9c6137000)
        librte_pmd_bond.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_pmd_bond.so.1 (0x00007fe9c5efa000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fe9c5cd9000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fe9c5ad4000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe9c58b8000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fe9c54f5000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fe9c9a51000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fe9c51f2000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fe9c4fea000)
[root@D129 app]# 

 

3.  但是與static的時候對比, 你會發現有如下的問題:  

　　用static鏈接的時候, rte_init的時候,會掃描所有的PCI設備,找到所有可用的port, 如下:

[root@D129 app]# ./testpmd
EAL: Detected 16 lcore(s)
EAL: Probing VFIO support...
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
EAL: PCI device 0000:00:04.0 on NUMA socket -1
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=267456, size=2304, socket=0
RING: Cannot reserve memory
EAL: Error - exiting with code: 1
  Cause: Creation of mbuf pool for socket 0 failed: Cannot allocate memory
[root@D129 app]# 

　　在使用shared so庫的時候, 會發現,dpdk app掃描不到任何 PCI 設備了. 如下:

[root@D129 app]# ./testpmd 
EAL: Detected 16 lcore(s)
EAL: Probing VFIO support...
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
EAL: No probed ethernet devices
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=267456, size=2304, socket=0
RING: Cannot reserve memory
EAL: Error - exiting with code: 1
  Cause: Creation of mbuf pool for socket 0 failed: Cannot allocate memory

　　這個時候,我們ldd app,會發現, 它並沒有ld到pmd的so,

[root@D129 app]# ldd testpmd |grep pmd
        librte_pmd_bond.so.1 => /root/Src/copyright/j/dpdk/x86_64-native-linuxapp-gcc/lib/librte_pmd_bond.so.1 (0x00007fa25689a000)
[root@D129 app]# 

　　猜測, dpdk在shared的情況下, pmd的庫,是需要動態加載的. 並找到如下參數:

[root@D129 app]# ./testpmd -h |grep -A1 LIB.so 
  -d LIB.so|DIR       Add a driver or driver directory
                      (can be used multiple times)

　　使用 -d 參數制定 virtio的so, (我的虛擬機是virtio的網絡設備), 果然生效了

[root@D129 app]# ./testpmd  -d ../lib/librte_pmd_virtio.so
EAL: Detected 16 lcore(s)
EAL: Probing VFIO support...
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
EAL: PCI device 0000:00:04.0 on NUMA socket -1
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=267456, size=2304, socket=0
RING: Cannot reserve memory
EAL: Error - exiting with code: 1
  Cause: Creation of mbuf pool for socket 0 failed: Cannot allocate memory
[root@D129 app]# 

 

4.  問題分析

　　參考 dpdk 的源碼 ../dpdk/lib/librte_eal/common/eal_common_options.c::eal_plugins_init()

　　在讀取 -d 參數的同時, dpdk在 rte_init函數中還會讀取 RTE_EAL_PMD_PATH 目錄下的文件,  所有讀到的文件會使用 dlopen() 打開. 詳細邏輯參見代碼.

　　變量RTE_EAL_PMD_PATH的值由CONFIG_RTE_EAL_PMD_PATH在編譯的時候決定.

 

5.  參考redhat的rpm spec, 最終的解決方案是這樣的.

5.1  生成一個編譯與運行時都存在的目錄: /lib64/tong-dpdk-pmds/, 

5.2  將所以pmd軟鏈接到這個目錄下

[root@D129 app]# ll /lib64/tong-dpdk-pmds/
total 0
lrwxrwxrwx 1 root root 28 Jul 25 13:37 librte_pmd_af_packet.so.1 -> ../librte_pmd_af_packet.so.1
lrwxrwxrwx 1 root root 23 Jul 25 13:37 librte_pmd_bnxt.so.1 -> ../librte_pmd_bnxt.so.1
lrwxrwxrwx 1 root root 23 Jul 25 13:37 librte_pmd_bond.so.1 -> ../librte_pmd_bond.so.1
lrwxrwxrwx 1 root root 24 Jul 25 13:37 librte_pmd_cxgbe.so.1 -> ../librte_pmd_cxgbe.so.1
lrwxrwxrwx 1 root root 24 Jul 25 13:37 librte_pmd_e1000.so.1 -> ../librte_pmd_e1000.so.1
lrwxrwxrwx 1 root root 22 Jul 25 13:37 librte_pmd_ena.so.1 -> ../librte_pmd_ena.so.1
lrwxrwxrwx 1 root root 23 Jul 25 13:37 librte_pmd_enic.so.1 -> ../librte_pmd_enic.so.1
lrwxrwxrwx 1 root root 24 Jul 25 13:37 librte_pmd_fm10k.so.1 -> ../librte_pmd_fm10k.so.1
lrwxrwxrwx 1 root root 23 Jul 25 13:37 librte_pmd_i40e.so.1 -> ../librte_pmd_i40e.so.1
lrwxrwxrwx 1 root root 24 Jul 25 13:37 librte_pmd_ixgbe.so.1 -> ../librte_pmd_ixgbe.so.1
lrwxrwxrwx 1 root root 30 Jul 25 13:37 librte_pmd_null_crypto.so.1 -> ../librte_pmd_null_crypto.so.1
lrwxrwxrwx 1 root root 23 Jul 25 13:37 librte_pmd_null.so.1 -> ../librte_pmd_null.so.1
lrwxrwxrwx 1 root root 23 Jul 25 13:37 librte_pmd_ring.so.2 -> ../librte_pmd_ring.so.2
lrwxrwxrwx 1 root root 24 Jul 25 13:37 librte_pmd_vhost.so.1 -> ../librte_pmd_vhost.so.1
lrwxrwxrwx 1 root root 25 Jul 25 13:37 librte_pmd_virtio.so.1 -> ../librte_pmd_virtio.so.1
lrwxrwxrwx 1 root root 30 Jul 25 13:37 librte_pmd_vmxnet3_uio.so.1 -> ../librte_pmd_vmxnet3_uio.so.1

5.2 在編譯前, 設置變量CONFIG_RTE_EAL_PMD_PATH

[root@D129 app]# cat ../.config |grep PATH
CONFIG_RTE_EAL_PMD_PATH="/lib64/tong-dpdk-pmds/"

 

以上, dpdk app便可以使用 shared lib 正常運行了.

 

6. 另一個方案:  使用 -d, 需要神馬加神馬, 比如:

[root@D129 app]# ./testpmd  -d ../lib/librte_pmd_virtio.so  -d ../lib/librte_pmd_ixgbe.so

 

完.