Igb_uio內核驅動
Igb_uio驅動主要做的就是注冊一個pci設備。但是igbuio_pci_driver對應的保存pci設備信息的id_table指針為空，這樣在內核注冊此pci設備時，會找不到匹配的設備，就不會調用igb_uio驅動中的探測probe函數，只會在/sys目錄下創建Igb_uio相應的目錄。
如何probe
在插入igb_uio.ko時是probe不到設備的，那是什么時候偵測到的呢？dpdk提供了一個Python腳本dpdk_nic_bind.py。
python tools/dpdk_nic_bind.py --bind=igb_uio eth1
運行上述命令就是將eth1網卡綁定到igb_uio模塊。這時dmesg就會看到igb_uio模塊的probe函數執行了，也就是意味着掃描到了匹配的pci設備。
經過分析dpdk_nic_bind.py，此腳本文件在上述情況下主要做了以下幾步：
◆獲取參數指定的網卡eth1的設備信息。使用lspci–Dvmmn查看。
| Slot: 0000:06:00.1 Class: 0200 Vendor: 8086 Device: 1521 SVendor: 15d9 SDevice: 1521 Rev: 01 |
可以查看到slot槽位信息、廠商號vendor ID、設備號device ID等信息。
◆unbind之前的igb模塊。
將前面獲取到的eth1對應的slot信息0000:06:00.1值寫入igb的unbind文件。
echo 0000:06:00.1 > /sys/bus/pci/drivers/igb/unbind
從內核代碼分析此unbind的動作就是將igb模塊信息和此pci設備Dev去關聯。將dev->driver指針置為空，這個很重要。在內核處理pci設備注冊的函數中，就算驅動的vendor ID和device ID與設備的都匹配上了，如果此設備的dev->driver指針不為空，也不會調用probe函數的。
◆bind新的igb_uio模塊
將eth1設備的vendor和device ID信息寫入igb_uio的new_id文件。
echo 0x8086 0x1521 > /sys/bus/pci/drivers/igb_uio/new_id
內核中處理此步的函數為store_new_id，此函數中是將寫入的vendor和device存入到此driver，也就是igb_uio的id_table，然后以此與PCI上的設備進行匹配，這個時候肯定會匹配成功，然后調用igb_uio模塊的probe函數進行初始化動作。
echo 1 > /sys/bus/pci/rescan
echo 1 > /sys/bus/pci/devices/0000:00:1e.0/rescan
Other Network devices ===================== 0000:06:00.0 'Hi1822 Family (2*25GE) 0200' unused=hinic,igb_uio,vfio-pci
[root@localhost dpdk-19.11]# ls /sys/bus/pci/drivers ahci ehci-pci hibmc-drm hisi_sas_v3_hw igb_uio megaraid_sas ohci-pci pci-stub uhci_hcd xhci_hcd ata_piix exar_serial hinic hns3 ipmi_si nvme pcieport serial vfio-pci [root@localhost dpdk-19.11]# ls /sys/bus/pci/drivers/vfio-pci/ 0000:05:00.0 bind module new_id remove_id uevent unbind [root@localhost dpdk-19.11]# ls /sys/bus/pci/drivers/vfio-pci/module/ coresize holders initstate parameters rhelversion srcversion uevent drivers initsize notes refcnt sections taint version [root@localhost dpdk-19.11]# ls /sys/bus/pci/drivers/vfio-pci/module/drivers/ pci:vfio-pci [root@localhost dpdk-19.11]# ls /sys/bus/pci/drivers/vfio-pci/ 0000:05:00.0 bind module new_id remove_id uevent unbind [root@localhost dpdk-19.11]# cd /sys/bus/pci/drivers/vfio-pci/ [root@localhost vfio-pci]#
NVMe的驅動解綁和VFIO的綁定 查看pci信息，找到nvme： $ lspci -nn 00:04.0 Non-Volatile memory controller [0108]: Intel Corporation QEMU NVM Express Controller [8086:5845] (rev 02) 記住兩個字段：00:04.0 和 8086:5845 00:04.0 全稱是0000:00:04.0， 對應[[[[<domain>]:]<bus>]:][<device>][.[<func>]] 8086:5845 對應[<vendor>]:[<device>] 從內核驅動解綁 echo "0000:00:04.0" >/sys/bus/pci/devices/0000:00:04.0/driver/unbind 綁定到VFIO echo 8086 5845 | tee /sys/bus/pci/drivers/vfio-pci/new_id 檢查： - lsblk 看不到nvme盤 - vfio下面多了一個設備文件：ls /dev/vfio/4 到此為止，已經完成了NVMe設備的解綁和綁定，接下來就可以開發用戶態NVMe驅動來驅動NVMe盤，我們這里使用qemu的用戶態driver來測試和學習。
# echo igb > /sys/bus/pci/devices/0000\:08\:00.0/driver_override # echo 0000:08:00.0 > /sys/bus/pci/drivers_probe # lspci -s 0000:08:00.0 -k 08:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) Subsystem: Intel Corporation Device 0000 Kernel driver in use: igb Kernel modules: igb
echo 1 > /sys/bus/pci/rescan
# ethtool -i enp8s16 | grep bus bus-info: 0000:08:10.0 # lspci -s 0000:08:10.0 -n 08:10.0 0200: 8086:10ca (rev 01) # modprobe vfio # modprobe vfio-pci # echo 0000:08:10.0 > /sys/bus/pci/devices/0000\:08\:10.0/driver/unbind # echo “8086 10ca” > /sys/bus/pci/drivers/vfio-pci/new_id # echo 0000:08:10.0 > /sys/bus/pci/drivers/vfio-pci/bind # lspci -s 0000:08:10.0 -k 08:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) Subsystem: Intel Corporation Device 0000 Kernel driver in use: vfio-pci Kernel modules: igbvf
[root@localhost vfio-pci]# lspci -n -v -s 0000\:06\:00.0 06:00.0 0200: 19e5:0200 (rev 45) Subsystem: 19e5:d139
[root@localhost vfio-pci]# ls /sys/bus/pci/devices/ 0000:00:00.0 0000:00:12.0 0000:05:00.0 0000:74:03.0 0000:7b:00.0 0000:7d:00.3 0000:81:00.0 0000:ba:00.0 0000:00:08.0 0000:02:00.0 0000:06:00.0 0000:74:04.0 0000:7c:00.0 0000:80:00.0 0000:84:00.0 0000:ba:01.0 0000:00:0c.0 0000:03:00.0 0000:07:00.0 0000:7a:00.0 0000:7d:00.0 0000:80:08.0 0000:b4:02.0 0000:ba:02.0 0000:00:10.0 0000:04:00.0 0000:08:00.0 0000:7a:01.0 0000:7d:00.1 0000:80:0c.0 0000:b4:03.0 0000:bb:00.0 0000:00:11.0 0000:04:01.0 0000:74:02.0 0000:7a:02.0 0000:7d:00.2 0000:80:10.0 0000:b4:04.0 0000:bc:00.0 [root@localhost vfio-pci]# ls /sys/bus/pci/devices/:06\:00.0/driver/ ls: cannot access /sys/bus/pci/devices/:06:00.0/driver/: No such file or directory [root@localhost vfio-pci]# echo 0000\:06\:00.0 > /sys/bus/pci/devices/0000\:06\:00.0/driver/unbind -bash: /sys/bus/pci/devices/0000:06:00.0/driver/unbind: No such file or directory [root@localhost vfio-pci]# echo -n 0000\:06\:00.0 > bind -bash: echo: write error: No such device [root@localhost vfio-pci]# pwd /sys/bus/pci/drivers/vfio-pci [root@localhost vfio-pci]# ls /dev/vfio/ 25 vfio [root@localhost vfio-pci]# echo -n 19e5:0200 > new_id -bash: echo: write error: Invalid argument [root@localhost vfio-pci]# echo -n "19e5:0200" > new_id -bash: echo: write error: Invalid argument [root@localhost vfio-pci]# echo -n 0000\:06\:00.0 > bind -bash: echo: write error: No such device [root@localhost vfio-pci]#
[root@localhost vfio-pci]# echo -n "19e5 0200" > new_id [root@localhost vfio-pci]# echo -n 0000\:06\:00.0 > bind -bash: echo: write error: No such device [root@localhost vfio-pci]# echo 0000:06:00.0 > bind -bash: echo: write error: No such device [root@localhost vfio-pci]# ls /dev/vfio/ 25 33 vfio [root@localhost vfio-pci]#
echo 0000:0b:00.0 >>/sys/bus/pci/devices/0000\:0b\:00.0/driver/unbind 注意：這條指令執行后/sys/bus/pci/devices/0000\:0b\:00.0/下的driver目錄會消失
=========================================
[root@localhost dpdk-19.11]# ls /dev/vfio/ 25 33 vfio [root@localhost dpdk-19.11]# ls /sys/bus/pci/devices/0000\:06\:00.0/driver 0000:05:00.0 0000:06:00.0 bind module new_id remove_id uevent unbind [root@localhost dpdk-19.11]# echo 0000\:06\:00.0 > /sys/bus/pci/devices/0000\:06\:00.0/driver/unbind [root@localhost dpdk-19.11]# ls /sys/bus/pci/devices/0000\:06\:00.0/driver ls: cannot access /sys/bus/pci/devices/0000:06:00.0/driver: No such file or directory [root@localhost dpdk-19.11]# ls /dev/vfio/ 25 vfio [root@localhost dpdk-19.11]# echo -n "19e5 0200" > new_id /sys/bus/pci/drivers/vfio-pci/new_id [root@localhost dpdk-19.11]# echo -n "19e5 0200" > /sys/bus/pci/drivers/vfio-pci/new_id [root@localhost dpdk-19.11]# ls /dev/vfio/ 25 33 vfio [root@localhost dpdk-19.11]# echo -n "0000:06:00.0" /sys/bus/pci/drivers/vfio-pci/bind 0000:06:00.0 /sys/bus/pci/drivers/vfio-pci/bind[root@localhost dpdk-19.11]# [root@localhost dpdk-19.11]# ls /sys/bus/pci/devices/0000\:06\:00.0/driver 0000:05:00.0 0000:06:00.0 bind module new_id remove_id uevent unbind [root@localhost dpdk-19.11]#
[root@localhost dpdk-19.11]# ./usertools/dpdk-devbind.py -s Network devices using DPDK-compatible driver ============================================ 0000:05:00.0 'Hi1822 Family (2*25GE) 0200' drv=vfio-pci unused=hinic,igb_uio 0000:06:00.0 'Hi1822 Family (2*25GE) 0200' drv=vfio-pci unused=hinic,igb_uio
[root@localhost dpdk-19.11]# ./usertools/dpdk-devbind.py -s Network devices using DPDK-compatible driver ============================================ 0000:05:00.0 'Hi1822 Family (2*25GE) 0200' drv=vfio-pci unused=hinic,igb_uio 0000:06:00.0 'Hi1822 Family (2*25GE) 0200' drv=vfio-pci unused=hinic,igb_uio Network devices using kernel driver =================================== 0000:7d:00.0 'HNS GE/10GE/25GE RDMA Network Controller a222' if=enp125s0f0 drv=hns3 unused=hns_roce_hw_v2,igb_uio,vfio-pci *Active* 0000:7d:00.1 'HNS GE/10GE/25GE Network Controller a221' if=enp125s0f1 drv=hns3 unused=igb_uio,vfio-pci 0000:7d:00.2 'HNS GE/10GE/25GE RDMA Network Controller a222' if=enp125s0f2 drv=hns3 unused=hns_roce_hw_v2,igb_uio,vfio-pci 0000:7d:00.3 'HNS GE/10GE/25GE Network Controller a221' if=enp125s0f3 drv=hns3 unused=igb_uio,vfio-pci No 'Baseband' devices detected ============================== No 'Crypto' devices detected ============================ No 'Eventdev' devices detected ============================== No 'Mempool' devices detected ============================= No 'Compress' devices detected ============================== No 'Misc (rawdev)' devices detected ===================================
[root@localhost dpdk-19.11]# ls /sys/class/net/ ----沒dpdk的網卡 enp125s0f0 enp125s0f1 enp125s0f2 enp125s0f3 lo
[root@localhost dpdk-19.11]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp125s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether b0:08:75:5f:b7:d9 brd ff:ff:ff:ff:ff:ff inet 10.10.16.229/24 brd 10.10.16.255 scope global noprefixroute enp125s0f0 valid_lft forever preferred_lft forever inet6 fe80::f9ec:a960:7518:b745/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: enp125s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000 link/ether b0:08:75:5f:b7:da brd ff:ff:ff:ff:ff:ff 4: enp125s0f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000 link/ether b0:08:75:5f:b7:db brd ff:ff:ff:ff:ff:ff 5: enp125s0f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether b0:08:75:5f:b7:dc brd ff:ff:ff:ff:ff:ff [root@localhost dpdk-19.11]# lspci | grep ether [root@localhost dpdk-19.11]# lspci | grep -i ether 05:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (2*25GE) (rev 45) 06:00.0 Ethernet controller: Huawei Technologies Co., Ltd. Hi1822 Family (2*25GE) (rev 45) 7d:00.0 Ethernet controller: Huawei Technologies Co., Ltd. HNS GE/10GE/25GE RDMA Network Controller (rev 21) 7d:00.1 Ethernet controller: Huawei Technologies Co., Ltd. HNS GE/10GE/25GE Network Controller (rev 21) 7d:00.2 Ethernet controller: Huawei Technologies Co., Ltd. HNS GE/10GE/25GE RDMA Network Controller (rev 21) 7d:00.3 Ethernet controller: Huawei Technologies Co., Ltd. HNS GE/10GE/25GE Network Controller (rev 21) [root@localhost dpdk-19.11]#