xilinx zcu106 四攝像頭接mipi案例解析

官方案例地址：https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/1010303572/Zynq+UltraScale+MPSoC+VCU+TRD+2020.2+-+Quad+Sensor+MIPI+CSI+Video+Capture+and+HDMI+Display

 

 

一、Quad Sensor MIPI CSI Video Capture and HDMI Display案例介紹

總體框架如下圖所示(不包含HDMI)：

　　

 

 

 

圖中ISP部分：

 

 

 

 

四個AR0231 sensor進來經過AMZ9286將思路圖像匯總成一路CSI-2接到pl端的mipi接口，通過一個“AXI Stream Switch”將四個虛擬通道分別送到四路ISP處理，最后送到buffer中。

四個isp分別為：

Sensor Demasaic：去馬賽克，也就是通過插值算法把RAW圖像轉換為RGB圖像

GamamaLUT：伽馬矯正

Video Processing Subsystem：第一個做rgb顏色校正，第二個將rgb轉yuv。

 

 

二、vivado和petalinux工程搭建

構建vivado工程和petalinux工程參考：https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/1010368517/Zynq+UltraScale+MPSoC+VCU+TRD+2020.2+-+Run+and+Build+Flow    的3 Build Flow章節

2.1下載官方案例

下載地址：https://www.xilinx.com/cgi-bin/docs/ctdoc?cid=bigfile;d=rdf0428-zcu106-vcu-trd-2020-2.zip　　　　　　　　　　

2.2、搭建vivado工程

1、將下載的案例rdf0428-zcu106-vcu-trd-2020.2復制到虛擬機

2、設置環境變量

　　export TRD_HOME=</path/to/downloaded/zipfile>/rdf0428-zcu106-vcu-trd-2020-2

3、source vivado命令：

　　source /opt/xilinx/vitis_2020/Vivado/2020.2/settings64.sh

4、進入rdf0428-zcu106-vcu-trd-2020.2/pl目錄(不能用secureCRT，要用虛擬機中的終端，否則無法自動打開vivado)

　　cd </path/to/downloaded/zipfile>/rdf0428-zcu106-vcu-trd-2020.2/pl

5、輸入命令導入tcl文件：

　　vivado -source ./designs/zcu106_Quad_Sensor/project.tcl

注：如果遇到project.tcl無法執行，則執行chmod 777 project.tcl

vivado工程構建完成后，會生成如下所示的工程：

新建的工程默認保存在</path/to/downloaded/zipfile>/rdf0428-zcu106-vcu-trd-2020-2/pl/build 目錄中

2.3 創建petalinux 工程

1、到處petalinx環境變量：source /opt/xilinx/petalinux/settings.sh

2、進入到rdf0428-zcu106-vcu-trd-2020.1/apu/vcu_petalinux_bsp目錄

3、用rdf0428-zcu106-vcu-trd-2020.1/apu/vcu_petalinux_bsp/xilinx-vcu-zcu106-v2020.1-final.bsp建立petalinux工程:

　　　petalinux-create -t project -s ./

4、導入rdf0428-zcu106-vcu-trd-2020.1\pl\prebuild\zcu106_Quad_Sensor\zcu106_Quad_Sensor_wrapper.xsa 文件

　　petalinux-config --get-hw-description ../../pl/prebuild/zcu106_Quad_Sensor/

3、將vcu_quad_sensor.dtsi軟件\\軟連接成system-user.dtsi

　　ln -s xilinx-vcu-zcu106-v2020.1-final/project-spec/meta-user/recipes-bsp/device-tree/files/vcu_quad_sensor.dtsi system-user.dtsi

4、petalinux-bulid

5、用qemu虛擬機運行：petalinux-boot --qemu --prebuilt 3

退出qemu：Ctrl+A，然后放掉再按X

 

編譯過程中如果出現qemu無法獲取失敗，參考https://forums.xilinx.com/t5/Embedded-Linux/qemu-fails-in-petalinux-buld-due-to-Fetcher-failure/m-p/1168900

三、設備樹解析：

Quad Sensor的設備樹是petalinux_prj/project-spec/meta-user/recipes-bsp/device-tree/files/vcu_quad_sensor.dtsi。vcu_quad_sensor.dtsi主要是配置pl端的各個模塊，包過mipi，buffer、isp模塊，hdmi等，vcu_quad_sensor.dtsi中還包含了一個配置AR0231 sensor和AMZ9286的設備樹

“avnet-ar0231-multicam-fmc.dtsi”，如下所示：

設備樹中各個模塊的連接就是按上面總框架圖的連接。

四、gstreamer命令

分析官方案例https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/1010303572/Zynq+UltraScale+MPSoC+VCU+TRD+2020.2+-+Quad+Sensor+MIPI+CSI+Video+Capture+and+HDMI+Display#4-Appendix-B---CSI-2-Rx/HDMI-Tx-Link-up-and-GStreamer-Commands

的第四小節4 Appendix B - CSI-2 Rx/HDMI-Tx Link-up and GStreamer Commands的gstreamer命令。

1、配置pl：sh /media/card/quad_sensor_media_graph_setting.sh

2、用gst-launch-1.0命令顯示raw數據

$ gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! queue ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=34 render-rectangle=<0,0,1920,1080>" v4l2src device=/dev/video1 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! queue ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=35 render-rectangle=<1920,0,1920,1080>" v4l2src device=/dev/video2 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! queue ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=36 render-rectangle=<0,1080,1920,1080>" v4l2src device=/dev/video3 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! queue ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=37 render-rectangle=<1920,1080,1920,1080>" -v

這條命令顯示四個攝像頭的圖像，四個攝像頭的取流節點分別為/dev/video0、/dev/video1、/dev/video2、/dev/video3。可以把這條命名分為四行，每行對應一個攝像頭和一個顯示畫面，如下所示：

 

$ gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! queue ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=34 render-rectangle=<0,0,1920,1080>" \
                 v4l2src device=/dev/video1 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! queue ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=35 render-rectangle=<1920,0,1920,1080>"  \
                 v4l2src device=/dev/video2 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! queue ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=36 render-rectangle=<0,1080,1920,1080>"  \
                 v4l2src device=/dev/video3 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! queue ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=37 render-rectangle=<1920,1080,1920,1080>" -v

 fpsdisplaysink是個渲染插件可以理解為送顯，a0270000.v_mix是個video mixer，這個是xilinx的一個視頻混合器ip，這條視頻管道為v4l2src -->queue-->fpsdisplaysink

3、capture → encode → decode → display

下面是視頻捕獲-->vcu編碼-->vcu解碼-->hdmi顯示的管道

$ gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=3 low-latency=0 ! queue max-size-bytes=0 ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=34 render-rectangle=<0,0,1920,1080>" v4l2src device=/dev/video1 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=3 low-latency=0 ! queue max-size-bytes=0 ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=35 render-rectangle=<1920,0,1920,1080>" v4l2src device=/dev/video2 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=3 low-latency=0 ! queue max-size-bytes=0 ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=36 render-rectangle=<0,1080,1920,1080>" v4l2src device=/dev/video3 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=3 low-latency=0 ! queue max-size-bytes=0 ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=37 render-rectangle=<1920,1080,1920,1080>" -v

同樣把這個長命令分為4行，每行代表一個顯示管道

$ gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=3 low-latency=0 ! queue max-size-bytes=0 ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=34 render-rectangle=<0,0,1920,1080>" 
                 v4l2src device=/dev/video1 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=3 low-latency=0 ! queue max-size-bytes=0 ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=35 render-rectangle=<1920,0,1920,1080>" 
                 v4l2src device=/dev/video2 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=3 low-latency=0 ! queue max-size-bytes=0 ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=36 render-rectangle=<0,1080,1920,1080>" 
                 v4l2src device=/dev/video3 io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! omxh265enc qp-mode=auto gop-mode=basic gop-length=60 b-frames=0 target-bitrate=15000 num-slices=8 control-rate=constant prefetch-buffer=true low-bandwidth=false filler-data=true cpb-size=1000 initial-delay=500 ! video/x-h265, profile=main, alignment=au ! queue ! omxh265dec internal-entropy-buffers=3 low-latency=0 ! queue max-size-bytes=0 ! fpsdisplaysink text-overlay=false video-sink="kmssink bus-id="a0270000.v_mix" show-preroll-frame=false plane-id=37 render-rectangle=<1920,1080,1920,1080>" -

omxh265enc 和omxh265dec 分別是xilinx的vcu編碼和解碼插件，詳情見<<ug1449 Multimedia User Guide>>的GStreamer Multimedia Framework章節。
這條視頻管道為：v4l2src-->omxh265enc-->queue-->omxh265dec-->fpsdisplaysink