nvidia tk1使用記錄--基本環境搭建

前言

　　項目最開始是在X86+Nvidia（ubuntu+opencv+cuda）平台上實現，達到了期望性能，最近考慮將其移植到嵌入式平台，特別是最近nvidia出了tegra X1，基於和我們使用的gtx960相同的架構的gpu且是64位 8核心cpu，於是開始了嘗試移植過程。當前TX1才出來，購買不是那么方便，於是拿它的前一代TK1進行一個最基本的評估。下面記錄了評估過程的一些操作。

tk1系統更新

　　拿到的tk1時，板子自帶的系統版本(非內核版本)是Jetson TK1 R19.3 - June 2014，非常老了，有必要對其進行升級。當前(2016/04/11)，最新的系統版本為Linux For Tegra R21.4，它的相關介紹及下載包地址在這里。系統更新有兩種方法，不管是哪種方式，都需要用數據線（一端接tk1的micro usb口，一端接PC機的usb口）連接開發板與主機，然后上電的時候讓設備進入recovery模式，最后在主機上啟動升級。

注意：tk1開箱上電啟動時，默認是在非圖形界面下的，ls一下會看到目錄NVIDIA-INSTALLER，cd進去，執行sudo ./installer來進行圖形化安裝，然后sudo reboot重啟即可進入圖形界面了。默認的用戶名是ubuntu，密碼ubuntu。

自動更新

　　JetPack for L4T傻瓜式自動更新，提供圖形界操作。它是由官方提供的升級包，可以說它是所有需要安裝包的集合,包括最新系統,cuda,opencv,cross-develop-environment,samples等等,非常方便,缺點就是主機系統只能是ubuntu12.04或者ubuntu14.04，安裝后的軟件包版本可能並非期望的,比如opencv是2.4。關於這種更新方式的詳細介紹請參考這里，我沒有采用這種升級方式，所以不再過多描述。

手動更新

　　這種升級方式實際上和上面的升級方式一樣，只不過它僅僅是上面升級里面的一個步驟而已，JetPack for L4T不僅僅更新系統，還會同時更新其他軟件包，如cuda、opencv等等。具體升級的步驟如下：

1. 下載內核及驅動包http://developer.download.nvidia.com/embedded/L4T/r21_Release_v4.0/Tegra124_Linux_R21.4.0_armhf.tbz2；
2. 下載文件系統包
   http://developer.download.nvidia.com/embedded/L4T/r21_Release_v4.0/Tegra_Linux_Sample-Root-Filesystem_R21.4.0_armhf.tbz2;
3. 構建升級包(包括了boot、kernel、fs)，執行sudo tar --numeric-owner -jxpf Tegra124_Linux_R21.4.0_armhf.tbz2解壓內核及驅動包，然后cd linux_for_tegra/rootfs下,再次解壓文件系統包sudo tar --numeric-owner -jxpf ../../Tegra_Linux_Sample-Root-Filesystem_R21.4.0_armhf.tbz2，然后cd ../，即返回到linux_for_tegra，執行sudo ./apply_binaries.sh構建升級包。注意，執行命令前,請先確保系統還有20G空間,如果沒有,那么將14GiB改下點吧,雖然這樣會導致板子上flash沒充分利用,但至少能夠更新系統;
4. 開始升級，USB端插到ubuntu PC上，小頭插到TK1上，也就是耳機口旁邊的口。TK1上有三個按鈕，最左邊的是電源，中間的是reset，最右邊的是force recoverry，按住最右邊的force recoverry 按鈕別松手，再按一下中間的reset鍵，指示燈會閃一下，板子就重啟進入了刷機模式。然后在ubuntu PC上，當前目錄，即linux_for_tegra目錄下，執行終端命令：sudo ./flash.sh -S 14GiB jetson-tk1mmcblk0p1，這個時候ubuntu上會彈出一個設備，就是tk1了，然后就會進入刷機過程，若干小時后，最后成功了的話，會出現success字樣。這時重啟TK1開發板，就發現這板子已經成功被初始化了。進入ubuntu桌面環境，默認的用戶名是ubuntu，密碼ubuntu。

tk1 cuda安裝

　　cuda的安裝可以參考elinux的指導，這部分也沒有需要特別注意的，確保cuda安裝在opencv之前以及所有操作都是在tk1系統里進行的就可以了。下面還是把關鍵步驟給摘抄下來吧(略加修改)！

Download the .deb file for the CUDA Toolkit for L4T from http://developer.download.nvidia.com/embedded/L4T/r21_Release_v3.0/cuda-repo-l4t-r21.3-6-5-prod_6.5-42_armhf.deb

On the device, install the .deb file and the CUDA Toolkit. eg:

cd ~/Downloads
# Install the CUDA repo metadata that you downloaded manually for L4T
sudo dpkg -i cuda-repo-l4t-r21.3-6-5-prod_6.5-42_armhf.deb
# Download & install the actual CUDA Toolkit including the OpenGL toolkit from NVIDIA. (It only downloads around 15MB)
sudo apt-get update
# Install "cuda-toolkit-6-0" if you downloaded CUDA 6.0, or "cuda-toolkit-6-5" if you downloaded CUDA 6.5, etc.
sudo apt-get install cuda-toolkit-6-5
# Add yourself to the "video" group to allow access to the GPU
sudo usermod -a -G video $USER

Add the 32-bit CUDA paths to your .bashrc login script, and start using it in your current console:

echo "# Add CUDA bin & library paths:" >> ~/.bashrc
echo "export PATH=/usr/local/cuda/bin:$PATH" >> ~/.bashrc
echo "export LD_LIBRARY_PATH=/usr/local/cuda/lib:$LD_LIBRARY_PATH" >> ~/.bashrc
source ~/.bashrc

Verify that the CUDA Toolkit is installed on your device:

nvcc -V

tk1 opencv安裝

　　opencv的安裝可以參考elinux的指導，我這里還是對每一個步驟詳細的描述下，因為這里會遇到些上面鏈接里沒有指出的問題，另外要注意，下面所有操作都是在tk1系統里進行的。opencv的安裝有兩種方式：

方法一，直接采用nvidia提供的安裝包。優點：簡單，且同時增加了cpu neon的優化；缺點：版本不夠新2.4，基於opencv-3.0.0實現的代碼可能需要修改代碼。操作方法如下：

1. 從http://developer.download.nvidia.com/embedded/OpenCV/L4T_21.2/libopencv4tegra-repo_l4t-r21_2.4.10.1_armhf.deb下載opencv deb包
2. 執行sudo dpkg -i libopencv4tegra-repo_l4t-r21_2.4.10.1_armhf.deb安裝
3. sudo apt-get update
4. sudo apt-get install libopencv4tegra libopencv4tegra-dev

方法二，從源碼編譯。優點：能夠使用最新的opencv版本代碼；缺點：復雜。下面的操作步驟中包含了對OpenGL的使能，需要注意的是如果不添加-DWITH_QT=ON，那么-DWITH_OPENGL=ON將沒有效果，因此如果你想開啟OPENGL使能，還得先參考后面怎么安裝qt環境。參考 ...

If you haven't added the "universal" repository to Ubuntu, then do it now:

sudo add-apt-repository universe
sudo apt-get update

Now you need to install many libraries:

# Some general development libraries
sudo apt-get -y install build-essential make cmake cmake-curses-gui g++
# libav video input/output development libraries
sudo apt-get -y install libavformat-dev libavutil-dev libswscale-dev
# Video4Linux camera development libraries
sudo apt-get -y install libv4l-dev
# Eigen3 math development libraries
sudo apt-get -y install libeigen3-dev
# OpenGL development libraries (to allow creating graphical windows)
sudo apt-get -y install libglew1.6-dev
# GTK development libraries (to allow creating graphical windows)
sudo apt-get -y install libgtk2.0-dev

Download the source code of OpenCV for Linux onto the device. eg: Open a web-browser to "www.opencv.org" & click on "OpenCV for Linux/Mac", or from the command-line you can run this on the device:

wget https://github.com/Itseez/opencv/archive/3.1.0.zip

Unzip the OpenCV source code:

cd Downloads
unzip 3.1.0.zip
mv opencv-3.1.0 ~

Configure OpenCV using CMake:

cd ~/opencv-3.1.0
mkdir build
cd build
cmake -DWITH_CUDA=ON -DCUDA_ARCH_BIN="3.2" -DCUDA_ARCH_PTX="" -DBUILD_TESTS=OFF -DWITH_OPENGL=ON -DWITH_QT=ON -DBUILD_PERF_TESTS=OFF ..

sudo make -j4 install

Finally, make sure your system searches the "/usr/local/lib" folder for libraries:

echo "# Use OpenCV and other custom-built libraries." >> ~/.bashrc
echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/" >> ~/.bashrc
source ~/.bashrc

編譯FAQ

錯誤1

 error: #error Please include the appropriate gl headers before including cuda_gl_interop.h
 #error Please include the appropriate gl headers before including cuda_gl_interop.h

錯誤2

/home/ubuntu/build/opencv/modules/highgui/src/window_QT.cpp:3150:12: error: 'GL_PERSPECTIVE_CORRECTION_HINT' was not declared in this scope
     glHint(GL_PERSPECTIVE_CORRECTION_HINT, GL_NICEST);

可以嘗試下面的解決辦法:

there is a workaround, if you delete folllowing lines in /usr/local/cuda/include/cuda_gl_interop.h

#ifndef GL_VERSION
#error Please include the appropriate gl headers before including cuda_gl_interop.h
#endif
#else

then gl headers is anyway included and opencv compiles.. with some rain dancing for me)
in my case, i had to add

#define GL_PERSPECTIVE_CORRECTION_HINT 0x0C50

to modules/highgui/src/window_QT.cpp and turn off TBB support and cvv module (i was building with opencv_contrib)

ref: https://github.com/Itseez/opencv/issues/5205

ref: http://code.opencv.org/issues/3411

tk1 qt安裝方式

注意：下面所有操作都是在tk1系統里進行的

方法一，直接采用倉庫里已經提供到安裝包。優點：簡單；缺點：一般不是最新的版本，而且是基於OpenES的，對於之前代碼是基OpenGL編寫的需要做到OpenES的移植。具體安裝步驟：

sudo apt-get install build-essential qt5-default qtcreator -y

which will load Qt Creator. Once Qt Creator is installed, there is still a little setup needed.

Open Qt Creator, and go to:

    Tools->Options->Build & Run->Compilers

Click the ‘Add’ button and select ‘GCC’. In the ‘Compiler path:’ text box, place the path to the gcc compiler. On a standard installation the path is: /usr/bin/gcc.

You should also check the ‘Kits’ setting, available from the ‘Kits’ tab. Selecting the Desktop (default) kit, you should see Compiler: GCC. The name GCC should match the Compiler named in the previous tab.

The installation was shown on Linux for Tegra (L4T) release 21.3. Note that no special compiler flags were set. Also, CUDA was not configured.

ref: http://jetsonhacks.com/2015/04/22/install-qt-creator-on-nvidia-jetson-tk1/

方法二，從源代碼編譯。優點：能夠基於任意版本(比如最新的qt)編譯，且可以基於OpenGL而不基於OpenES；缺點：復雜，編譯時間長。詳細步驟參考下面兩個鏈接:

http://blog.csdn.net/deyili/article/details/51012106

blog.csdn.net/deyili/article/details/49868987

需要注意的是,如果遇到如下編譯錯誤:

zlib.h:86:5: error: 'z_const' does not name a type

，那么嘗試在configue的時候去掉-zlib

如果在configure的時候xcb不過，通過官方解決方法(安裝xcb包http://doc.qt.io/qt-5/linux-requirements.html), 如果還解決不了,那么嘗試直接修改configure，如下:

    #if [ "$CFG_XKBCOMMON" != no ] && compileTest qpa/xcb "xcb" $QMAKE_CFLAGS_XCB $QMAKE_LIBS_XCB; then
    if [ "$CFG_XKBCOMMON" != no ]; then

編譯Qt Creator

　　同樣給出參考鏈接，里面已經寫的很詳細了。

關於OpenGL和OpenES的Qt

This board is powerful enough to build everything on its own without any cross-compilation. Configuring and building Qt is no different than in any desktop Linux environment. One option that needs special consideration however is -opengl es2 because Qt can be built either in a GLX + OpenGL or EGL + OpenGL ES configuration.

For example, the following configures Qt to use GLX and OpenGL:

configure -release -nomake examples -nomake tests

while adding -opengl es2 requests the usage of EGL and OpenGL ES:

configure -release -opengl es2 -nomake examples -nomake tests

If you are planning to run applications relying on modern, non-ES OpenGL features, or use CUDA, then go for the first. If you however have some existing code from the mobile or embedded world relying on EGL or OpenGL ES then it may be useful to go for #2.

The default platform plugin will be xcb, so running Qt apps without specifying the platform plugin will work just fine. This is the exact same plugin that is used on any ordinary X11-based Linux desktop system.

http://blog.qt.io/blog/2015/03/03/qt-weekly-28-qt-and-cuda-on-the-jetson-tk1/

另外一個問題(在iss上出現過的視頻斷層問題tearing,應該可以通過該方法解決)
Vsync gotchas

Once the build is done, you will most likely run some OpenGL-based Qt apps. And then comes the first surprise: applications are not synchronized to the vertical refresh rate of the screen.

When running for instance the example from qtbase/examples/opengl/qopenglwindow, we expect a nice and smooth 60 FPS animation with the rendering thread throttled appropriately. This unfortunately isn’t the case. Unless the application is fullscreen. Therefore many apps will want to replace calls like show() or showMaximized() with showFullScreen(). This way the thread is throttled as expected.

A further surprise may come in QWidget-based applications when opening a popup or a dialog. Unfortunately this also disables synchronization, even though the main window still covers the entire screen. In general we can conclude that the standard embedded recommendation of sticking to a single fullscreen window is very valid for this board too, even when using xcb, although for completely different reasons.

http://blog.qt.io/blog/2015/03/03/qt-weekly-28-qt-and-cuda-on-the-jetson-tk1/

cuda於Qt的結合

blog.qt.io/blog/2015/03/03/qt-weekly-28-qt-and-cuda-on-the-jetson-tk1/

FAQ

opencv error: the function/feature is not implemented

http://answers.opencv.org/question/84639/opencv-error-the-functionfeature-is-not-implemented/

/usr/include/qt4/QtOpenGL/qgl.h:85:17: error: 'GLdouble' has a previous declaration as 'typedef GLfloat GLdouble'

http://answers.ros.org/question/179989/rgbdslam_v2-error-with-make/

總結

　　評估過程可以說非常的順利，一周內將整個項目全部移植了過來，而且基本沒有代碼改動^_!!!!經過測試，我們的拼接算法單路能夠帶到65幀，2路能夠代碼35幀，3路沒測試，因為tk1只有2G內存，系統本身用了將近700MB，而我們程序2路的時候用了1000多M，只剩50多M了。總的來說，對於移植到TX1上，心里更加有底了…………

參考

1. Boot Jetson TK1 in recovery mode
2. Jetson TK1
3. Linux For Tegra R21.4
4. Building_Qt_5_from_Git

完！
2016年5月