最便捷的caffe編譯方法 ---- cmake+anaconda虛擬環境

caffe編譯是一個繁瑣個過程, 或多或少會遇到各種各樣的問題, 如我caffe編譯錯誤記錄記錄了遇到過得一部分問題. 原因是因為每個依賴如openblas, protobuf都需要編譯特定版本, 每台電腦裝的依賴庫各不相同, 環境變量也有沖突的情況(如system和anaconda). 於是我一直想找一種簡單方便的caffe編譯方法, 可以避免上述的問題. 也就有了今天的內容, 使用anaconda虛擬環境編譯caffe.

---

一些說明:

1. anaconda虛擬環境可以把環境和系統隔離, 相當於濃縮版/usr/lib+/usr/include, 在環境中安裝包使用conda install {包名稱}=版本號命令, 可以很方便的修改環境依賴;
2. 測試系統是Manjaro Linux, 但是使用anaconda+cmake, 應該和系統沒有關系. 即使用其他linux系統或者windows也可以;
3. anaconda環境名稱為caffe_27．　python 3在編譯完成caffe后, pychaffe編譯出錯. 於是換成了python2.7. python3我會后續補充;
4. 我的caffe是添加了多標簽的部分, 即label可以有多個, 可以直接用我修改好的caffe開始編譯;
5. cuda任然需要本地安裝, 我的是Manjaro Linux, 使用pacman -S cuda安裝.
6. 項目整體只在本地測試過, 如果有問題還請指出, 我會進一步完善.
7. NCCL, GPU默認開啟，如果需要關閉可以更改 CmakeList.txt相關位置．

>>>>> GitHub項目鏈接

---

1. 編譯過程說明:

(1) 生成anaconda虛擬環境

我編譯完成后, 把用到的conda虛擬環境輸出成了.yml文件. 重新編譯時, 可以使用conda env create -f caffe_36.yml生成和我一模一樣的虛擬環境,而不需要一個一個包重新安裝.

(2) 激活虛擬環境

環境名我定的名字是caffe_ 27, 代表python版本是2.7(python3.6環境編譯可以通過, 但是pycaffe編輯通過, 於是后來換成了python 2.7, ), 用於編譯caffe環境. 如果使用第一步命令生成環境, 那么可以運行一下source命令激活.
source activate caffe_27

(3) cmake

不使用make, 而是用camke進行編譯, 一方面可以跨平台, 另一方面cmake更加通俗易懂.
需要進行如下一些修改:

<1> 修改虛擬環境路徑:

<2> 如果cuda有問題, 手動添加cuda路徑:

執行如下命令:

mkdir build 
cd build 
cmake ..

camke過程如圖:

-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Warning (dev) at cmake/Misc.cmake:32 (set):
  implicitly converting 'BOOLEAN' to 'STRING' type.
Call Stack (most recent call first):
  CMakeLists.txt:29 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found CUDA: /opt/cuda (found version "10.1") 
-- Boost version: 1.60.0
-- Found the following Boost libraries:
--   system
--   thread
--   filesystem
--   chrono
--   date_time
--   atomic
-- Found GFlags: /home/geoffrey/.conda/envs/caffe_27/include  
-- Found gflags  (include: /home/geoffrey/.conda/envs/caffe_27/include, library: /home/geoffrey/.conda/envs/caffe_27/lib/libgflags.so)
-- Found Glog: /home/geoffrey/.conda/envs/caffe_27/include  
-- Found glog    (include: /home/geoffrey/.conda/envs/caffe_27/include, library: /home/geoffrey/.conda/envs/caffe_27/lib/libglog.so)
-- Found Protobuf: /home/geoffrey/.conda/envs/caffe_27/lib/libprotobuf-lite.so;/home/geoffrey/.conda/envs/caffe_27/lib/libprotoc.so;/home/geoffrey/.conda/envs/caffe_27/lib/libprotobuf.so (found version "2.5.0") 
-- Found PROTOBUF Compiler: /home/geoffrey/.conda/envs/caffe_27/bin/protoc
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- HDF5: Using hdf5 compiler wrapper to determine CXX configuration
-- Found HDF5: /home/geoffrey/.conda/envs/caffe_27/lib/libhdf5_cpp.so;/home/geoffrey/.conda/envs/caffe_27/lib/libhdf5.so;/usr/lib/librt.so;/usr/lib/libpthread.so;/home/geoffrey/.conda/envs/caffe_27/lib/libz.so;/usr/lib/libdl.so;/usr/lib/libm.so (found version "1.10.2") found components:  HL 
-- Found LMDB: /home/geoffrey/.conda/envs/caffe_27/include  
-- Found lmdb    (include: /home/geoffrey/.conda/envs/caffe_27/include, library: /home/geoffrey/.conda/envs/caffe_27/lib/liblmdb.so)
-- Found LevelDB: /home/geoffrey/.conda/envs/caffe_27/include  
-- Found LevelDB (include: /home/geoffrey/.conda/envs/caffe_27/include, library: /home/geoffrey/.conda/envs/caffe_27/lib/libleveldb.so)
-- Found Snappy: /home/geoffrey/.conda/envs/caffe_27/include  
-- Found Snappy  (include: /home/geoffrey/.conda/envs/caffe_27/include, library: /home/geoffrey/.conda/envs/caffe_27/lib/libsnappy.so)
-- CUDA detected: 10.1
-- Found cuDNN: ver. 7.6.0 found (include: /usr/include, library: /usr/lib/libcudnn.so)
-- Added CUDA NVCC flags for: sm_61
-- Found NCCL: /home/geoffrey/.conda/envs/caffe_27/include  
-- Found NCCL    (include: /home/geoffrey/.conda/envs/caffe_27/include, library: /home/geoffrey/.conda/envs/caffe_27/lib/libnccl.so)
-- OpenCV found (/home/geoffrey/.conda/envs/caffe_27/share/OpenCV)
-- Found Atlas: /home/geoffrey/.conda/envs/caffe_27/include  
-- Found Atlas (include: /home/geoffrey/.conda/envs/caffe_27/include library: /home/geoffrey/.conda/envs/caffe_27/lib/libblas.so lapack: /home/geoffrey/.conda/envs/caffe_27/lib/liblapack.so
-- Found PythonInterp: /home/geoffrey/.conda/envs/caffe_27/bin/python2.7 (found suitable version "2.7.16", minimum required is "2.7") 
-- Found PythonLibs: /home/geoffrey/.conda/envs/caffe_27/lib/libpython2.7.so (found suitable version "2.7.16", minimum required is "2.7") 
-- Found NumPy: /home/geoffrey/.conda/envs/caffe_27/lib/python2.7/site-packages/numpy/core/include (found suitable version "1.8.2", minimum required is "1.7.1") 
-- NumPy ver. 1.8.2 found (include: /home/geoffrey/.conda/envs/caffe_27/lib/python2.7/site-packages/numpy/core/include)
-- Boost version: 1.60.0
-- Found the following Boost libraries:
--   python
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
-- Found Git: /usr/bin/git (found version "2.22.0") 
-- 
-- ******************* Caffe Configuration Summary *******************
-- General:
--   Version           :   1.0.0
--   Git               :   unknown
--   System            :   Linux
--   C++ compiler      :   /usr/bin/c++
--   Release CXX flags :   -O3 -DNDEBUG -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
--   Debug CXX flags   :   -g -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
--   Build type        :   Release
-- 
--   BUILD_SHARED_LIBS :   ON
--   BUILD_python      :   ON
--   BUILD_matlab      :   OFF
--   BUILD_docs        :   ON
--   CPU_ONLY          :   OFF
--   USE_OPENCV        :   ON
--   USE_LEVELDB       :   ON
--   USE_LMDB          :   ON
--   USE_NCCL          :   ON
--   ALLOW_LMDB_NOLOCK :   OFF
--   USE_HDF5          :   ON
-- 
-- Dependencies:
--   BLAS              :   Yes (Atlas)
--   Boost             :   Yes (ver. 1.60)
--   glog              :   Yes
--   gflags            :   Yes
--   protobuf          :   Yes (ver. 2.5.0)
--   lmdb              :   Yes (ver. 0.9.23)
--   LevelDB           :   Yes (ver. 1.19)
--   Snappy            :   Yes (ver. ..)
--   OpenCV            :   Yes (ver. 3.4.2)
--   CUDA              :   Yes (ver. 10.1)
-- 
-- NVIDIA CUDA:
--   Target GPU(s)     :   Auto
--   GPU arch(s)       :   sm_61
--   cuDNN             :   Yes (ver. 7.6.0)
-- 
-- Python:
--   Interpreter       :   /home/geoffrey/.conda/envs/caffe_27/bin/python2.7 (ver. 2.7.16)
--   Libraries         :   /home/geoffrey/.conda/envs/caffe_27/lib/libpython2.7.so (ver 2.7.16)
--   NumPy             :   /home/geoffrey/.conda/envs/caffe_27/lib/python2.7/site-packages/numpy/core/include (ver 1.8.2)
-- 
-- Documentaion:
--   Doxygen           :   No
--   config_file       :   
-- 
-- Install:
--   Install path      :   /home/geoffrey/caffe/build/install
-- 
Caffe_INCLUDES: PUBLIC;/home/geoffrey/.conda/envs/caffe_27/include;PUBLIC;/home/geoffrey/.conda/envs/caffe_27/include;PUBLIC;/home/geoffrey/.conda/envs/caffe_27/include;PUBLIC;/home/geoffrey/.conda/envs/caffe_27/include;PUBLIC;/home/geoffrey/.conda/envs/caffe_27/include;PUBLIC;/home/geoffrey/.conda/envs/caffe_27/include;PUBLIC;/home/geoffrey/.conda/envs/caffe_27/include;PRIVATE;/home/geoffrey/.conda/envs/caffe_27/include;PUBLIC;/opt/cuda/include;PUBLIC;/usr/include;PUBLIC;/home/geoffrey/.conda/envs/caffe_27/include;/home/geoffrey/.conda/envs/caffe_27/include/opencv;PUBLIC;/home/geoffrey/.conda/envs/caffe_27/include;/home/geoffrey/.conda/envs/caffe_27/include;PRIVATE;/home/geoffrey/.conda/envs/caffe_27/include/python2.7;/home/geoffrey/.conda/envs/caffe_27/lib/python2.7/site-packages/numpy/core/include;PUBLIC;/home/geoffrey/.conda/envs/caffe_27/include
Caffe_LINKER_LIBS: PUBLIC;/home/geoffrey/.conda/envs/caffe_27/lib/libboost_system.so;/home/geoffrey/.conda/envs/caffe_27/lib/libboost_thread.so;-lpthread;/home/geoffrey/.conda/envs/caffe_27/lib/libboost_filesystem.so;/home/geoffrey/.conda/envs/caffe_27/lib/libboost_chrono.so;/home/geoffrey/.conda/envs/caffe_27/lib/libboost_date_time.so;/home/geoffrey/.conda/envs/caffe_27/lib/libboost_atomic.so;PRIVATE;-lpthread;PUBLIC;/home/geoffrey/.conda/envs/caffe_27/lib/libglog.so;PUBLIC;/home/geoffrey/.conda/envs/caffe_27/lib/libgflags.so;PUBLIC;/home/geoffrey/.conda/envs/caffe_27/lib/libprotobuf-lite.so;/home/geoffrey/.conda/envs/caffe_27/lib/libprotoc.so;/home/geoffrey/.conda/envs/caffe_27/lib/libprotobuf.so;PUBLIC;/home/geoffrey/.conda/envs/caffe_27/lib/libhdf5_cpp.so;/home/geoffrey/.conda/envs/caffe_27/lib/libhdf5.so;/usr/lib/librt.so;/usr/lib/libpthread.so;/home/geoffrey/.conda/envs/caffe_27/lib/libz.so;/usr/lib/libdl.so;/usr/lib/libm.so;/home/geoffrey/.conda/envs/caffe_27/lib/libhdf5_hl_cpp.so;/home/geoffrey/.conda/envs/caffe_27/lib/libhdf5_hl.so;/home/geoffrey/.conda/envs/caffe_27/lib/libhdf5_cpp.so;/home/geoffrey/.conda/envs/caffe_27/lib/libhdf5.so;/usr/lib/librt.so;/usr/lib/libpthread.so;/home/geoffrey/.conda/envs/caffe_27/lib/libz.so;/usr/lib/libdl.so;/usr/lib/libm.so;/home/geoffrey/.conda/envs/caffe_27/lib/libhdf5_hl_cpp.so;/home/geoffrey/.conda/envs/caffe_27/lib/libhdf5_hl.so;PUBLIC;/home/geoffrey/.conda/envs/caffe_27/lib/liblmdb.so;PUBLIC;/home/geoffrey/.conda/envs/caffe_27/lib/libleveldb.so;PRIVATE;/home/geoffrey/.conda/envs/caffe_27/lib/libsnappy.so;PUBLIC;/opt/cuda/lib64/libcudart.so;/opt/cuda/lib64/libcurand.so;/opt/cuda/lib64/libcublas.so;PUBLIC;/usr/lib/libcudnn.so;/home/geoffrey/.conda/envs/caffe_27/lib/libnccl.so;PUBLIC;opencv_core;opencv_highgui;opencv_imgproc;opencv_imgcodecs;PUBLIC;/home/geoffrey/.conda/envs/caffe_27/lib/liblapack.so;/home/geoffrey/.conda/envs/caffe_27/lib/libcblas.so;/home/geoffrey/.conda/envs/caffe_27/lib/libblas.so;PRIVATE;/home/geoffrey/.conda/envs/caffe_27/lib/libpython2.7.so;PUBLIC;/home/geoffrey/.conda/envs/caffe_27/lib/libboost_python.so
-- Configuring done
-- Generating done
-- Build files have been written to: /home/geoffrey/caffe/build

(4) 編譯

make all -j8

如果出現一下字樣, 編譯成功.

用pycaffe測試一下:

2. 問題搜集

1. ".Doxyfile" cannot be read.`問題
   
   解決方法:
   Doxyfile是文檔系統, 卸載掉本地的即可.

2. 未定義的應用
   
   安裝了libopencv, 但是缺少頭文件, 我conda install opencv解決.