圖像語義分割代碼實現（1）

谷歌最新語義圖像分割模型 DeepLab-v3+ 現已開源 https://www.oschina.net/news/94257/google-open-sources-pixel-2-portrait-code

https://blog.csdn.net/zizi7/article/details/77163969

針對《圖像語義分割（1）- FCN》介紹的FCN算法，以官方的代碼為基礎，在 SIFT-Flow 數據集上做訓練和測試。

介紹了如何制作自己的訓練數據

---

數據准備

參考文章《FCN網絡的訓練——以SIFT-Flow 數據集為例》

1） 首先 clone 官方工程

git clone https://github.com/shelhamer/fcn.berkeleyvision.org.git

工程是基於 CAFFE 的，所以也需要提前安裝好

2）下載數據集及模型 
- 到這里下載 SIFT-Flow 數據集，解壓縮到 fcn/data/sift-flow/ 下 
- 到這里下載 VGG-16 預訓練模型，移動到 fcn/ilsvrc-nets/ 下 
- 參考文章《 FCN模型訓練中遇到的困難》，到這里下載 VGG_ILSVRC_16_layers_deploy.prototxt 
　或者直接 copy 以下內容：

name: "VGG_ILSVRC_16_layers" input: "data" input_dim: 10 input_dim: 3 input_dim: 224 input_dim: 224 layers { bottom: "data" top: "conv1_1" name: "conv1_1" type: CONVOLUTION convolution_param { num_output: 64 pad: 1 kernel_size: 3 } } layers { bottom: "conv1_1" top: "conv1_1" name: "relu1_1" type: RELU } layers { bottom: "conv1_1" top: "conv1_2" name: "conv1_2" type: CONVOLUTION convolution_param { num_output: 64 pad: 1 kernel_size: 3 } } layers { bottom: "conv1_2" top: "conv1_2" name: "relu1_2" type: RELU } layers { bottom: "conv1_2" top: "pool1" name: "pool1" type: POOLING pooling_param { pool: MAX kernel_size: 2 stride: 2 } } layers { bottom: "pool1" top: "conv2_1" name: "conv2_1" type: CONVOLUTION convolution_param { num_output: 128 pad: 1 kernel_size: 3 } } layers { bottom: "conv2_1" top: "conv2_1" name: "relu2_1" type: RELU } layers { bottom: "conv2_1" top: "conv2_2" name: "conv2_2" type: CONVOLUTION convolution_param { num_output: 128 pad: 1 kernel_size: 3 } } layers { bottom: "conv2_2" top: "conv2_2" name: "relu2_2" type: RELU } layers { bottom: "conv2_2" top: "pool2" name: "pool2" type: POOLING pooling_param { pool: MAX kernel_size: 2 stride: 2 } } layers { bottom: "pool2" top: "conv3_1" name: "conv3_1" type: CONVOLUTION convolution_param { num_output: 256 pad: 1 kernel_size: 3 } } layers { bottom: "conv3_1" top: "conv3_1" name: "relu3_1" type: RELU } layers { bottom: "conv3_1" top: "conv3_2" name: "conv3_2" type: CONVOLUTION convolution_param { num_output: 256 pad: 1 kernel_size: 3 } } layers { bottom: "conv3_2" top: "conv3_2" name: "relu3_2" type: RELU } layers { bottom: "conv3_2" top: "conv3_3" name: "conv3_3" type: CONVOLUTION convolution_param { num_output: 256 pad: 1 kernel_size: 3 } } layers { bottom: "conv3_3" top: "conv3_3" name: "relu3_3" type: RELU } layers { bottom: "conv3_3" top: "pool3" name: "pool3" type: POOLING pooling_param { pool: MAX kernel_size: 2 stride: 2 } } layers { bottom: "pool3" top: "conv4_1" name: "conv4_1" type: CONVOLUTION convolution_param { num_output: 512 pad: 1 kernel_size: 3 } } layers { bottom: "conv4_1" top: "conv4_1" name: "relu4_1" type: RELU } layers { bottom: "conv4_1" top: "conv4_2" name: "conv4_2" type: CONVOLUTION convolution_param { num_output: 512 pad: 1 kernel_size: 3 } } layers { bottom: "conv4_2" top: "conv4_2" name: "relu4_2" type: RELU } layers { bottom: "conv4_2" top: "conv4_3" name: "conv4_3" type: CONVOLUTION convolution_param { num_output: 512 pad: 1 kernel_size: 3 } } layers { bottom: "conv4_3" top: "conv4_3" name: "relu4_3" type: RELU } layers { bottom: "conv4_3" top: "pool4" name: "pool4" type: POOLING pooling_param { pool: MAX kernel_size: 2 stride: 2 } } layers { bottom: "pool4" top: "conv5_1" name: "conv5_1" type: CONVOLUTION convolution_param { num_output: 512 pad: 1 kernel_size: 3 } } layers { bottom: "conv5_1" top: "conv5_1" name: "relu5_1" type: RELU } layers { bottom: "conv5_1" top: "conv5_2" name: "conv5_2" type: CONVOLUTION convolution_param { num_output: 512 pad: 1 kernel_size: 3 } } layers { bottom: "conv5_2" top: "conv5_2" name: "relu5_2" type: RELU } layers { bottom: "conv5_2" top: "conv5_3" name: "conv5_3" type: CONVOLUTION convolution_param { num_output: 512 pad: 1 kernel_size: 3 } } layers { bottom: "conv5_3" top: "conv5_3" name: "relu5_3" type: RELU } layers { bottom: "conv5_3" top: "pool5" name: "pool5" type: POOLING pooling_param { pool: MAX kernel_size: 2 stride: 2 } } layers { bottom: "pool5" top: "fc6" name: "fc6" type: INNER_PRODUCT inner_product_param { num_output: 4096 } } layers { bottom: "fc6" top: "fc6" name: "relu6" type: RELU } layers { bottom: "fc6" top: "fc6" name: "drop6" type: DROPOUT dropout_param { dropout_ratio: 0.5 } } layers { bottom: "fc6" top: "fc7" name: "fc7" type: INNER_PRODUCT inner_product_param { num_output: 4096 } } layers { bottom: "fc7" top: "fc7" name: "relu7" type: RELU } layers { bottom: "fc7" top: "fc7" name: "drop7" type: DROPOUT dropout_param { dropout_ratio: 0.5 } } layers { bottom: "fc7" top: "fc8" name: "fc8" type: INNER_PRODUCT inner_product_param { num_output: 1000 } } layers { bottom: "fc8" top: "prob" name: "prob" type: SOFTMAX }

---

訓練腳本修改

1）生成 test、trainval、deploy

a. 執行 fcn/siftflow-fcn32s/net.py 生成 test.prototxt 和 trainval.prototxt 
b. cp test.prototxt 為 deploy.protxt

將第一個 data 層換成

layer {
  name: "input" type: "Input" top: "data" input_param { # These dimensions are purely for sake of example; # see infer.py for how to reshape the net to the given input size. shape { dim: 1 dim: 3 dim: 256 dim: 256 } } }

刪除網絡后面包含 loss 的層（一共2個）

2）修改 fcn/siftflow-fcn32s/solve.py

import caffe import surgery, score import numpy as np import os import sys try: import setproctitle setproctitle.setproctitle(os.path.basename(os.getcwd())) except: pass vgg_weights = '../ilsvrc-nets/vgg16-fcn.caffemodel' vgg_proto = '../ilsvrc-nets/VGG_ILSVRC_16_layers_deploy.prototxt' # init caffe.set_device(0) caffe.set_mode_gpu() solver = caffe.SGDSolver('solver.prototxt') #solver.net.copy_from(weights) vgg_net = caffe.Net(vgg_proto, vgg_weights, caffe.TRAIN) surgery.transplant(solver.net, vgg_net) del vgg_net # surgeries interp_layers = [k for k in solver.net.params.keys() if 'up' in k] surgery.interp(solver.net, interp_layers) # scoring test = np.loadtxt('../data/sift-flow/test.txt', dtype=str) for _ in range(50): solver.step(2000) # N.B. metrics on the semantic labels are off b.c. of missing classes; # score manually from the histogram instead for proper evaluation score.seg_tests(solver, False, test, layer='score_sem', gt='sem') score.seg_tests(solver, False, test, layer='score_geo', gt='geo')

3）修改 fcn/siftflow-fcn32s/solve.prototxt 
添加快照設置：

snapshot:4000 snapshot_prefix:"snapshot/train"

---

訓練及測試

1） 復制 fcn/ 下的 infer.py、score.py、siftflow_layers.py、surgery.py 到 fcn/siftflow-fcn32s 下

2）python train.py 開始訓練

3）修改 infer.py 的模型路徑及測試圖片路徑

　　　　　　　　　　 
　　　　　　　　　　　　　　　　　　　　　　　圖1. 迭代72000次的分割結果

4）之后可以以 fcn32s 的訓練結果為基礎，訓練 fcn16s 和 fcn8s 
　需要注意的是，對於 fcn16s 和 fcn8s，由於不需要重新構造網絡層，因此 solve.py 不需要改

import caffe import surgery, score import numpy as np import os import sys try: import setproctitle setproctitle.setproctitle(os.path.basename(os.getcwd())) except: pass weights = '../siftflow-fcn32s/snapshot/train_iter_100000.caffemodel' # init caffe.set_device(0) caffe.set_mode_gpu() solver = caffe.SGDSolver('solver.prototxt') solver.net.copy_from(weights) # surgeries interp_layers = [k for k in solver.net.params.keys() if 'up' in k] surgery.interp(solver.net, interp_layers) # scoring test = np.loadtxt('../data/sift-flow/test.txt', dtype=str) for _ in range(50): solver.step(2000) # N.B. metrics on the semantic labels are off b.c. of missing classes; # score manually from the histogram instead for proper evaluation score.seg_tests(solver, False, test, layer='score_sem', gt='sem') score.seg_tests(solver, False, test, layer='score_geo', gt='geo')

---

如何制作自己的訓練數據

相比 detect（使用LabelImg框選目標），segment的數據需要耗費很大精力去准備

參考這篇帖子，MIT提供了一個在線標注多邊形的工具LabelMe，但一般在工程上，為了盡量精確，更多還是使用 photoshop 的“快速選擇”工具

1）首先用 ps 打開待標記圖像，“圖像->模式->灰度”，將圖像轉為灰度圖 
2）使用“快速選擇”工具，選出目標區域，“右鍵->填充->顏色”，假設該區域的 label 為 9 ，那么設置 RGB 為 (9,9,9)

　　　　　　　　　　　 
　　　　　　　　　　　　　　　　　　　　　　　　　　　圖2. 選擇區域並填充

3）所有類別填充完成后，“文件->存儲為”label 圖像

注意：以上方法針對 SegNet 里的 CamVid 數據格式（圖3）

　　　　　　　　　　　　　　　　　　　　　　　
　　　　　　　　　　　　　　　　　　　　　　　　　圖3. CamVid 數據格式

如圖3所示，train和test里為RGB圖像，trainannot和testannot里為標記過的label圖像（灰度） 
　　　　　　一組訓練（圖3右）數據包含兩張圖像