图像语义分割代码实现（1）

谷歌最新语义图像分割模型 DeepLab-v3+ 现已开源 https://www.oschina.net/news/94257/google-open-sources-pixel-2-portrait-code

https://blog.csdn.net/zizi7/article/details/77163969

针对《图像语义分割（1）- FCN》介绍的FCN算法，以官方的代码为基础，在 SIFT-Flow 数据集上做训练和测试。

介绍了如何制作自己的训练数据

---

数据准备

参考文章《FCN网络的训练——以SIFT-Flow 数据集为例》

1） 首先 clone 官方工程

git clone https://github.com/shelhamer/fcn.berkeleyvision.org.git

工程是基于 CAFFE 的，所以也需要提前安装好

2）下载数据集及模型 
- 到这里下载 SIFT-Flow 数据集，解压缩到 fcn/data/sift-flow/ 下 
- 到这里下载 VGG-16 预训练模型，移动到 fcn/ilsvrc-nets/ 下 
- 参考文章《 FCN模型训练中遇到的困难》，到这里下载 VGG_ILSVRC_16_layers_deploy.prototxt 
　或者直接 copy 以下内容：

name: "VGG_ILSVRC_16_layers" input: "data" input_dim: 10 input_dim: 3 input_dim: 224 input_dim: 224 layers { bottom: "data" top: "conv1_1" name: "conv1_1" type: CONVOLUTION convolution_param { num_output: 64 pad: 1 kernel_size: 3 } } layers { bottom: "conv1_1" top: "conv1_1" name: "relu1_1" type: RELU } layers { bottom: "conv1_1" top: "conv1_2" name: "conv1_2" type: CONVOLUTION convolution_param { num_output: 64 pad: 1 kernel_size: 3 } } layers { bottom: "conv1_2" top: "conv1_2" name: "relu1_2" type: RELU } layers { bottom: "conv1_2" top: "pool1" name: "pool1" type: POOLING pooling_param { pool: MAX kernel_size: 2 stride: 2 } } layers { bottom: "pool1" top: "conv2_1" name: "conv2_1" type: CONVOLUTION convolution_param { num_output: 128 pad: 1 kernel_size: 3 } } layers { bottom: "conv2_1" top: "conv2_1" name: "relu2_1" type: RELU } layers { bottom: "conv2_1" top: "conv2_2" name: "conv2_2" type: CONVOLUTION convolution_param { num_output: 128 pad: 1 kernel_size: 3 } } layers { bottom: "conv2_2" top: "conv2_2" name: "relu2_2" type: RELU } layers { bottom: "conv2_2" top: "pool2" name: "pool2" type: POOLING pooling_param { pool: MAX kernel_size: 2 stride: 2 } } layers { bottom: "pool2" top: "conv3_1" name: "conv3_1" type: CONVOLUTION convolution_param { num_output: 256 pad: 1 kernel_size: 3 } } layers { bottom: "conv3_1" top: "conv3_1" name: "relu3_1" type: RELU } layers { bottom: "conv3_1" top: "conv3_2" name: "conv3_2" type: CONVOLUTION convolution_param { num_output: 256 pad: 1 kernel_size: 3 } } layers { bottom: "conv3_2" top: "conv3_2" name: "relu3_2" type: RELU } layers { bottom: "conv3_2" top: "conv3_3" name: "conv3_3" type: CONVOLUTION convolution_param { num_output: 256 pad: 1 kernel_size: 3 } } layers { bottom: "conv3_3" top: "conv3_3" name: "relu3_3" type: RELU } layers { bottom: "conv3_3" top: "pool3" name: "pool3" type: POOLING pooling_param { pool: MAX kernel_size: 2 stride: 2 } } layers { bottom: "pool3" top: "conv4_1" name: "conv4_1" type: CONVOLUTION convolution_param { num_output: 512 pad: 1 kernel_size: 3 } } layers { bottom: "conv4_1" top: "conv4_1" name: "relu4_1" type: RELU } layers { bottom: "conv4_1" top: "conv4_2" name: "conv4_2" type: CONVOLUTION convolution_param { num_output: 512 pad: 1 kernel_size: 3 } } layers { bottom: "conv4_2" top: "conv4_2" name: "relu4_2" type: RELU } layers { bottom: "conv4_2" top: "conv4_3" name: "conv4_3" type: CONVOLUTION convolution_param { num_output: 512 pad: 1 kernel_size: 3 } } layers { bottom: "conv4_3" top: "conv4_3" name: "relu4_3" type: RELU } layers { bottom: "conv4_3" top: "pool4" name: "pool4" type: POOLING pooling_param { pool: MAX kernel_size: 2 stride: 2 } } layers { bottom: "pool4" top: "conv5_1" name: "conv5_1" type: CONVOLUTION convolution_param { num_output: 512 pad: 1 kernel_size: 3 } } layers { bottom: "conv5_1" top: "conv5_1" name: "relu5_1" type: RELU } layers { bottom: "conv5_1" top: "conv5_2" name: "conv5_2" type: CONVOLUTION convolution_param { num_output: 512 pad: 1 kernel_size: 3 } } layers { bottom: "conv5_2" top: "conv5_2" name: "relu5_2" type: RELU } layers { bottom: "conv5_2" top: "conv5_3" name: "conv5_3" type: CONVOLUTION convolution_param { num_output: 512 pad: 1 kernel_size: 3 } } layers { bottom: "conv5_3" top: "conv5_3" name: "relu5_3" type: RELU } layers { bottom: "conv5_3" top: "pool5" name: "pool5" type: POOLING pooling_param { pool: MAX kernel_size: 2 stride: 2 } } layers { bottom: "pool5" top: "fc6" name: "fc6" type: INNER_PRODUCT inner_product_param { num_output: 4096 } } layers { bottom: "fc6" top: "fc6" name: "relu6" type: RELU } layers { bottom: "fc6" top: "fc6" name: "drop6" type: DROPOUT dropout_param { dropout_ratio: 0.5 } } layers { bottom: "fc6" top: "fc7" name: "fc7" type: INNER_PRODUCT inner_product_param { num_output: 4096 } } layers { bottom: "fc7" top: "fc7" name: "relu7" type: RELU } layers { bottom: "fc7" top: "fc7" name: "drop7" type: DROPOUT dropout_param { dropout_ratio: 0.5 } } layers { bottom: "fc7" top: "fc8" name: "fc8" type: INNER_PRODUCT inner_product_param { num_output: 1000 } } layers { bottom: "fc8" top: "prob" name: "prob" type: SOFTMAX }

---

训练脚本修改

1）生成 test、trainval、deploy

a. 执行 fcn/siftflow-fcn32s/net.py 生成 test.prototxt 和 trainval.prototxt 
b. cp test.prototxt 为 deploy.protxt

将第一个 data 层换成

layer {
  name: "input" type: "Input" top: "data" input_param { # These dimensions are purely for sake of example; # see infer.py for how to reshape the net to the given input size. shape { dim: 1 dim: 3 dim: 256 dim: 256 } } }

删除网络后面包含 loss 的层（一共2个）

2）修改 fcn/siftflow-fcn32s/solve.py

import caffe import surgery, score import numpy as np import os import sys try: import setproctitle setproctitle.setproctitle(os.path.basename(os.getcwd())) except: pass vgg_weights = '../ilsvrc-nets/vgg16-fcn.caffemodel' vgg_proto = '../ilsvrc-nets/VGG_ILSVRC_16_layers_deploy.prototxt' # init caffe.set_device(0) caffe.set_mode_gpu() solver = caffe.SGDSolver('solver.prototxt') #solver.net.copy_from(weights) vgg_net = caffe.Net(vgg_proto, vgg_weights, caffe.TRAIN) surgery.transplant(solver.net, vgg_net) del vgg_net # surgeries interp_layers = [k for k in solver.net.params.keys() if 'up' in k] surgery.interp(solver.net, interp_layers) # scoring test = np.loadtxt('../data/sift-flow/test.txt', dtype=str) for _ in range(50): solver.step(2000) # N.B. metrics on the semantic labels are off b.c. of missing classes; # score manually from the histogram instead for proper evaluation score.seg_tests(solver, False, test, layer='score_sem', gt='sem') score.seg_tests(solver, False, test, layer='score_geo', gt='geo')

3）修改 fcn/siftflow-fcn32s/solve.prototxt 
添加快照设置：

snapshot:4000 snapshot_prefix:"snapshot/train"

---

训练及测试

1） 复制 fcn/ 下的 infer.py、score.py、siftflow_layers.py、surgery.py 到 fcn/siftflow-fcn32s 下

2）python train.py 开始训练

3）修改 infer.py 的模型路径及测试图片路径

　　　　　　　　　　 
　　　　　　　　　　　　　　　　　　　　　　　图1. 迭代72000次的分割结果

4）之后可以以 fcn32s 的训练结果为基础，训练 fcn16s 和 fcn8s 
　需要注意的是，对于 fcn16s 和 fcn8s，由于不需要重新构造网络层，因此 solve.py 不需要改

import caffe import surgery, score import numpy as np import os import sys try: import setproctitle setproctitle.setproctitle(os.path.basename(os.getcwd())) except: pass weights = '../siftflow-fcn32s/snapshot/train_iter_100000.caffemodel' # init caffe.set_device(0) caffe.set_mode_gpu() solver = caffe.SGDSolver('solver.prototxt') solver.net.copy_from(weights) # surgeries interp_layers = [k for k in solver.net.params.keys() if 'up' in k] surgery.interp(solver.net, interp_layers) # scoring test = np.loadtxt('../data/sift-flow/test.txt', dtype=str) for _ in range(50): solver.step(2000) # N.B. metrics on the semantic labels are off b.c. of missing classes; # score manually from the histogram instead for proper evaluation score.seg_tests(solver, False, test, layer='score_sem', gt='sem') score.seg_tests(solver, False, test, layer='score_geo', gt='geo')

---

如何制作自己的训练数据

相比 detect（使用LabelImg框选目标），segment的数据需要耗费很大精力去准备

参考这篇帖子，MIT提供了一个在线标注多边形的工具LabelMe，但一般在工程上，为了尽量精确，更多还是使用 photoshop 的“快速选择”工具

1）首先用 ps 打开待标记图像，“图像->模式->灰度”，将图像转为灰度图 
2）使用“快速选择”工具，选出目标区域，“右键->填充->颜色”，假设该区域的 label 为 9 ，那么设置 RGB 为 (9,9,9)

　　　　　　　　　　　 
　　　　　　　　　　　　　　　　　　　　　　　　　　　图2. 选择区域并填充

3）所有类别填充完成后，“文件->存储为”label 图像

注意：以上方法针对 SegNet 里的 CamVid 数据格式（图3）

　　　　　　　　　　　　　　　　　　　　　　　
　　　　　　　　　　　　　　　　　　　　　　　　　图3. CamVid 数据格式

如图3所示，train和test里为RGB图像，trainannot和testannot里为标记过的label图像（灰度） 
　　　　　　一组训练（图3右）数据包含两张图像