理解caffe的fine-tuning

來源：知乎 https://www.zhihu.com/question/40850491

 

 

比如說，先設計出一個CNN結構。
然后用一個大的數據集A，訓練該CNN網絡，得到網絡a。
可是在數據集B上，a網絡預測效果並不理想（可能的原因是數據集A和B存在一些差異，比如數據來源不同導致的代表性差異）。如果直接用B的一部分訓練的話，數據量太小，CNN不適用。

解決方法：
將數據集B分為train集和test，以a網絡的參數為初始參數，以較小的學習率，以B的train集為訓練數據，繼續訓練，得到網絡b。

這樣，b在B的test集中一般可實現較好的預測精度。

作者：王藝程
鏈接：https://www.zhihu.com/question/40850491/answer/88651844
來源：知乎
著作權歸作者所有，轉載請聯系作者獲得授權。

 

——————————————————————————————————————————————————————————————————————————————

就是把現成的模型略加修改然后再作少量training，主要用於樣本數量不足的情形。

 

——————————————————————————————————————————————————————————————————————————————

把已經訓練過的模型應用到新的數據集上。主要優點是相比於從scratch訓練能在更短時間內達到相同的效果。
例子：
1.fine-tuning: 先拿CNN在cifar100上訓練，然后僅僅修改最后一層softmax的輸出節點個數（100改為10），再放到cifar10上訓練。
2. train from scratch: 相同結構的CNN直接用cifar10訓練。
結果：
第一種情況可能只要1000次迭代就能到達60%accuracy，第二種情況需要4000次才能達到60%的accuracy.
caffe官網有fine-tuning的例子，解釋更詳細。

作者：錢飛鴻
鏈接：https://www.zhihu.com/question/40850491/answer/88763800
來源：知乎
著作權歸作者所有，轉載請聯系作者獲得授權。

 

———————————————————————————————————————————————————————————————————————————————

這就是遷移學習，大概意思就是將一個任務訓練好的參數直接拿到另一個任務作為他的神經網絡初始參數值，然后進行訓練，這樣比直接隨機初始化的參數精度有提高。同時可以按照自己的需求設置某一些層的參數不變。

 

———————————————————————————————————————————————————————————————————————————————

參考文獻：文章 Fine-tuning Deep Convolutional Networks for Plant Recognition

Angie K. Reyes1 , Juan C. Caicedo2 , and Jorge E. Camargo1

1 Laboratory for Advanced Computational Science and Engineering Research, Universidad Antonio Nari˜no, Colombia

angreyes,jorgecamargo{@uan.edu.co},

2 Fundaci´on Universitaria Konrad Lorenz, Colombia

[e-mail]

 

主要內容是：用ImageNet的非常大量的數據集（1000多個分類）訓練得到的模型基礎上，fine-tune某個自定義的植物識別的數據集。

其中3.2節 Fine-tuning the CNN

 

      We initialized the CNN to recognize 1,000 categories of generic objects that are part of the ImageNet hierarchy following the procedure described in the previous section. Then, we proceed to finetune the network for the Plant Identification task.

      Fine-tuning a network is a procedure based on the concept of transfer learning [1, 3]. We start training a CNN to learn features for a broad domain with a classification function targeted at minimizing error in that domain. Then, we replace the classification function and optimize the network again to minimize error in another, more specific domain. Under this setting, we are transferring the features and the parameters of the network from the broad domain to the specific one.

     The classification function in the original CNN is a softmax classifier that computes the probability of 1,000 classes of the ImageNet dataset. To start the fine-tuning procedure, we remove this softmax classifier and initialize a new one with random values. The new softmax classifier is trained from scratch using the back-propagation algorithm with data from the Plant Identification task, which also has 1,000 different categories.

     In order to start the back-propagation algorithm for fine-tuning, it is key to set the learning rates of each layer appropriately. The classification layer, i.e., the new softmax classifier, needs a large learning rate because it has been just initialized with random values. The rest of the layers need a relatively small learning rate because we want to preserve the parameters of the previous network to transfer that knowledge into the new network. However, notice that the learning rate is not set to zero in the rest of the layers: they will be optimized again at a slower pace.

     In our experiments we set the learning rate of the top classification layer to 10, while leaving the learning rate of all the other seven layers to 0.1. We run the back-propagation algorithm for 50,000 iterations, which optimizes the network parameters using stochastic gradient descent (SGD). Figure 3 shows how the precision of classifying single images improves with more iterations. Our implementation is based on the open source Deep Learning library Caffe [7], and we run the experiments using a NVIDIA Titan Z GPU (5,760 cores and 12 GB of RAM).

 

 

Fig. 3. Evolution of image classification accuracy in a validation set during the finetuning process. Accuracy improves quickly during the first iterations and stabilizes after 40,000 iterations.

 

主要內容為：

      本文欲進行的植物分類的種類也是1000多種，但是分類會和ImageNet不同（ImageNet不僅僅有植物）。因此需要重新定義分類函數softmax的參數。目標是，使得根據scratch訓練得到的new softmax的向后傳播（back-propagation）的誤差（error）最小。

     啟動向后傳播算法的關鍵是，在每一層設置合適的學習率（learning rates）。最高層分類層，也就是新的softmax函數那一層，學習率要大一點，因為新的softmax函數是隨機初始化的。而其余的層都要用相對小一點的學習率，因為我們想保留之前pre-trained的網絡信息。但是注意，並不能為了保留之前信息就把其余層的參數設置為0，否則網絡會優化得更慢。

     因此，實驗中分類層的學習率設置為10，其余層的學習率設置為0.1。迭代50000次，用隨機梯度下降方法（stochastic gradient descent，SGD）來優化網絡（也就是使向后傳播誤差最小）。從圖中可以看到，第一次迭代准確率（Accuracy）就上升得特別快，大約在40000次迭代的時候趨於穩定。