0. AlexNet
1. VGG
VGG網絡相對來說，結構簡單，通俗易懂，作者通過分析2013年imagenet的比賽的最好模型，並發現感受野還是小的好，然后再加上《network in network》中的\(1*1\)卷積核，使得全文只在卷積網絡的深度上做文章，從而得出了網絡還是越深越好的結論
VGG
2. Inception
與VGG同期出來的有googlenet，該網絡通過關注減少模型參數，而不降低模型性能的角度出發，設計出了inception結構，提出了googlenet；
然后google人員發現因為網絡在訓練過程中，下層參數的變化會導致下層輸出的數據分布不斷變化，從而當前層乃至后面層都需要不斷地去擬合新的分布，這個想法也是來自於遷移學習中的covariate shift問題(即，訓練樣本和測試樣本其實不一致)。
而借鑒遷移學習的這個概念，如果細化到網絡內部的每一層的話，該現象又叫做internal covariate shift。
那么通過白化角度，的確可以解決這個問題，可是如果白化獨立於網絡的優化算法而存在，會導致網絡沒什么改變，而且會因為兩者不相關而讓其中的網絡參數不斷變大，即使loss穩定了，也還是會變大。所以就需要設計出一種即放入優化算法，而又能夠對每層網絡的輸入進行類白化的效果。而如果基於mini-batch進行0均值和1方差，本身又會引入新的問題：
從概率論角度出發：后驗概率=先驗概率*聯合概率。
如果只是做類白化，那么就相當於引入了聯合概率，從而數據的分布其實還是會變化的，而且拿sigmoid舉例的話，也會導致網絡根本達不到其非線性部分，而一直處在線性部分。
那么解決該問題的角度就是，引入修復變量，進行線性變換，使得變換后的數據分布能夠修復之前0均值1方差帶來的損失，從而提出了BN(被大家認為是inception v2)
inception v2
inception V3
inception V4
3. HighwayNet
4. ResNet
5. DenseNet
6. MobileNet
7. ShuffleNet
8. SENet
9. NASNet
參考文獻：
- [原理] Zeiler M D, Fergus R. Visualizing and understanding convolutional networks[C]//European conference on computer vision. Springer, Cham, 2014: 818-833.
- [alexnet] Krizhevsky A, Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural networks[C]//Advances in neural information processing systems. 2012: 1097-1105.
- [vgg] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint arXiv:1409.1556, 2014.
- [googlenet] Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 1-9.
.. [bn&inception v2] Ioffe S, Szegedy C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift[C]//Proceedings of the 32nd International Conference on Machine Learning (ICML-15). 2015: 448-456.
.. [BRN] Ioffe S. Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models[J]. arXiv preprint arXiv:1702.03275, 2017
.. [inception v3] Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 2818-2826.
.. [inception v4] Szegedy C, Ioffe S, Vanhoucke V, et al. Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning[C]//AAAI. 2017: 4278-4284. - [highway net] R. K. Srivastava, K. Greff, and J. Schmidhuber. Highway networks. arXiv:1505.00387, 2015.
.. [highway net v2] Srivastava R K, Greff K, Schmidhuber J. Training very deep networks[C]//Advances in neural information processing systems. 2015: 2377-2385. - [resnet v1] He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.
.. [resnet v2] He K, Zhang X, Ren S, et al. Identity mappings in deep residual networks[C]//European Conference on Computer Vision. Springer International Publishing, 2016: 630-645.
.. [wider or deeper] Wu Z, Shen C, Hengel A. Wider or deeper: Revisiting the resnet model for visual recognition[J]. arXiv preprint arXiv:1611.10080, 2016.
.. [WRN] Zagoruyko S, Komodakis N. Wide residual networks[J]. arXiv preprint arXiv:1605.07146, 2016.
..[ResNext] Xie S, Girshick R, Dollár P, et al. Aggregated residual transformations for deep neural networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017: 5987-5995. - [Dense Net] Huang G, Liu Z, Weinberger K Q, et al. Densely connected convolutional networks[J]. arXiv preprint arXiv:1608.06993, 2016.
.. [Dense Net] Pleiss G, Chen D, Huang G, et al. Memory-Efficient Implementation of DenseNets[J]. arXiv preprint arXiv:1707.06990, 2017.
.. [why&how] .DenseNet 的“what”、“why”和“how”
.. [多尺度DenseNet] Huang G, Chen D, Li T, et al. Multi-Scale Dense Convolutional Networks for Efficient Prediction[J]. arXiv preprint arXiv:1703.09844, 2017. - [MobileNet] Howard A G, Zhu M, Chen B, et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications[J]. arXiv preprint arXiv:1704.04861, 2017.
..[MobileNetV2] Sandler M, Howard A, Zhu M, et al. MobileNetV2: Inverted Residuals and Linear Bottlenecks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 4510-4520. - [SENet] Hu J, Shen L, Sun G. Squeeze-and-Excitation Networks[J]. arXiv preprint arXiv:1709.01507, 2017.
- [xception] Chollet F. Xception: Deep Learning with Depthwise Separable Convolutions[J]. arXiv preprint arXiv:1610.02357, 2016.
- [NASNet] B. Zoph and Q. V. Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017.
.. [NASNet] Zoph B, Vasudevan V, Shlens J, et al. Learning transferable architectures for scalable image recognition[J]. arXiv preprint arXiv:1707.07012, 2017. .