TensorFlow學習筆記——LeNet-5（訓練自己的數據集）

　　在之前的TensorFlow學習筆記——圖像識別與卷積神經網絡（鏈接：請點擊我）中了解了一下經典的卷積神經網絡模型LeNet模型。那其實之前學習了別人的代碼實現了LeNet網絡對MNIST數據集的訓練。而這篇文章是想自己完成LeNet網絡來訓練自己的數據集。LeNet主要用來進行手寫字符的識別與分類，下面記錄一下自己學習的過程。

　　我的學習步驟分為以下四步：

• 1，溫習LeNet-5的網絡層
• 2，使用LeNet-5訓練MNIST數據集
• 3，使用LeNet-5訓練TFRecord格式的MNIST數據集
• 4，使用LeNet-5訓練自己的數據集（TFRecord格式）

　　LeNet是出自論文Gradient-Based Learning Applied to Document Recognition，是一種用於手寫體字符識別的非常高效的卷積神經網絡。那我要訓練的是印刷體的數字和字母，可能難點就是字母大小尺寸不一。下面我嘗試使用LeNet-5來進行識別。首先學習其網絡結構。

1，LeNet-5的網絡層解析

　　LeNet-5這個網絡雖然很小，但是包含了深度學習的基本模塊：卷積層，池化層，全連接層。是其他深度學習模型的基礎。LeNet網絡奠定了現代卷積神經網絡的基礎。LeNet-5模型總共有7層，這里我們對LeNet-5進行深入分析。

　　下圖展示了以MNIST數據集為例的LeNet-5模型架構：

 　　下面再啰嗦一次。

第一層：卷積層（C1層）

　　這一層為原始的圖像元素，LeNet-5模型接受的輸入層大小為32*32*1，第一層卷積層過濾器的尺寸為5*5，深度為6，不使用全0填充，所以這一層的輸出的尺寸為32-5+1=28面四個並打印為6，這一個卷積層總共有5*5*1*6+6=156個參數，其中6個未偏置項參數，因為下一層的節點矩陣有28*28*6=4704個結點，每個節點和5*5=25個當前層節點相連，每個神經元對應一個偏置項（這就是5*5+1的原因）所以本卷積層共有（5*5+1）*6*（28*28）=122304個連接。

　　那么也就是說，過濾器尺寸為[5, 5]，通道為1，深度為6。（除去輸入層和輸出層，我們有六個特征平面，包括兩個卷積層，兩個池化層，兩個全連接層），特征圖有6個，說明6個不同的卷積核，所以深度為6。

第二層：池化層（S2層）

　　池化層又叫做下采樣層，目的是壓縮數據，降低數據維度。

　　這一層的輸入為第一層的輸出，是一個28*28*6的節點矩陣，本層采用的過濾器大小為2*2，長和寬的步長均為2，所以本層的輸出矩陣大小為14*14*6

　　6個14*14的特征圖，每個圖中的每個單元與C1特征圖中的一個2*2鄰域相連接，不重疊。因此S2中每個特征圖的大小是C1中特征圖大小的1/4。

第三層：卷積層（C3層）

　　本層輸入的矩陣大小為14*14*6，使用的過濾器大小為5*5，深度為16.本層不使用全0填充，步長為1.本層的輸出矩陣大小為10*10*16。按照標准的卷積層，本層應該有5*5*6*16+16=2416個參數，10*10*16*（25+1）=41600個連接。

第四層：池化層（S4層）

　　本層的輸入矩陣大小為10*10*16，采用的過濾器大小為2*2，步長為2，本層的輸出矩陣大小為5*5*16

第五層：全連接層（C5層）

　　本層的輸入矩陣大小為5*5*16，在LeNet-5模型的論文中將這一層稱為卷積層，但是因為過濾器的大小就是5*5，所以和全連接層沒有區別，在之后的TensorFlow程序實現中也會將這一層看成全連接層。如果將5*5*16矩陣中的節點拉成一個向量，那么這一層和之前學習的全連接層就是一樣的了。

　　本層的輸出節點個數為120個，總共有5*5*16*120+120=48120個參數。

　　這一層還是卷積層，且有120個神經元，可以看做是120個特征圖，每張特征圖的大小為5*5，每個單元與S4層的全部16個單元的5*5領域相連，因此正好和池化層匹配（S4和C5之間的全連接）。

第六層：全連接層（F6層）

　　本層的輸入節點個數為120個，輸出節點個數為84個，總共參數為120*84+84=10164個。

　　之所以有84個單元，是因為輸出層的設計，與C5層全相連，F6層計算輸入向量和權重向量之間的點積，再加上一個偏置，所以84=7*12

第七層：全連接層（F7層）

　　本層的輸入節點個數為84個，輸出節點為10個，總共參數為84*10+10=850個。

　　該層有十個神經元，可以理解這是對於手寫體10個數，哪個輸出的數大，那個神經元代表的數字就是輸出。

1，卷積層tf.nn.conv2d()

　　函數類型如下：

def conv2d(  # pylint: disable=redefined-builtin,dangerous-default-value
    input,
    filter=None,
    strides=None,
    padding=None,
    use_cudnn_on_gpu=True,
    data_format="NHWC",
    dilations=[1, 1, 1, 1],
    name=None,
    filters=None):

　　參數說明：

• data_format：表示輸入的格式，有兩種分別為：“NHWC”和“NCHW”，默認為“NHWC”

• input：輸入是一個4維格式的（圖像）數據，數據的 shape 由 data_format 決定：當 data_format 為“NHWC”輸入數據的shape表示為[batch, in_height, in_width, in_channels]，分別表示訓練時一個batch的圖片數量、圖片高度、 圖片寬度、 圖像通道數。當 data_format 為“NHWC”輸入數據的shape表示為[batch, in_channels， in_height, in_width]

• filter：卷積核是一個4維格式的數據：shape表示為：[height,width,in_channels, out_channels]，分別表示卷積核的高、寬、深度（與輸入的in_channels應相同）、輸出 feature map的個數（即卷積核的個數）。

• strides：表示步長：一個長度為4的一維列表，每個元素跟data_format互相對應，表示在data_format每一維上的移動步長。當輸入的默認格式為：“NHWC”，則 strides = [batch , in_height , in_width, in_channels]。其中 batch 和 in_channels 要求一定為1，即只能在一個樣本的一個通道上的特征圖上進行移動，in_height , in_width表示卷積核在特征圖的高度和寬度上移動的布長，即 。

• padding：表示填充方式：“SAME”表示采用填充的方式，簡單地理解為以0填充邊緣，當stride為1時，輸入和輸出的維度相同；“VALID”表示采用不填充的方式，多余地進行丟棄。

2，池化層 tf.nn.max_pool()  /  tf.nn.avg_pool()

• value：表示池化的輸入：一個4維格式的數據，數據的 shape 由 data_format 決定，默認情況下shape 為[batch, height, width, channels]

• ksize：表示池化窗口的大小：一個長度為4的一維列表，一般為[1, height, width, 1]，因不想在batch和channels上做池化，則將其值設為1。

• 其他參數與上面類似。

3，softmax函數

　　舉個簡單的例子，假設一個五分類，然后一個樣本I的標簽  y = [0, 0, 0, 1, 0]，也就是說樣本I的真實標簽是4，假設模型預測的結果概率（softmax的輸出）p=[0.1,0.15,0.05,0.6,0.1]，可以看出這個預測是對的，那么對應的損失L=-log(0.6)，也就是當這個樣本經過這樣的網絡參數產生這樣的預測p時，它的損失是-log(0.6)。那么假設p=[0.15,0.2,0.4,0.1,0.15]，這個預測結果就很離譜了，因為真實標簽是4，而你覺得這個樣本是4的概率只有0.1（遠不如其他概率高，如果是在測試階段，那么模型就會預測該樣本屬於類別3），對應損失L=-log(0.1)。那么假設p=[0.05,0.15,0.4,0.3,0.1]，這個預測結果雖然也錯了，但是沒有前面那個那么離譜，對應的損失L=-log(0.3)。我們知道log函數在輸入小於1的時候是個負數，而且log函數是遞增函數，所以-log(0.6) < -log(0.3) < -log(0.1)。簡單講就是你預測錯比預測對的損失要大，預測錯得離譜比預測錯得輕微的損失要大。

 　　下面簡單說一下損失函數 softmax loss：

 　　首先，L是損失，Sj是softmax的輸出向量S的第j個值，那Sj表示這個樣本屬於第j個類別的概率。yj 前面有個求和符號，j的范圍是1~T（即類別1到類別T），因此y是一個1*T的向量，里面的T個值，而且只有1個值是1，其他 T-1個值都是0，那么那個位置的值是1呢？真實標簽對應的位置的那個值是1，其他都是0，所以這個公式可以改為：

 　　當然，此時要限定 j是指當前樣本的真實標簽。

4，cross  entropy交叉熵

　　損失函數是在訓練過程中，得到的輸出值與理想輸出值的差距，這里有很多衡量的標准，比如均方誤差的方法，也就是輸出值和理想值的方差來表示：

loss = tf.reduce_mean(tf.square(yout - Y))

　　公式如下：

 　　TensorFlow使用的函數如下：

tf.nn.dropout(x, keep_prob, noise_shape=None, seed=None,name=None) 

　　參數意義：

• 第一個參數x：指輸入　
• 第二個參數keep_prob: 設置神經元被選中的概率,在初始化時keep_prob是一個占位符,  keep_prob = tf.placeholder(tf.float32) 。tensorflow在run時設置keep_prob具體的值，例如keep_prob: 0.5
• 第五個參數name：指定該操作的名字。

5，執行訓練

　　執行訓練的過程就是讓已有的輸入和輸出數據一次又一次的填入模型中，根據數據進行訓練。什么時候進行呢？那就是需要搭建一個平台，讓它在這個平台上運行，這個平台叫做會話，tf.Session()，就好比你修好了路，當然不作為擺設，你要在某個時間段內開始通車。

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for i in range(1000):
    	sess.run(optimizer, feed_dict = {X : x, Y : y})

　　

6，LeNet-5的局限性

　　卷積網絡在本質上是一種輸入到輸出的映射，它能夠學習大量的輸入與輸出之間的映射關系，而不需要任何輸入和輸出之間的精確的數學表達式。

　　CNN能夠得出原始圖像的有效表征，這使得CNN能夠直接從原始像素中，經過極少的預處理，識別視覺上面的規律。然而，由於當時缺乏大規模訓練數據，計算機的計算能力也跟不上，LeNet-5對於復雜問題的處理結果並不理想。

　　2006年起，人們設計了很多方法，想要克服難以訓練深度CNN的困難。其中，最出名的時，Krizhevsky et al 提出了一個經典的CNN結構，並在圖像識別任務上取得了重大突破，其方法的整體框架叫做AlexNet，與LeNet-5類似，但是要加深一點，此后深度卷積如雨后春筍一般，出現了很多。

 

 2，Lenet-5實現MNIST數據訓練（使用官方格式數據）

　　首先，我們熱個手，使用MNIST數據的官方格式，直接進行訓練。這里數據沒有進行處理，也沒有轉化。

　　代碼：

# _*_coding:utf-8_*_
'''
LeNet是第一個成功應用於數字識別問題的卷積神經網絡
LeNet模型總共有7層，在MNIST上LeNet-5模型可以達到99.2%的正確率
'''
import tensorflow as tf
import tensorflow.examples.tutorials.mnist.input_data as input_data
import time
import os

os.environ['CUDA_VISIBLE_DEVICES'] = '1,2'


# 獲取mnist數據
mnist_data_set = input_data.read_data_sets('MNIST_data', one_hot=True)
# 聲明輸入圖片數據類型，mnist的手寫體圖片大小為28*28=784
# None表示行向量的維度是任意的，也就是一次可以輸出多張圖片
# placeholder 基本都是占位符，先定義，后面會用到的
x = tf.placeholder('float', [None, 784])
# y_為網絡的輸出,數字十個類別
y_ = tf.placeholder('float', [None, 10])

# 把輸入的數據轉變成二維形式，用於卷積計算
# -1表示一次可以存儲多張照片，1表示圖像的通道數為1
x_image = tf.reshape(x, [-1, 28, 28, 1])

# 卷積核初始化，大小為6個5*5的卷積核，1和x_image的1對應，即為圖像的通道數
filter1 = tf.Variable(tf.truncated_normal([5, 5, 1, 6]))
# 偏置項
bias1 = tf.Variable(tf.truncated_normal([6]))
# 二維卷積計算
conv1 = tf.nn.conv2d(x_image, filter1, strides=[1, 1, 1, 1], padding='SAME')
h_conv1 = tf.nn.sigmoid(conv1 + bias1)
# 池化層
maxPool2 = tf.nn.max_pool(h_conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')

filter2 = tf.Variable(tf.truncated_normal([5, 5, 6, 16]))
bias2 = tf.Variable(tf.truncated_normal([16]))
conv2 = tf.nn.conv2d(maxPool2, filter2, strides=[1, 1, 1, 1], padding='SAME')
h_conv2 = tf.nn.sigmoid(conv2 + bias2)

maxPool3 = tf.nn.max_pool(h_conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')

filter3 = tf.Variable(tf.truncated_normal([5, 5, 16, 120]))
bias3 = tf.Variable(tf.truncated_normal([120]))
conv3 = tf.nn.conv2d(maxPool3, filter3, strides=[1, 1, 1, 1], padding='SAME')
h_conv3 = tf.nn.sigmoid(conv3 + bias3)

# 全連接層，權重初始化
W_fc1 = tf.Variable(tf.truncated_normal([7 * 7 * 120, 80]))
# 偏置項
b_fc1 = tf.Variable(tf.truncated_normal([80]))
# 將卷積的輸出展開
h_pool2_flat = tf.reshape(h_conv3, [-1, 7 * 7 * 120])
h_fc1 = tf.nn.sigmoid(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)

W_fc2 = tf.Variable(tf.truncated_normal([80, 10]))
b_fc2 = tf.Variable(tf.truncated_normal([10]))
# 輸出層，使用softmax進行多分類
y_conv = tf.nn.softmax(tf.matmul(h_fc1, W_fc2) + b_fc2)

# 損失函數，交叉熵
cross_entropy = -tf.reduce_sum(y_ * tf.log(y_conv))
# 使用梯度下降法來更新權重，學習速率為0.001，改為0.01的話會導致權重更新有問題，准確率會滴
train_step = tf.train.GradientDescentOptimizer(0.001).minimize(cross_entropy)

sess = tf.InteractiveSession()

# 測試准確率，tf.argmax() 計算行或列的最大值，返回最大值下標的向量
# tf.equal() 計算兩個向量對應的元素是否相等，返回數據為bool
# tf.cast() 把bool數據轉化為浮點型
correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, 'float'))

# 對所有變量進行初始化
sess.run(tf.global_variables_initializer())

start_time = time.time()
for i in range(50000):
    # 獲取訓練數據，每次100張
    # 我們后面取數據的時候，直接獲取feed_dict={x: batch[0], y_true: batch[1]}
    # batch = mnist_data_set.train.next_batch(60)
    batch_xs, batch_ys = mnist_data_set.train.next_batch(100)
    train_step.run(feed_dict={x: batch_xs, y_: batch_ys})
    # 每個100次輸出當前的准確率
    if i % 100 == 0:
        train_accuracy = accuracy.eval(feed_dict={x: batch_xs, y_: batch_ys})
        print('step %d, training accuracy %g' % (i, train_accuracy))
        # 計算時間間隔
        end_time = time.time()
        print('time: ', (end_time - start_time))
        start_time = end_time

print("Test accuracy: {}".format(accuracy.eval(session=sess,
                                               feed_dict={
                                                   x: mnist_data_set.test.images,
                                                   y_: mnist_data_set.test.labels})))

# 關閉會話
sess.close()

　　訓練過程如下：

step 48200, training accuracy 1
time:  0.240248441696167
step 48300, training accuracy 1
time:  0.24075675010681152
step 48400, training accuracy 1
time:  0.3005177974700928
step 48500, training accuracy 1
time:  0.2406449317932129
step 48600, training accuracy 1
time:  0.24107027053833008
step 48700, training accuracy 0.99
time:  0.24148797988891602
step 48800, training accuracy 0.98
time:  0.24003911018371582
step 48900, training accuracy 0.99
time:  0.23934340476989746
step 49000, training accuracy 1
time:  0.28536438941955566
step 49100, training accuracy 0.98
time:  0.24154186248779297
step 49200, training accuracy 1
time:  0.24214410781860352
step 49300, training accuracy 0.96
time:  0.24277353286743164
step 49400, training accuracy 1
time:  0.2415931224822998
step 49500, training accuracy 1
time:  0.2908365726470947
step 49600, training accuracy 1
time:  0.24100923538208008
step 49700, training accuracy 1
time:  0.2422494888305664
step 49800, training accuracy 1
time:  0.24294066429138184
step 49900, training accuracy 0.99
time:  0.24169301986694336
Test accuracy: 0.9772999882698059

　　訓練的准確率基本在98%~100%之間，並且測試集達到了97.7%的准確率，效果還是不錯的。

　　注意：

# *****************   改進1  *****************
x = tf.placeholder('float', shape=[None, 28 * 28])
y_ = tf.placeholder('float', shape=[None, 10])

# 當然這里定義數據類型的時候也可以這樣寫
x = tf.placeholder(tf.float32, [None, 784])
y_ = tf.placeholder(tf.float32, [None, 10])


# *****************   改進2  *****************
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
#這里用Ａdam優化器優化函數　也可以使用隨機梯度下降優化函數
learning_rate = 0.001
optimizer= tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)


# *****************   改進3  *****************
# 通過tf.reshape函數將第五層的輸出變成一個batch的向量
# reshaped = tf.reshape(relu3, [pool_shape[0], nodes])
     改為：（不然會報錯）
reshaped = tf.layers.flatten(relu3)        

　　

3，使用LeNet-5訓練MNIST數據集（將MNIST數據轉化為TFRecord格式）

　　大多數書上和網絡上實例就是使用已經封裝好的mnist數據集來訓練卷積神經網絡。但是這里我想用神經網絡來訓練自己的數據集，所以如何將自己的數據集導入神經網絡？這里我首先將MNIST數據轉化為TFRecord數據格式進行訓練，下一步訓練自己的數據集。

　　我的解決思路如下：

1，將自己的數據集轉換為TensorFlow支持的神經網絡輸入格式TFRecord。

2，重建LeNet-5卷積神經網絡

3，對神經網絡進行訓練

1，制作TFRecord的MNIST數據集

　　首先，將自己的數據集轉化為TFRecord，我之前的博客有說，這里不再學習。

　　想學習，請參考博客：高效讀取數據的方法（TFRecord）

　　將MNIST數據集中的所有文件存儲到一個TFRecord文件中，然后我們用的時候，直接讀取封裝好的MNIST數據的TFRecord文件。

 　　但是后面需要使用如何讀取TFRecord文件的函數，這里注意，你生成TFRecord文件的函數和讀取的函數一定要保持一致，不然會報錯。

　　將MNIST數據集格式轉化為TFRecord格式，前面有講過，這里不再贅述。

　　想學習，請參考博客：使用TensorFlow操作MNIST數據（2）

 

　　這里粘貼一下處理MNIST數據的代碼：

1，讀取MNIST數據，並轉化為TFRecord格式：

#_*_coding:utf-8_*_
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
import os
import numpy as np

# 生成整數的屬性
def _int64_feature(value):
    return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))

# 生成字符串類型的屬性
def _bytes_feature(value):
    return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))

def create_records(images, labels, num_example, outpout):
    '''
    實現將MNIST數據集轉化為records
    注意：讀取的圖像數據默認為uint8，然后轉化為tf的字符串型BytesList保存
    :return:
    '''
    # 訓練圖像的分辨率，作為example的屬性
    pixels = images.shape[1]
    # 創建一個writer來寫TFRecord文件
    writer = tf.python_io.TFRecordWriter(outpout)

    # 將每張圖片都轉化為一個Example
    for i in range(num_example):
        # 將圖像轉化為字符串
        image_raw = images[i].tostring()

        # 將一個樣例轉化為Example，Protocal Buffer並將所有信息寫入這個數據結構
        example = tf.train.Example(features=tf.train.Features(
            feature={
                'pixels': _int64_feature(pixels),
                'labels': _int64_feature(np.argmax(labels[i])),
                'image_raw': _bytes_feature(image_raw)
            }
        ))

        # 將Example寫入TFRecord文件
        writer.write(example.SerializeToString())
    print("data processing success")
    writer.close()



if __name__ == '__main__':
    if not os.path.exists('mnistrecord'):
        os.mkdir('mnistrecord')
    # 導入MNIST數據集
    mnist = input_data.read_data_sets('MNIST_data/', dtype=tf.uint8, one_hot=True)
    train_images = mnist.train.images
    train_labels = mnist.train.labels
    train_num_example = mnist.train.num_examples
    # 存儲train_TFRecord文件的地址
    train_filename = 'mnistrecord/trainmnist28.tfrecords'
    create_records(train_images, train_labels, train_num_example, train_filename)

    test_images = mnist.test.images
    test_labels = mnist.test.labels
    test_num_example = mnist.test.num_examples
    # 存儲train_TFRecord文件的地址
    test_filename = 'mnistrecord/testmnist28.tfrecords'
    create_records(test_images, test_labels, test_num_example, test_filename)

 

2，讀取MNIST的TFRecord文件

# _*_coding:utf-8_*_
import tensorflow as tf


def read_record(filename):
    '''
    讀取TFRecord文件
    :return:
    '''
    # 創建一個隊列來維護輸入文件列表
    filename_queue = tf.train.string_input_producer([filename])
    # 創建一個reader來讀取TFRecord文件中Example
    reader = tf.TFRecordReader()
    # 從文件中讀取一個Example
    _, serialized_example = reader.read(filename_queue)

    # 用FixedLenFeature 將讀入的Example解析成tensor
    features = tf.parse_single_example(
        serialized_example,
        features={
            'image_raw': tf.FixedLenFeature([], tf.string),
            'pixels': tf.FixedLenFeature([], tf.int64),
            'labels': tf.FixedLenFeature([], tf.int64)
        }
    )

    # tf.decode_raw將字符串解析成圖像對應的像素數組
    images = tf.decode_raw(features['image_raw'], tf.uint8)
    labels = tf.cast(features['labels'], tf.int32)
    pixels = tf.cast(features['pixels'], tf.int32)

    init_op = tf.global_variables_initializer()

    with tf.Session() as sess:
        sess.run(init_op)
        # 啟動多線程處理輸入數據
        coord = tf.train.Coordinator()
        threads = tf.train.start_queue_runners(sess=sess, coord=coord)

        # 每次運行讀出一個Example,當所有樣例讀取完之后，再次樣例中程序會重頭 讀取
        for i in range(10):
            # 在會話中會取出image 和label
            image, label = sess.run([images, labels])
        coord.request_stop()
        coord.join(threads)
        print("end code")


if __name__ == '__main__':
    train_filename = 'mnistrecord/trainmnist28.tfrecords'
    test_filename = 'mnistrecord/testmnist28.tfrecords'
    read_record(filename=train_filename)

　　

 

2，定義神經網絡結構

　　定義LeNet-5的結構，注意之前已經說過了，這里不再贅述，只貼代碼。其實網上各個博客啊，書啊寫的代碼都大同小異，雖然定義形式不同，但是LeNet-5的結構確是一步都不少，所以各位還是按照自己寫代碼的風格寫就行。

　　下面粘貼一個TensorFlow實戰中定義的LeNet-5結構的定義代碼：

# _*_coding:utf-8_*_
import tensorflow as tf

# 配置神經網絡的參數
INPUT_NODE = 784  # 這里輸入的參數是圖片的尺寸 這里是28*28=784
OUTPUT_NODE = 51  # 這里是輸出的圖片類型，總共51類

IMAGE_SIZE = 28
NUM_CHANNELS = 3
NUM_LABELS = 51

# 第一層卷積層的尺寸和深度
CONV1_DEEP = 6
CONV1_SIZE = 5

# 第三層卷積層的尺寸和深度
CONV2_DEEP = 16
CONV2_SIZE = 5

# 第五層卷積層的尺寸和深度
CONV3_DEEP = 120
CONV3_SIZE = 5

# 全連接層的節點個數
FC_SIZE = 84


# 定義卷積神經網絡的前向傳播過程
# 這里添加一個新的參數train，用於區分訓練過程和測試過程
# 在這個程序中將用到Dropout方法，dropout可以進一步提升模型可靠性，並防止過擬合
# 注意dropout層只能在訓練過程中使用
def inference(input_tensor, train, regularizer):
    '''
    聲明第一層卷積層的變量並實現前向傳播過程，通過使用不同的命名空間來隔離不同層的變量
    這可以讓每一層中變量命名只需要考慮在當前層的作用，而不需要擔心命名重復的問題
    和標准的LeNet-5模型不太一樣，這里定義的卷積層輸入為28*28*1的原始MNIST圖片箱數
    因為卷積層使用了全0填充，所以輸出為28*28*6的矩陣
    :param input_tensor:
    :param train:
    :param regularizer:
    :return:
    '''
    with tf.variable_scope('layer1-conv1'):  # [5,5,3,6]
        conv1_weights = tf.get_variable("weight",
                                        [CONV1_SIZE, CONV1_SIZE, NUM_CHANNELS, CONV1_DEEP],
                                        initializer=tf.truncated_normal_initializer(stddev=0.1))
        conv1_biases = tf.get_variable('bias', [CONV1_DEEP],
                                       initializer=tf.constant_initializer(0.0))

        # 使用邊長為5， 深度為6的過濾器，過濾器移動的步長為1，且使用全0填充
        conv1 = tf.nn.conv2d(input_tensor, conv1_weights,
                             strides=[1, 1, 1, 1], padding='SAME')
        relu1 = tf.nn.relu(tf.nn.bias_add(conv1, conv1_biases))

    # 實現第二層池化層的前向傳播過程，這里選用最大池化層
    # 池化層過濾器的邊長為2，使用全零填充且移動的步長為2，這一層的輸入為上一層的輸出
    # 也就是28*28*6的矩陣  輸出為14*14*6的矩陣
    with tf.name_scope('layer2-pool1'):
        pool1 = tf.nn.max_pool(relu1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1],
                               padding='SAME')

    # 聲明第三層卷積層的變量並實現前向傳播過程，這一層輸入為14*14*6的矩陣
    # 因為卷積層沒有使用全零填充，所以輸出為10*10*16的矩陣
    with tf.variable_scope('layer3-conv2'):  # [5,5,3,16]
        conv2_weights = tf.get_variable("weight",
                                        [CONV2_SIZE, CONV2_SIZE, NUM_CHANNELS, CONV2_DEEP],
                                        initializer=tf.truncated_normal_initializer(stddev=0.1))
        conv2_biases = tf.get_variable('bias', [CONV2_DEEP],
                                       initializer=tf.constant_initializer(0.0))

        # 使用邊長為5， 深度為16的過濾器，過濾器移動的步長為1，bububu不使用全0填充
        conv2 = tf.nn.conv2d(pool1, conv2_weights,
                             strides=[1, 1, 1, 1], padding='VALID')
        relu2 = tf.nn.relu(tf.nn.bias_add(conv2, conv2_biases))

    # 實現第四層池化層的前向傳播過程，這一層和第二層的結構是一樣的
    # 這里輸入10*10*16的矩陣，輸出為5*5*16的矩陣
    with tf.name_scope('layer4-pool2'):
        pool2 = tf.nn.max_pool(relu2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1],
                               padding='VALID')

    # 聲明第五層全連接層（實際上為卷積層）的變量並實現前向傳播過程
    # 這一層輸入是5*5*16的矩陣，因為沒有使用全0填充，所以輸出為1*1*120
    with tf.name_scope('layer5-conv3'):
        conv3_weights = tf.get_variable("weight",
                                        [CONV3_SIZE, CONV3_SIZE, NUM_CHANNELS, CONV3_DEEP],
                                        initializer=tf.truncated_normal_initializer(stddev=0.1))
        conv3_biases = tf.get_variable('bias', [CONV3_DEEP],
                                       initializer=tf.constant_initializer(0.0))

        # 使用邊長為5， 深度為6的過濾器，過濾器移動的步長為1，bububu不使用全0填充
        conv3 = tf.nn.conv2d(pool2, conv3_weights,
                             strides=[1, 1, 1, 1], padding='VALID')
        relu3 = tf.nn.relu(tf.nn.bias_add(conv3, conv3_biases))

    # 將第五層卷積層的輸出轉化為第六層全連接層的輸入格式
    # 第五層的輸出為1*1*120的矩陣，然而第六層全連接層需要的輸出格式為向量
    # 所以這里需要將這個1*1*120的矩陣拉直成一個向量
    # relu3.get_shape函數可以得到第五層輸出矩陣的維度而不需要手工計算。
    # 注意因為每一層神經網絡的輸入輸出都為一個batch的矩陣，
    # 所以這里得到的維度也包含了一個batch中數據的個數。
    pool_shape = relu3.get_shape().as_list()

    # 計算將矩陣拉直成向量之后的長度，這個長度就是矩陣長度及深度的乘積
    # 注意這里pool_shape[0]為一個batch中數據的個數
    nodes = pool_shape[1] * pool_shape[2] * pool_shape[3]
    # 通過tf.reshape函數將第五層的輸出變成一個batch的向量
    reshaped = tf.reshape(relu3, [pool_shape[0], nodes])

    # 聲明第六層全連接層的變量並實現前向傳播過程，這一層的輸入是拉直之后的一組向量
    # 向量的長度為1120，輸出是一組長度為84的向量
    # 這一層和之前的LeNet基本一致，唯一的區別就是引入的dropout層
    # dropout在訓練時會隨機將部分節點的輸出改為0
    # dropout可以避免過擬合問題，從而使得在測試數據上的效果更好
    # dropout一般只在全連接層而不是卷積層或者池化層使用
    with tf.variable_scope('layer6-fc1'):
        fc1_weights = tf.get_variable('weight', [nodes, FC_SIZE],
                                      initializer=tf.truncated_normal_initializer(stddev=0.1))
        # 只有全連接層的權重需要加入正則化
        if regularizer != None:
            tf.add_to_collection('losses', regularizer(fc1_weights))

        fc1_biases = tf.get_variable('bias', [FC_SIZE])
        initializer = tf.constant_initializer(0.1)
        fc1 = tf.nn.relu(tf.matmul(reshaped, fc1_weights) + fc1_biases)
        if train:
            fc1 = tf.nn.dropout(fc1, 0.5)

    # 聲明第七層全連接的變量並實現前向傳播過程，這一層的輸入為一組長度為為84的向量
    #輸出為51的向量，這一層的輸出通過softmax之后就得到了最后的分類結果
    with tf.variable_scope('layer7-fc2'):
        fc2_weights = tf.get_variable('weight',
                                      [FC_SIZE, NUM_LABELS],
                                      initializer=tf.truncated_normal_initializer(stddev=0.1))
        if regularizer != None:
            tf.add_to_collection('losses', regularizer(fc2_weights))
        fc2_biases = tf.get_variable('bias', [NUM_LABELS])
        initializer = tf.constant_initializer(0.1)
        logit = tf.matmul(fc1, fc2_weights) + fc2_biases

    # 返回第七層的輸出
    return logit

　　

3，輸入數據的解析和預處理

　　下面對輸入的數據進行解析。

# _*_coding:utf-8_*_
import tensorflow as tf
import os

# os.environ['CUDA_VISIBLE_DEVICES'] = '1'

# 1,輸入數據的解析和預處理
def read_records(filename, resize_height, resize_width, type=None):
    '''
    解析record文件：源文件的圖像數據是RGB，uint8 【0， 255】一般作為訓練數據時，需要歸一化到[0, 1]
    :param filename:
    :param resize_height:
    :param resize_width:
    :param type: 選擇圖像數據的返回類型
                 None:默認將uint8-[0,255]轉為float32-[0,255]
                 normalization:歸一化float32-[0,1]
                 centralization:歸一化float32-[0,1],再減均值中心化
    :return:
    '''
    # 創建文件隊列，不限讀取的數量
    filename_queue = tf.train.string_input_producer([filename])
    # create a reader from file queue
    reader = tf.TFRecordReader()
    # reader 從文件隊列中讀入一個序列化的樣本
    _, serialized_example = reader.read(filename_queue)
    # get feature from serialized example
    # 解析符號化的樣本
    features = tf.parse_single_example(
        serialized_example,
        features={
            'image_raw': tf.FixedLenFeature([], tf.string),
            'height': tf.FixedLenFeature([], tf.int64),
            'width': tf.FixedLenFeature([], tf.int64),
            'depth': tf.FixedLenFeature([], tf.int64),
            'label': tf.FixedLenFeature([], tf.int64)
        }
    )
    tf_image = tf.decode_raw(features['image_raw'], tf.uint8)  # 獲得圖像原始的數據

    tf_height = features['height']
    tf_width = features['width']
    tf_depth = features['depth']
    tf_label = tf.cast(features['label'], tf.int32)
    # PS: 恢復原始圖像數據，reshape的大小必須與保存之前的圖像shape一致，否則出錯
    tf_image = tf.reshape(tf_image, [resize_height, resize_width, 3])  # 設置圖像的維度

    # 存儲的圖像類型為uint8，TensorFlow訓練時數據必須是tf.float32
    if type is None:
        tf_image = tf.cast(tf_image, tf.float32)
    elif type == 'normalization':  # [1]若需要歸一化請使用:
        # 僅當輸入數據是uint8,才會歸一化[0,255]
        # tf_image = tf.image.convert_image_dtype(tf_image, tf.float32)
        tf_image = tf.cast(tf_image, tf.float32) * (1. / 255.0)  # 歸一化
    elif type == 'centralization':
        # 若需要歸一化,且中心化,假設均值為0.5,請使用:
        tf_image = tf.cast(tf_image, tf.float32) * (1. / 255) - 0.5  # 中心化

        # 這里僅僅返回圖像和標簽
        # return tf_image, tf_height,tf_width,tf_depth,tf_label
    return tf_image, tf_label


def get_batch_images(images, labels, batch_size, labels_nums, one_hot=False,
                     shuffle=False, num_threads=1):
    '''
    :param images: 圖像
    :param labels: 標簽
    :param batch_size:
    :param labels_nums: 標簽個數
    :param one_hot: 是否將labels轉化為one_hot 的形式
    :param shuffle: 是否打亂順序，一般train時，shuffle=True，驗證時shuffle=False
    :param num_threads:
    :return: 返回batch的images和labels
    '''
    min_after_dequeue = 200
    # 保證 capacity必須大於 min_after_dequeue的參數值
    capacity = min_after_dequeue + 3 * batch_size
    if shuffle:
        images_batch, labels_batch = tf.train.shuffle_batch([images, labels],
                                                            batch_size=batch_size,
                                                            capacity=capacity,
                                                            min_after_dequeue=min_after_dequeue,
                                                            num_threads=num_threads)
    else:
        images_batch, labels_batch = tf.train.batch([images, labels],
                                                    batch_size=batch_size,
                                                    capacity=capacity,
                                                    num_threads=num_threads)
    if one_hot:
        labels_batch = tf.one_hot(labels_batch, labels_nums, 1, 0)
    return images_batch, labels_batch

　　處理之后，返回的時批量的image和對應的label。

　　這里我沒有用上面的數據訓練，其實是這樣的。就是說，我做了一個實驗，將MNIST的數據讀出來，並寫成圖片，我發現，每個字母和數字均在一個文件夾下面。而不是說這10類，每個都在自己的文件夾下面，這么整齊。所以我沒有直接將MNIST寫成TFRecord格式，然后讀取TFRecord格式，我發現這是不可行的。

　　所以，我將MNIST數據轉化為圖片，並且每個數字放在每個數字的文件夾下面，然后訓練。這樣就可以按照我想要的方式訓練了，而且可以驗證一下MNIST的准確率，是否和上面的接近，畢竟對於源數據，我只做了的操作就是，將其轉換為圖片，再轉化為TFRecord格式再訓練。如果類似，那就沒問題。下面嘗試。

4，MNIST數據轉換為圖片

　　結果如圖，即轉化為這樣：

 　　第二層，（隨便進一個目錄）

 　　第三層（隨便進入一個目錄）

　　代碼如下：

# _*_coding:utf-8_*_
import cv2
import numpy as np
import tensorflow as tf
from tensorflow.examples.tutorials import mnist
from tensorflow.examples.tutorials.mnist import input_data
from PIL import Image
import os


# save raw image
def save_raw():
    # read data from mnist. if data not exist, will download automatically
    mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
    # we save image raw data to data/raw/ folder
    # if the folder not there, create it
    save_dir = 'MNIST_data/raw/'
    if os.path.exists(save_dir) is False:
        os.makedirs(save_dir)

    # save 20 images in training dataset
    for i in range(20):
        # please attention，mnist.train.images[i, :] is ith image, sequence started from 0
        image_array = mnist.train.images[i, :]
        # the image in MNIST of TensorFlow, image is 784 length vector, we recover it to 28x28 image
        image_array = image_array.reshape(28, 28)
        # save image as mnist_train_0.jpg, mnist_train_1.jpg, ... ,mnist_train_19.jpg
        filename = save_dir + 'mnist_train_%d.jpg' % i
        # save image_array as image
        # use Image.fromarray to convert image，then call save function to save
        # because Image.fromarray is not good to support float, we have to convert it to uint8 and then read as 'L'
        Image.fromarray((image_array * 255).astype('uint8'), mode='L').convert('RGB').save(filename)


def convert_mnist_img_raw(data, data1, save_path):
    for i in range(data.images.shape[0]):
        # TensorFlow中的MNIST圖片是一個784維的向量，我們重新把它還原為28x28維的圖像。
        image_array = data.images[i, :]
        image_array = image_array.reshape(28, 28)
        img_num = (image_array * 255).astype(np.uint8)
        # img_num = (image_array * 255).astype('uint8')
        label = data1.labels[i]
        # cv2.imshow('image', img)
        # cv2.waitKey(500)
        filename = save_path + '/{}_{}.jpg'.format(label, i)
        print(filename)
        cv2.imwrite(filename, img_num)
        # Image.fromarray(img_num, mode='L').save(filename)


def convert_mnist_img(data, data1, save_path):
    for i in range(data.images.shape[0]):
        # TensorFlow中的MNIST圖片是一個784維的向量，我們重新把它還原為28x28維的圖像。
        image_array = data.images[i, :]
        image_array = image_array.reshape(28, 28)
        img_num = (image_array * 255).astype(np.uint8)
        # img_num = (image_array * 255).astype('uint8')
        label = data1.labels[i]
        if not os.path.exists(os.path.join(save_path, str(label))):
            os.mkdir(os.path.join(save_path, str(label)))
        # cv2.imshow('image', img)
        # cv2.waitKey(500)
        filename = save_path + '/' + str(label) + '/{}_{}.jpg'.format(label, i)
        print(filename)
        cv2.imwrite(filename, img_num)
        # Image.fromarray(img_num, mode='L').save(filename)




if __name__ == '__main__':
    mnist = input_data.read_data_sets('MNIST_data/', one_hot=True)
    mnist1 = input_data.read_data_sets('MNIST_data/')
    content_name = ['MNIST_train', 'MNIST_test', 'MNIST_validation']
    # print(mnist.validation.images.shape[0])
    for i in content_name:
        if not os.path.exists(i):
            os.mkdir(i)
    # convert_mnist_img_raw(mnist.validation, mnist1.validation, 'MNIST_validation')  # 55000
    convert_mnist_img(mnist.train, mnist1.train, 'MNIST_train')  # 55000
    print('convert training data to image complete')
    convert_mnist_img(mnist.test, mnist1.test, 'MNIST_test')  # 10000
    print('convert test data to image complete')
    convert_mnist_img(mnist.validation, mnist1.validation, 'MNIST_validation')  # 5000
    print('convert validation data to image complete')

　　轉化之后呢，我們其實就和下面要學習的訓練自己的數據集一樣了。這里就不粘貼訓練代碼了，因為下面訓練自己的數據的代碼和這個其實一樣，只不過是數據集變了而已。

5，使用LeNet-5訓練TFRecord格式的MNIST數據

　　下面只展示一下，使用上面定義的LeNet-5模型，和調用slim中谷歌官方定義好的LeNet-5模型訓練的結果，看是否類似。（如果效果不好，就調參，這種模型，我將其訓練批次調大了，學習率提高了，比較模型深度不是很深）

1，slim自帶的LeNet-5模型訓練結果

　　訓練結果如下：

2019-12-07 17:24:49.511197: Step [90100]  train Loss : 0.034825, training accuracy :  0.99
2019-12-07 17:24:51.960556: Step [90200]  train Loss : 0.031647, training accuracy :  0.98
2019-12-07 17:24:54.401640: Step [90300]  train Loss : 0.013821, training accuracy :  1
2019-12-07 17:24:56.827365: Step [90400]  train Loss : 0.009416, training accuracy :  1
2019-12-07 17:24:59.285245: Step [90500]  train Loss : 0.050181, training accuracy :  0.99
2019-12-07 17:25:00.231120: Step [90500]  val Loss : 0.036682, val accuracy :  0.9886
2019-12-07 17:25:02.613078: Step [90600]  train Loss : 0.043965, training accuracy :  1
2019-12-07 17:25:05.262525: Step [90700]  train Loss : 0.003886, training accuracy :  1
2019-12-07 17:25:07.643164: Step [90800]  train Loss : 0.008936, training accuracy :  0.99
2019-12-07 17:25:10.086635: Step [90900]  train Loss : 0.026223, training accuracy :  1
2019-12-07 17:25:12.562384: Step [91000]  train Loss : 0.044388, training accuracy :  1
2019-12-07 17:25:13.759702: Step [91000]  val Loss : 0.036466, val accuracy :  0.989
------save:mnistmodelslenet/best_models_91000_0.9890.ckpt
2019-12-07 17:25:16.260234: Step [91100]  train Loss : 0.004893, training accuracy :  1
2019-12-07 17:25:18.732957: Step [91200]  train Loss : 0.008099, training accuracy :  1
2019-12-07 17:25:21.232881: Step [91300]  train Loss : 0.010900, training accuracy :  1
2019-12-07 17:25:23.690347: Step [91400]  train Loss : 0.006691, training accuracy :  1
2019-12-07 17:25:26.077798: Step [91500]  train Loss : 0.019277, training accuracy :  1
2019-12-07 17:25:27.110041: Step [91500]  val Loss : 0.036495, val accuracy :  0.9884
2019-12-07 17:25:29.541535: Step [91600]  train Loss : 0.005347, training accuracy :  1
2019-12-07 17:25:32.070571: Step [91700]  train Loss : 0.030143, training accuracy :  0.99
2019-12-07 17:25:34.517069: Step [91800]  train Loss : 0.001880, training accuracy :  1
2019-12-07 17:25:36.974632: Step [91900]  train Loss : 0.013069, training accuracy :  1
2019-12-07 17:25:39.431848: Step [92000]  train Loss : 0.017495, training accuracy :  1
2019-12-07 17:25:40.314588: Step [92000]  val Loss : 0.036441, val accuracy :  0.9892
------------save:mnistmodelslenet/model.ckpt--92000
------save:mnistmodelslenet/best_models_92000_0.9892.ckpt
2019-12-07 17:25:42.887467: Step [92100]  train Loss : 0.020958, training accuracy :  0.99
2019-12-07 17:25:45.348507: Step [92200]  train Loss : 0.017127, training accuracy :  1
2019-12-07 17:25:47.756854: Step [92300]  train Loss : 0.019064, training accuracy :  1
2019-12-07 17:25:50.219324: Step [92400]  train Loss : 0.032903, training accuracy :  0.99
2019-12-07 17:25:52.707338: Step [92500]  train Loss : 0.011051, training accuracy :  1
2019-12-07 17:25:53.819837: Step [92500]  val Loss : 0.036477, val accuracy :  0.9888
2019-12-07 17:25:56.214045: Step [92600]  train Loss : 0.005283, training accuracy :  1
2019-12-07 17:25:58.574659: Step [92700]  train Loss : 0.038343, training accuracy :  0.99
2019-12-07 17:26:01.022120: Step [92800]  train Loss : 0.029472, training accuracy :  1
2019-12-07 17:26:03.614888: Step [92900]  train Loss : 0.029430, training accuracy :  1
2019-12-07 17:26:06.048187: Step [93000]  train Loss : 0.005918, training accuracy :  1
2019-12-07 17:26:07.162769: Step [93000]  val Loss : 0.037015, val accuracy :  0.9886
2019-12-07 17:26:09.589461: Step [93100]  train Loss : 0.013980, training accuracy :  1
2019-12-07 17:26:11.909845: Step [93200]  train Loss : 0.022814, training accuracy :  1
2019-12-07 17:26:14.209764: Step [93300]  train Loss : 0.051712, training accuracy :  0.99
2019-12-07 17:26:16.525746: Step [93400]  train Loss : 0.003427, training accuracy :  1
2019-12-07 17:26:18.908377: Step [93500]  train Loss : 0.049566, training accuracy :  0.97
2019-12-07 17:26:19.907547: Step [93500]  val Loss : 0.037099, val accuracy :  0.9882
2019-12-07 17:26:22.199820: Step [93600]  train Loss : 0.005968, training accuracy :  1
2019-12-07 17:26:24.493318: Step [93700]  train Loss : 0.026799, training accuracy :  0.99
2019-12-07 17:26:26.861115: Step [93800]  train Loss : 0.040578, training accuracy :  1
2019-12-07 17:26:29.220762: Step [93900]  train Loss : 0.035674, training accuracy :  1
2019-12-07 17:26:31.592105: Step [94000]  train Loss : 0.018340, training accuracy :  1
2019-12-07 17:26:32.606197: Step [94000]  val Loss : 0.036546, val accuracy :  0.9886
------------save:mnistmodelslenet/model.ckpt--94000
2019-12-07 17:26:35.125010: Step [94100]  train Loss : 0.026360, training accuracy :  0.99
2019-12-07 17:26:37.567095: Step [94200]  train Loss : 0.041744, training accuracy :  0.99
2019-12-07 17:26:40.025393: Step [94300]  train Loss : 0.031769, training accuracy :  0.99
2019-12-07 17:26:42.449197: Step [94400]  train Loss : 0.031708, training accuracy :  1
2019-12-07 17:26:44.815271: Step [94500]  train Loss : 0.011965, training accuracy :  1
2019-12-07 17:26:45.691042: Step [94500]  val Loss : 0.036568, val accuracy :  0.9886
2019-12-07 17:26:48.046238: Step [94600]  train Loss : 0.098086, training accuracy :  0.98
2019-12-07 17:26:50.442090: Step [94700]  train Loss : 0.003194, training accuracy :  1
2019-12-07 17:26:52.843926: Step [94800]  train Loss : 0.011078, training accuracy :  1
2019-12-07 17:26:55.218067: Step [94900]  train Loss : 0.046635, training accuracy :  0.99
2019-12-07 17:26:57.702115: Step [95000]  train Loss : 0.085678, training accuracy :  0.99
2019-12-07 17:26:58.642553: Step [95000]  val Loss : 0.036330, val accuracy :  0.9888
2019-12-07 17:27:01.059867: Step [95100]  train Loss : 0.005422, training accuracy :  1
2019-12-07 17:27:03.646401: Step [95200]  train Loss : 0.033761, training accuracy :  0.99
2019-12-07 17:27:06.157741: Step [95300]  train Loss : 0.012471, training accuracy :  1
2019-12-07 17:27:08.633951: Step [95400]  train Loss : 0.006597, training accuracy :  1
2019-12-07 17:27:11.085771: Step [95500]  train Loss : 0.006471, training accuracy :  1
2019-12-07 17:27:12.274676: Step [95500]  val Loss : 0.036216, val accuracy :  0.9886
2019-12-07 17:27:14.697503: Step [95600]  train Loss : 0.010805, training accuracy :  1
2019-12-07 17:27:17.166638: Step [95700]  train Loss : 0.095174, training accuracy :  0.98
2019-12-07 17:27:19.631921: Step [95800]  train Loss : 0.011734, training accuracy :  1
2019-12-07 17:27:22.005459: Step [95900]  train Loss : 0.004943, training accuracy :  1
2019-12-07 17:27:24.429525: Step [96000]  train Loss : 0.041252, training accuracy :  0.99
2019-12-07 17:27:25.490167: Step [96000]  val Loss : 0.036082, val accuracy :  0.9892
------------save:mnistmodelslenet/model.ckpt--96000
2019-12-07 17:27:28.031883: Step [96100]  train Loss : 0.013173, training accuracy :  1
2019-12-07 17:27:30.490614: Step [96200]  train Loss : 0.135211, training accuracy :  0.99
2019-12-07 17:27:32.920059: Step [96300]  train Loss : 0.008340, training accuracy :  1
2019-12-07 17:27:35.366707: Step [96400]  train Loss : 0.029178, training accuracy :  0.99
2019-12-07 17:27:37.778924: Step [96500]  train Loss : 0.023783, training accuracy :  1
2019-12-07 17:27:38.662121: Step [96500]  val Loss : 0.035978, val accuracy :  0.989
2019-12-07 17:27:41.007801: Step [96600]  train Loss : 0.020418, training accuracy :  1
2019-12-07 17:27:43.373862: Step [96700]  train Loss : 0.005956, training accuracy :  1
2019-12-07 17:27:45.718948: Step [96800]  train Loss : 0.037369, training accuracy :  1
2019-12-07 17:27:48.053417: Step [96900]  train Loss : 0.001790, training accuracy :  1
2019-12-07 17:27:50.418627: Step [97000]  train Loss : 0.012481, training accuracy :  1
2019-12-07 17:27:51.278477: Step [97000]  val Loss : 0.035982, val accuracy :  0.9886
2019-12-07 17:27:53.621815: Step [97100]  train Loss : 0.036737, training accuracy :  1
2019-12-07 17:27:55.998817: Step [97200]  train Loss : 0.018639, training accuracy :  1
2019-12-07 17:27:58.385280: Step [97300]  train Loss : 0.094415, training accuracy :  0.99
2019-12-07 17:28:00.782485: Step [97400]  train Loss : 0.054417, training accuracy :  1
2019-12-07 17:28:03.282768: Step [97500]  train Loss : 0.018801, training accuracy :  1
2019-12-07 17:28:04.231569: Step [97500]  val Loss : 0.035901, val accuracy :  0.989
2019-12-07 17:28:06.564398: Step [97600]  train Loss : 0.015873, training accuracy :  1
2019-12-07 17:28:08.946638: Step [97700]  train Loss : 0.045842, training accuracy :  1
2019-12-07 17:28:11.314134: Step [97800]  train Loss : 0.032827, training accuracy :  1
2019-12-07 17:28:13.673403: Step [97900]  train Loss : 0.031645, training accuracy :  1
2019-12-07 17:28:16.060070: Step [98000]  train Loss : 0.021970, training accuracy :  1
2019-12-07 17:28:17.015618: Step [98000]  val Loss : 0.035966, val accuracy :  0.9888
------------save:mnistmodelslenet/model.ckpt--98000
2019-12-07 17:28:19.395503: Step [98100]  train Loss : 0.019510, training accuracy :  0.99
2019-12-07 17:28:21.769422: Step [98200]  train Loss : 0.004839, training accuracy :  1
2019-12-07 17:28:24.167367: Step [98300]  train Loss : 0.041200, training accuracy :  1
2019-12-07 17:28:26.626932: Step [98400]  train Loss : 0.003081, training accuracy :  1
2019-12-07 17:28:29.028384: Step [98500]  train Loss : 0.008532, training accuracy :  1
2019-12-07 17:28:30.105535: Step [98500]  val Loss : 0.036248, val accuracy :  0.9888
2019-12-07 17:28:32.468981: Step [98600]  train Loss : 0.005621, training accuracy :  1
2019-12-07 17:28:34.832019: Step [98700]  train Loss : 0.036646, training accuracy :  0.98
2019-12-07 17:28:37.111677: Step [98800]  train Loss : 0.013360, training accuracy :  1
2019-12-07 17:28:39.440640: Step [98900]  train Loss : 0.008248, training accuracy :  1
2019-12-07 17:28:41.834450: Step [99000]  train Loss : 0.027080, training accuracy :  1
2019-12-07 17:28:42.989248: Step [99000]  val Loss : 0.036153, val accuracy :  0.9894
------save:mnistmodelslenet/best_models_99000_0.9894.ckpt
2019-12-07 17:28:45.395565: Step [99100]  train Loss : 0.019798, training accuracy :  1
2019-12-07 17:28:47.756673: Step [99200]  train Loss : 0.002656, training accuracy :  1
2019-12-07 17:28:50.155678: Step [99300]  train Loss : 0.011562, training accuracy :  0.99
2019-12-07 17:28:52.494549: Step [99400]  train Loss : 0.052770, training accuracy :  0.99
2019-12-07 17:28:54.866737: Step [99500]  train Loss : 0.015489, training accuracy :  1
2019-12-07 17:28:56.039745: Step [99500]  val Loss : 0.036166, val accuracy :  0.9892
2019-12-07 17:28:58.406051: Step [99600]  train Loss : 0.016589, training accuracy :  0.99
2019-12-07 17:29:00.762953: Step [99700]  train Loss : 0.034062, training accuracy :  0.99
2019-12-07 17:29:03.175084: Step [99800]  train Loss : 0.013509, training accuracy :  0.99
2019-12-07 17:29:05.604738: Step [99900]  train Loss : 0.004104, training accuracy :  1
2019-12-07 17:29:08.008282: Step [100000]  train Loss : 0.009301, training accuracy :  1
2019-12-07 17:29:09.053212: Step [100000]  val Loss : 0.036054, val accuracy :  0.989
------------save:mnistmodelslenet/model.ckpt--100000

　　

2，上面定義的LeNet-5模型訓練結果

 　　訓練結果如下：

2019-12-07 16:37:20.799084: Step [94100]  train Loss : 0.207280, training accuracy :  0.97
2019-12-07 16:37:23.077235: Step [94200]  train Loss : 0.288277, training accuracy :  0.94
2019-12-07 16:37:25.308687: Step [94300]  train Loss : 0.230948, training accuracy :  0.95
2019-12-07 16:37:27.574248: Step [94400]  train Loss : 0.149708, training accuracy :  0.96
2019-12-07 16:37:29.899668: Step [94500]  train Loss : 0.215018, training accuracy :  0.93
2019-12-07 16:37:30.988016: Step [94500]  val Loss : 0.202200, val accuracy :  0.9526
2019-12-07 16:37:33.308174: Step [94600]  train Loss : 0.310347, training accuracy :  0.94
2019-12-07 16:37:35.577415: Step [94700]  train Loss : 0.171983, training accuracy :  0.96
2019-12-07 16:37:37.849625: Step [94800]  train Loss : 0.160050, training accuracy :  0.94
2019-12-07 16:37:40.166671: Step [94900]  train Loss : 0.186521, training accuracy :  0.95
2019-12-07 16:37:42.523405: Step [95000]  train Loss : 0.402581, training accuracy :  0.9
2019-12-07 16:37:43.603988: Step [95000]  val Loss : 0.192111, val accuracy :  0.9556
2019-12-07 16:37:45.922971: Step [95100]  train Loss : 0.194871, training accuracy :  0.97
2019-12-07 16:37:48.187889: Step [95200]  train Loss : 0.215393, training accuracy :  0.97
2019-12-07 16:37:50.445268: Step [95300]  train Loss : 0.158608, training accuracy :  0.98
2019-12-07 16:37:52.774339: Step [95400]  train Loss : 0.238952, training accuracy :  0.96
2019-12-07 16:37:55.142354: Step [95500]  train Loss : 0.190960, training accuracy :  0.94
2019-12-07 16:37:56.216869: Step [95500]  val Loss : 0.192757, val accuracy :  0.9556
2019-12-07 16:37:58.410845: Step [95600]  train Loss : 0.139895, training accuracy :  0.94
2019-12-07 16:38:00.652184: Step [95700]  train Loss : 0.314228, training accuracy :  0.96
2019-12-07 16:38:03.027260: Step [95800]  train Loss : 0.430718, training accuracy :  0.94
2019-12-07 16:38:05.350156: Step [95900]  train Loss : 0.289241, training accuracy :  0.93
2019-12-07 16:38:07.706012: Step [96000]  train Loss : 0.239347, training accuracy :  0.93
2019-12-07 16:38:08.765761: Step [96000]  val Loss : 0.184876, val accuracy :  0.9594
------------save:mnistmodelslenet/model.ckpt--96000
------save:mnistmodelslenet/best_models_96000_0.9594.ckpt
2019-12-07 16:38:11.129844: Step [96100]  train Loss : 0.219303, training accuracy :  0.91
2019-12-07 16:38:13.435976: Step [96200]  train Loss : 0.141345, training accuracy :  0.97
2019-12-07 16:38:15.708837: Step [96300]  train Loss : 0.133305, training accuracy :  0.97
2019-12-07 16:38:18.028155: Step [96400]  train Loss : 0.179352, training accuracy :  0.97
2019-12-07 16:38:20.361665: Step [96500]  train Loss : 0.175561, training accuracy :  0.96
2019-12-07 16:38:21.455463: Step [96500]  val Loss : 0.196051, val accuracy :  0.9538
2019-12-07 16:38:23.723929: Step [96600]  train Loss : 0.207480, training accuracy :  0.96
2019-12-07 16:38:26.097808: Step [96700]  train Loss : 0.289101, training accuracy :  0.92
2019-12-07 16:38:28.399593: Step [96800]  train Loss : 0.205839, training accuracy :  0.95
2019-12-07 16:38:30.676930: Step [96900]  train Loss : 0.162070, training accuracy :  0.97
2019-12-07 16:38:32.984553: Step [97000]  train Loss : 0.287059, training accuracy :  0.89
2019-12-07 16:38:34.081359: Step [97000]  val Loss : 0.187286, val accuracy :  0.9586
2019-12-07 16:38:36.332393: Step [97100]  train Loss : 0.289821, training accuracy :  0.93
2019-12-07 16:38:38.690675: Step [97200]  train Loss : 0.180355, training accuracy :  0.94
2019-12-07 16:38:41.001430: Step [97300]  train Loss : 0.133220, training accuracy :  0.95
2019-12-07 16:38:43.329969: Step [97400]  train Loss : 0.226960, training accuracy :  0.95
2019-12-07 16:38:45.693958: Step [97500]  train Loss : 0.194481, training accuracy :  0.93
2019-12-07 16:38:46.738666: Step [97500]  val Loss : 0.187791, val accuracy :  0.9576
2019-12-07 16:38:49.010076: Step [97600]  train Loss : 0.172692, training accuracy :  0.94
2019-12-07 16:38:51.285696: Step [97700]  train Loss : 0.150980, training accuracy :  0.95
2019-12-07 16:38:53.544084: Step [97800]  train Loss : 0.150606, training accuracy :  0.96
2019-12-07 16:38:55.860818: Step [97900]  train Loss : 0.191217, training accuracy :  0.93
2019-12-07 16:38:58.213331: Step [98000]  train Loss : 0.252721, training accuracy :  0.94
2019-12-07 16:38:59.307352: Step [98000]  val Loss : 0.191428, val accuracy :  0.9552
------------save:mnistmodelslenet/model.ckpt--98000
2019-12-07 16:39:01.620290: Step [98100]  train Loss : 0.218954, training accuracy :  0.95
2019-12-07 16:39:04.016540: Step [98200]  train Loss : 0.353565, training accuracy :  0.92
2019-12-07 16:39:06.314311: Step [98300]  train Loss : 0.169997, training accuracy :  0.99
2019-12-07 16:39:08.590750: Step [98400]  train Loss : 0.201064, training accuracy :  0.96
2019-12-07 16:39:10.943120: Step [98500]  train Loss : 0.100543, training accuracy :  1
2019-12-07 16:39:12.034701: Step [98500]  val Loss : 0.191718, val accuracy :  0.9548
2019-12-07 16:39:14.344574: Step [98600]  train Loss : 0.220537, training accuracy :  0.95
2019-12-07 16:39:16.655703: Step [98700]  train Loss : 0.195116, training accuracy :  0.94
2019-12-07 16:39:18.929639: Step [98800]  train Loss : 0.203131, training accuracy :  0.96
2019-12-07 16:39:21.243188: Step [98900]  train Loss : 0.167681, training accuracy :  0.95
2019-12-07 16:39:23.582402: Step [99000]  train Loss : 0.339591, training accuracy :  0.93
2019-12-07 16:39:24.708781: Step [99000]  val Loss : 0.188099, val accuracy :  0.9536
2019-12-07 16:39:26.917352: Step [99100]  train Loss : 0.359073, training accuracy :  0.93
2019-12-07 16:39:29.100260: Step [99200]  train Loss : 0.283282, training accuracy :  0.94
2019-12-07 16:39:31.351049: Step [99300]  train Loss : 0.289421, training accuracy :  0.93
2019-12-07 16:39:33.642279: Step [99400]  train Loss : 0.240311, training accuracy :  0.92
2019-12-07 16:39:35.913139: Step [99500]  train Loss : 0.192217, training accuracy :  0.94
2019-12-07 16:39:36.802653: Step [99500]  val Loss : 0.191198, val accuracy :  0.958
2019-12-07 16:39:39.002652: Step [99600]  train Loss : 0.188396, training accuracy :  0.97
2019-12-07 16:39:41.357152: Step [99700]  train Loss : 0.111778, training accuracy :  0.96
2019-12-07 16:39:43.685702: Step [99800]  train Loss : 0.178156, training accuracy :  0.94
2019-12-07 16:39:46.043234: Step [99900]  train Loss : 0.124271, training accuracy :  0.97
2019-12-07 16:39:48.407596: Step [100000]  train Loss : 0.094911, training accuracy :  0.97
2019-12-07 16:39:49.310749: Step [100000]  val Loss : 0.193183, val accuracy :  0.9572
------------save:mnistmodelslenet/model.ckpt--100000

　　總的來說，人家谷歌封裝在slim中:LeNet-5 比我自己搭建的Lenet-5模型效果好點，但是呢，我的大體上已經達到要求了，就不強求了，哈哈哈。

　　到此為止，我們使用LeNet-5模型訓練了MNIST數據集，下面使用LeNet-5訓練自己的模型。其實已經很簡單了。直接更換數據集即可。

4，使用LeNet-5訓練自己的數據集（TFRecord格式）

　　這里的數據集處理格式就粘貼了，前面有講，而且有MNIST數據格式轉化的代碼。

　　這里的LeNet-5模型是使用 slim 中官方定義的模型。

　　（注意：這里自己的數據集通道為3，但是MNIST數據集的通道為1，看是否一致）

　　最后使用LeNet-5模型訓練自己的數據集，這里我將數據集尺寸大小重置為28*28，比較LeNet-5默認定義的圖片大小為28*28，而這里數據使用之前的51類進行分類訓練，下面直接貼上代碼：

# _*_coding:utf-8_*_
import tensorflow as tf
import numpy as np
import pdb
import os
from datetime import datetime
import slim.nets.lenet as lenet
from create_tf_record import *
import tensorflow.contrib.slim as slim

labels_nums = 51  # 類別個數
batch_size = 16
resize_height = 28  # mobilenet_v1.default_image_size 指定存儲圖片高度
resize_width = 28  # mobilenet_v1.default_image_size 指定存儲圖片高度
depths = 3
data_shape = [batch_size, resize_height, resize_width, depths]

# 定義input_images為圖片數據
input_images = tf.placeholder(dtype=tf.float32,
                              shape=[None, resize_height, resize_width, depths],
                              name='input')

# 定義input_labels為lables數據
# input_labels = tf.placeholder(dtype=tf.int32, shape=[None], name='label')
input_labels = tf.placeholder(dtype=tf.int32,
                              shape=[None, labels_nums],
                              name='label')

# 定義 dropout的概率
keep_prob = tf.placeholder(tf.float32, name='keep_prob')
is_training = tf.placeholder(tf.bool, name='is_training')


# 評價函數
def net_evaluation(sess, loss, accuracy, val_images_batch,
                   val_labels_batch, val_nums):
    val_max_steps = int(val_nums / batch_size)
    val_losses = []
    val_accs = []
    for _ in range(val_max_steps):
        val_x, val_y = sess.run([val_images_batch, val_labels_batch])
        # print('labels:',val_y)
        # val_loss = sess.run(loss, feed_dict={x: val_x, y: val_y, keep_prob: 1.0})
        # val_acc = sess.run(accuracy,feed_dict={x: val_x, y: val_y, keep_prob: 1.0})
        val_loss, val_acc = sess.run([loss, accuracy],
                                     feed_dict={input_images: val_x,
                                                input_labels: val_y,
                                                keep_prob: 1.0,
                                                is_training: False})
        val_losses.append(val_loss)
        val_accs.append(val_acc)
    mean_loss = np.array(val_losses, dtype=np.float32).mean()
    mean_acc = np.array(val_accs, dtype=np.float32).mean()
    return mean_loss, mean_acc


def step_train(train_op, loss, accuracy,
               train_images_batch, train_labels_batch, train_nums, train_log_step,
               val_images_batch, val_labels_batch, val_nums, val_log_step,
               snapshot_prefix, snapshot):
    '''
    循環迭代訓練過程
    :param train_op: 訓練op
    :param loss:     loss函數
    :param accuracy: 准確率函數
    :param train_images_batch: 訓練images數據
    :param train_labels_batch: 訓練labels數據
    :param train_nums:         總訓練數據
    :param train_log_step:   訓練log顯示間隔
    :param val_images_batch: 驗證images數據
    :param val_labels_batch: 驗證labels數據
    :param val_nums:         總驗證數據
    :param val_log_step:     驗證log顯示間隔
    :param snapshot_prefix: 模型保存的路徑
    :param snapshot:        模型保存間隔
    :return: None
    '''
    saver = tf.train.Saver(max_to_keep=5)
    max_acc = 0.0
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        sess.run(tf.local_variables_initializer())
        coord = tf.train.Coordinator()
        threads = tf.train.start_queue_runners(sess=sess, coord=coord)
        for i in range(max_steps + 1):
            batch_input_images, batch_input_labels = sess.run([train_images_batch, train_labels_batch])
            _, train_loss = sess.run([train_op, loss], feed_dict={
                input_images: batch_input_images,
                input_labels: batch_input_labels,
                keep_prob: 0.8, is_training: True
            })

            # train測試（這里僅僅測試訓練集的一個batch）
            if i % train_log_step == 0:
                train_acc = sess.run(accuracy, feed_dict={
                    input_images: batch_input_images,
                    input_labels: batch_input_labels,
                    keep_prob: 1.0, is_training: False
                })
                print("%s: Step [%d]  train Loss : %f, training accuracy :  %g" % (
                    datetime.now(), i, train_loss, train_acc))

            # val測試（測試全部val數據）
            if i % val_log_step == 0:
                mean_loss, mean_acc = net_evaluation(sess, loss, accuracy,
                                                     val_images_batch, val_labels_batch, val_nums)
                print("%s: Step [%d]  val Loss : %f, val accuracy :  %g" % (
                    datetime.now(), i, mean_loss, mean_acc))

            # 模型保存：每迭代snapshot次或者最后一次保存模型
            if (i % snapshot == 0 and i > 0) or i == max_steps:
                print('------------save:{}--{}'.format(snapshot_prefix, i))
                saver.save(sess, snapshot_prefix, global_step=i)
            # 保存val准確率最高的模型
            if mean_acc > max_acc and mean_acc > 0.7:
                max_acc = mean_acc
                path = os.path.dirname(snapshot_prefix)
                best_models = os.path.join(path, 'best_models_{}_{:.4f}.ckpt'.format(i, max_acc))
                print('------save:{}'.format(best_models))
                saver.save(sess, best_models)

        # 停止所有線程
        coord.request_stop()
        coord.join(threads)


def train(train_record_file,
          train_log_step,
          train_param,
          val_record_file,
          val_log_step,
          labels_nums,
          data_shape,
          snapshot,
          snapshot_prefix):
    '''
    :param train_record_file: 訓練的tfrecord文件
    :param train_log_step: 顯示訓練過程log信息間隔
    :param train_param: train參數
    :param val_record_file: 驗證的tfrecord文件
    :param val_log_step: 顯示驗證過程log信息間隔
    :param val_param: val參數
    :param labels_nums: labels數
    :param data_shape: 輸入數據shape
    :param snapshot: 保存模型間隔
    :param snapshot_prefix: 保存模型文件的前綴名
    :return:
    '''
    [base_lr, max_steps] = train_param
    [batch_size, resize_height, resize_width, depths] = data_shape

    # 獲得訓練和測試的樣本數
    train_nums = get_example_nums(train_record_file)
    val_nums = get_example_nums(val_record_file)
    print("train nums : %d, val nums: %d" % (train_nums, val_nums))
    # train nums : 6120, val nums: 3978

    # 從record中讀取圖片和labels數據
    # train數據，訓練數據一般要求打亂順序 shuffle=True
    train_images, train_labels = read_records(train_record_file, resize_height, resize_width,
                                              type='normalization')
    train_images_batch, train_labels_batch = get_batch_images(train_images, train_labels,
                                                              batch_size=batch_size,
                                                              labels_nums=labels_nums,
                                                              one_hot=True, shuffle=True)
    # val數據，驗證數據可以不需要打亂數據
    val_images, val_labels = read_records(val_record_file, resize_height, resize_width,
                                          type='normalization')
    val_images_batch, val_labels_batch = get_batch_images(val_images, val_labels,
                                                          batch_size=batch_size,
                                                          labels_nums=labels_nums,
                                                          one_hot=True, shuffle=False)

    # Define the model
    with slim.arg_scope(lenet.lenet_arg_scope(weight_decay=0.0)):
        out, end_points = lenet.lenet(images=input_images,
                                      num_classes=labels_nums,
                                      is_training=is_training,
                                      dropout_keep_prob=keep_prob,
                                      prediction_fn=slim.softmax,
                                      scope='LeNet')

    # Specify the loss function : tf.losses定義的loss函數都會自動添加到loss函數，不需要add_loss()
    tf.losses.softmax_cross_entropy(onehot_labels=input_labels, logits=out)  # 添加交叉熵損失loss=1.6
    # slim.losses.add_loss(my_loss)
    # 添加正則加損失loss=2.2
    loss = tf.losses.get_total_loss(add_regularization_losses=True)

    # Specify the optimization scheme
    # 在定義訓練的時候，注意到我們使用了batch_norm層時，需要更新每一層的average和variance參數
    # 更新的過程不包含在正常的訓練過程中，需要我們手動的像下面這樣更新
    # 通過tf.get_collection 獲得所有需要更新的op
    update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
    # 使用 TensorFlow的控制流，先執行更新算子，再執行訓練
    with tf.control_dependencies(update_ops):
        print('update_ops:{}'.format(update_ops))
        # create_train_op that ensuers that when we evaluate it to get the loss
        # the update_ops are done and the gradient updates are computed
        # 是否改變優化方案，改變后可以適當的對模型精度提升
        # train_op = tf.train.MomentumOptimizer(learning_rate=base_lr, momentum=0.9).minimize(loss)
        train_op = tf.train.AdadeltaOptimizer(learning_rate=base_lr).minimize(loss)

    accuracy = tf.reduce_mean(tf.cast(tf.equal(tf.argmax(out, 1),
                                               tf.argmax(input_labels, 1)),
                                      tf.float32))

    # 循環迭代過程
    step_train(train_op=train_op, loss=loss, accuracy=accuracy,
               train_images_batch=train_images_batch,
               train_labels_batch=train_labels_batch,
               train_nums=train_nums,
               train_log_step=train_log_step,
               val_images_batch=val_images_batch,
               val_labels_batch=val_labels_batch,
               val_nums=val_nums,
               val_log_step=val_log_step,
               snapshot_prefix=snapshot_prefix,
               snapshot=snapshot)


if __name__ == '__main__':
    train_record_file = 'NewOCRdataset/record/train28.tfrecords'
    val_record_file = 'NewOCRdataset/record/val28.tfrecords'

    train_log_step = 100
    base_lr = 0.001  # 學習率 
    max_steps = 100000  # 迭代次數
    train_param = [base_lr, max_steps]

    val_log_step = 500
    snapshot = 2000  # 保存文件間隔
    if not os.path.exists('modelslenet'):
        os.mkdir('modelslenet')
    snapshot_prefix = 'modelslenet/model.ckpt'
    train(train_record_file=train_record_file,
          train_log_step=train_log_step,
          train_param=train_param,
          val_record_file=val_record_file,
          val_log_step=val_log_step,
          labels_nums=labels_nums,
          data_shape=data_shape,
          snapshot=snapshot,
          snapshot_prefix=snapshot_prefix)

　　這里結果就展示點吧，效果不是很理想。所以放棄了Lenet-5模型。

2019-12-05 12:25:32.825701: Step [90100]  train Loss : 1.667434, training accuracy :  0.625
2019-12-05 12:25:40.093754: Step [90200]  train Loss : 2.222520, training accuracy :  0.5625
2019-12-05 12:25:47.339653: Step [90300]  train Loss : 1.856835, training accuracy :  0.625
2019-12-05 12:25:54.696472: Step [90400]  train Loss : 2.095065, training accuracy :  0.5625
2019-12-05 12:26:02.085547: Step [90500]  train Loss : 1.520860, training accuracy :  0.4375
2019-12-05 12:26:18.967219: Step [90500]  val Loss : 1.778432, val accuracy :  0.500504
2019-12-05 12:26:26.168373: Step [90600]  train Loss : 1.707498, training accuracy :  0.5
2019-12-05 12:26:33.473946: Step [90700]  train Loss : 1.377500, training accuracy :  0.625
2019-12-05 12:26:40.770281: Step [90800]  train Loss : 1.829581, training accuracy :  0.3125
2019-12-05 12:26:48.130398: Step [90900]  train Loss : 1.638364, training accuracy :  0.5625
2019-12-05 12:26:55.445982: Step [91000]  train Loss : 2.001791, training accuracy :  0.5
2019-12-05 12:27:12.287448: Step [91000]  val Loss : 1.776911, val accuracy :  0.488407
2019-12-05 12:27:19.420265: Step [91100]  train Loss : 0.916756, training accuracy :  0.625
2019-12-05 12:27:26.596468: Step [91200]  train Loss : 1.587896, training accuracy :  0.5
2019-12-05 12:27:33.850329: Step [91300]  train Loss : 1.035284, training accuracy :  0.75
2019-12-05 12:27:41.145560: Step [91400]  train Loss : 2.336789, training accuracy :  0.5625
2019-12-05 12:27:48.426554: Step [91500]  train Loss : 1.717655, training accuracy :  0.375
2019-12-05 12:28:05.334864: Step [91500]  val Loss : 1.772030, val accuracy :  0.500252
2019-12-05 12:28:12.469763: Step [91600]  train Loss : 1.584374, training accuracy :  0.5
2019-12-05 12:28:19.762539: Step [91700]  train Loss : 1.466176, training accuracy :  0.625
2019-12-05 12:28:27.011968: Step [91800]  train Loss : 1.744376, training accuracy :  0.5625
2019-12-05 12:28:34.218536: Step [91900]  train Loss : 1.227141, training accuracy :  0.625
2019-12-05 12:28:41.620515: Step [92000]  train Loss : 1.566387, training accuracy :  0.3125
2019-12-05 12:28:58.556259: Step [92000]  val Loss : 1.791969, val accuracy :  0.5
------------save:mymodelslenet/model.ckpt--92000
2019-12-05 12:29:05.803047: Step [92100]  train Loss : 1.478750, training accuracy :  0.375
2019-12-05 12:29:13.062233: Step [92200]  train Loss : 1.355619, training accuracy :  0.5
2019-12-05 12:29:20.247134: Step [92300]  train Loss : 2.353539, training accuracy :  0.4375
2019-12-05 12:29:27.534936: Step [92400]  train Loss : 1.222931, training accuracy :  0.625
2019-12-05 12:29:34.948421: Step [92500]  train Loss : 1.148062, training accuracy :  0.5625
2019-12-05 12:29:51.862244: Step [92500]  val Loss : 1.774340, val accuracy :  0.502016
2019-12-05 12:29:59.027918: Step [92600]  train Loss : 1.045481, training accuracy :  0.75
2019-12-05 12:30:06.393487: Step [92700]  train Loss : 1.258873, training accuracy :  0.625
2019-12-05 12:30:13.601435: Step [92800]  train Loss : 1.582610, training accuracy :  0.375
2019-12-05 12:30:20.928894: Step [92900]  train Loss : 1.717821, training accuracy :  0.6875
2019-12-05 12:30:28.184054: Step [93000]  train Loss : 1.723401, training accuracy :  0.625
2019-12-05 12:30:45.144546: Step [93000]  val Loss : 1.730723, val accuracy :  0.510081
2019-12-05 12:30:52.356409: Step [93100]  train Loss : 1.926704, training accuracy :  0.5625
2019-12-05 12:30:59.587315: Step [93200]  train Loss : 1.445692, training accuracy :  0.6875
2019-12-05 12:31:06.923380: Step [93300]  train Loss : 1.933412, training accuracy :  0.625
2019-12-05 12:31:14.279226: Step [93400]  train Loss : 1.961663, training accuracy :  0.4375
2019-12-05 12:31:21.532947: Step [93500]  train Loss : 1.296774, training accuracy :  0.75
2019-12-05 12:31:38.626041: Step [93500]  val Loss : 1.752177, val accuracy :  0.50378
2019-12-05 12:31:45.816855: Step [93600]  train Loss : 1.182588, training accuracy :  0.5
2019-12-05 12:31:53.085064: Step [93700]  train Loss : 1.512232, training accuracy :  0.6875
2019-12-05 12:32:00.468827: Step [93800]  train Loss : 0.875019, training accuracy :  0.6875
2019-12-05 12:32:07.780115: Step [93900]  train Loss : 1.275869, training accuracy :  0.625
2019-12-05 12:32:15.071957: Step [94000]  train Loss : 1.543662, training accuracy :  0.625
2019-12-05 12:32:32.318524: Step [94000]  val Loss : 1.726128, val accuracy :  0.512853
------------save:mymodelslenet/model.ckpt--94000
2019-12-05 12:32:39.468931: Step [94100]  train Loss : 1.062788, training accuracy :  0.8125
2019-12-05 12:32:46.736611: Step [94200]  train Loss : 1.787313, training accuracy :  0.5
2019-12-05 12:32:54.031457: Step [94300]  train Loss : 1.186714, training accuracy :  0.625
2019-12-05 12:33:01.426793: Step [94400]  train Loss : 1.544687, training accuracy :  0.5625
2019-12-05 12:33:08.624096: Step [94500]  train Loss : 1.322899, training accuracy :  0.8125
2019-12-05 12:33:25.620912: Step [94500]  val Loss : 1.726196, val accuracy :  0.519153
2019-12-05 12:33:32.794832: Step [94600]  train Loss : 2.151494, training accuracy :  0.5625
2019-12-05 12:33:40.046742: Step [94700]  train Loss : 1.678557, training accuracy :  0.5625
2019-12-05 12:33:47.290850: Step [94800]  train Loss : 1.842036, training accuracy :  0.25
2019-12-05 12:33:54.702479: Step [94900]  train Loss : 0.905821, training accuracy :  0.625
2019-12-05 12:34:02.024179: Step [95000]  train Loss : 1.969129, training accuracy :  0.4375
2019-12-05 12:34:19.092089: Step [95000]  val Loss : 1.730408, val accuracy :  0.490927
2019-12-05 12:34:26.311982: Step [95100]  train Loss : 1.550606, training accuracy :  0.5
2019-12-05 12:34:33.544530: Step [95200]  train Loss : 1.448371, training accuracy :  0.6875
2019-12-05 12:34:40.829789: Step [95300]  train Loss : 1.356851, training accuracy :  0.625
2019-12-05 12:34:48.111609: Step [95400]  train Loss : 1.838099, training accuracy :  0.4375
2019-12-05 12:34:55.371314: Step [95500]  train Loss : 1.696585, training accuracy :  0.5625
2019-12-05 12:35:12.379847: Step [95500]  val Loss : 1.685892, val accuracy :  0.52999
2019-12-05 12:35:19.393994: Step [95600]  train Loss : 1.275214, training accuracy :  0.625
2019-12-05 12:35:26.592804: Step [95700]  train Loss : 1.552697, training accuracy :  0.5625
2019-12-05 12:35:33.791779: Step [95800]  train Loss : 1.835150, training accuracy :  0.4375
2019-12-05 12:35:41.090170: Step [95900]  train Loss : 1.301018, training accuracy :  0.4375
2019-12-05 12:35:48.413445: Step [96000]  train Loss : 1.930672, training accuracy :  0.625
2019-12-05 12:36:05.248309: Step [96000]  val Loss : 1.695277, val accuracy :  0.518397
------------save:mymodelslenet/model.ckpt--96000
2019-12-05 12:36:12.443342: Step [96100]  train Loss : 1.927574, training accuracy :  0.5625
2019-12-05 12:36:19.681978: Step [96200]  train Loss : 1.431527, training accuracy :  0.5
2019-12-05 12:36:27.045711: Step [96300]  train Loss : 1.635297, training accuracy :  0.5
2019-12-05 12:36:34.389801: Step [96400]  train Loss : 1.687902, training accuracy :  0.4375
2019-12-05 12:36:41.629381: Step [96500]  train Loss : 1.419575, training accuracy :  0.5625
2019-12-05 12:36:58.706836: Step [96500]  val Loss : 1.715443, val accuracy :  0.514617
2019-12-05 12:37:05.938670: Step [96600]  train Loss : 1.536647, training accuracy :  0.375
2019-12-05 12:37:13.301357: Step [96700]  train Loss : 1.238835, training accuracy :  0.625
2019-12-05 12:37:20.592203: Step [96800]  train Loss : 1.266527, training accuracy :  0.5625
2019-12-05 12:37:27.934468: Step [96900]  train Loss : 1.691167, training accuracy :  0.6875
2019-12-05 12:37:35.309330: Step [97000]  train Loss : 2.081594, training accuracy :  0.375
2019-12-05 12:37:52.024257: Step [97000]  val Loss : 1.711750, val accuracy :  0.513609
2019-12-05 12:37:59.349490: Step [97100]  train Loss : 1.534474, training accuracy :  0.75
2019-12-05 12:38:06.693443: Step [97200]  train Loss : 1.625568, training accuracy :  0.5625
2019-12-05 12:38:13.966219: Step [97300]  train Loss : 1.221727, training accuracy :  0.4375
2019-12-05 12:38:21.219683: Step [97400]  train Loss : 1.358784, training accuracy :  0.6875
2019-12-05 12:38:28.494533: Step [97500]  train Loss : 1.037435, training accuracy :  0.75
2019-12-05 12:38:45.459774: Step [97500]  val Loss : 1.705711, val accuracy :  0.510081
2019-12-05 12:38:52.731272: Step [97600]  train Loss : 1.877534, training accuracy :  0.625
2019-12-05 12:38:59.991482: Step [97700]  train Loss : 1.321345, training accuracy :  0.6875
2019-12-05 12:39:07.301566: Step [97800]  train Loss : 1.329551, training accuracy :  0.3125
2019-12-05 12:39:14.515567: Step [97900]  train Loss : 1.898095, training accuracy :  0.375
2019-12-05 12:39:21.768601: Step [98000]  train Loss : 1.913123, training accuracy :  0.3125
2019-12-05 12:39:38.743417: Step [98000]  val Loss : 1.692561, val accuracy :  0.519405
------------save:mymodelslenet/model.ckpt--98000
2019-12-05 12:39:46.101140: Step [98100]  train Loss : 1.209852, training accuracy :  0.5625
2019-12-05 12:39:53.435749: Step [98200]  train Loss : 1.233232, training accuracy :  0.75
2019-12-05 12:40:00.776940: Step [98300]  train Loss : 1.467005, training accuracy :  0.5625
2019-12-05 12:40:08.027132: Step [98400]  train Loss : 1.147114, training accuracy :  0.625
2019-12-05 12:40:15.285183: Step [98500]  train Loss : 0.794502, training accuracy :  0.6875
2019-12-05 12:40:32.167491: Step [98500]  val Loss : 1.725495, val accuracy :  0.513861
2019-12-05 12:40:39.370767: Step [98600]  train Loss : 1.257332, training accuracy :  0.75
2019-12-05 12:40:46.653981: Step [98700]  train Loss : 1.566149, training accuracy :  0.5625
2019-12-05 12:40:53.827023: Step [98800]  train Loss : 1.465775, training accuracy :  0.6875
2019-12-05 12:41:01.112937: Step [98900]  train Loss : 1.453553, training accuracy :  0.5625
2019-12-05 12:41:08.475515: Step [99000]  train Loss : 1.633591, training accuracy :  0.5
2019-12-05 12:41:25.320005: Step [99000]  val Loss : 1.686813, val accuracy :  0.525454
2019-12-05 12:41:32.512437: Step [99100]  train Loss : 2.085576, training accuracy :  0.5
2019-12-05 12:41:39.877570: Step [99200]  train Loss : 1.099397, training accuracy :  0.5625
2019-12-05 12:41:47.156825: Step [99300]  train Loss : 0.851584, training accuracy :  0.75
2019-12-05 12:41:54.393155: Step [99400]  train Loss : 1.380483, training accuracy :  0.5625
2019-12-05 12:42:01.674840: Step [99500]  train Loss : 1.607050, training accuracy :  0.5
2019-12-05 12:42:18.704520: Step [99500]  val Loss : 1.660345, val accuracy :  0.524446
2019-12-05 12:42:25.987405: Step [99600]  train Loss : 0.604428, training accuracy :  0.6875
2019-12-05 12:42:33.089198: Step [99700]  train Loss : 1.789382, training accuracy :  0.5625
2019-12-05 12:42:40.334938: Step [99800]  train Loss : 1.493351, training accuracy :  0.625
2019-12-05 12:42:47.615859: Step [99900]  train Loss : 1.497630, training accuracy :  0.5625
2019-12-05 12:42:54.920376: Step [100000]  train Loss : 1.704501, training accuracy :  0.5
2019-12-05 12:43:11.680941: Step [100000]  val Loss : 1.681928, val accuracy :  0.520413
------------save:mymodelslenet/model.ckpt--100000

　　當然也可以使用上面我們自己定制的 lenet-5模型，只需要修改上面模型即可。更改的關鍵點在這里：

 

 　　更改完之后，就可以使用自己定制的模型進行訓練了。

 

 

 

https://blog.csdn.net/Choose_lzn/article/details/88627769

https://www.cnblogs.com/DOMLX/p/8954892.html

https://blog.csdn.net/chai_zheng/article/details/79751020