CS231N Assignment4 Two Layer Net
Begin
本文主要介紹CS231N系列課程的第四項作業，寫一個兩層神經網絡訓練模型。
課程主頁：網易雲課堂CS231N系列課程
語言：Python3.6
1神經網絡
神經網絡理解起來比較簡單，在線形分類器的基礎上加一個非線性激活函數，使其可以表示非線性含義，再增加
多層分類器就成為多層神經網絡，如下圖所示，由輸入X經過第一層計算得到W1X，在后再用隱含層的激活函數max(0,s)
得到隱含層的輸出。到輸出層乘以W2得到輸出層，最后的分類計分。
下圖中最左側為3072代表每幅圖像有3072個特征，經過第一層網絡到達中間層叫做隱藏層，隱含層變為100個特征了，在經過第二層計算到輸出層最終得到10個類的得分。此神經網絡叫做兩層的神經網絡（包含W1、W2）也叫有一個隱含層的神經網絡。
對於激活函數只有在隱含層計算時有激活函數。
對於激活函數，有很多種，如下所示，上述中我們采用的是RELU
2編寫一個兩層神經網絡
類似於之前我們書寫的SVM等，編寫任何一個訓練器需要包含以下幾部分
1、LOSS損失函數（前向傳播）與梯度（后向傳播）計算
2、訓練函數
3、預測函數
4、參數訓練
2.1 loss函數
損失函數計算采用softmaxu損失方法
1、首先計算前向傳輸，計算分數，就是上面那三個公式的調用
##############################
#Computing the class scores of the input
##############################
Z1 = X.dot(W1) + b1#第一層
S1 = np.maximum(0,Z1)#隱藏層激活函數
score = S1.dot(W2) + b2#輸出層
2、計算完之后，插入一句話，當沒有y參數時，直接輸出分數，主要用在計算預測函數時需要計算分數。
if Y is None:
return score
loss = None
3、之后計算損失softmax計算，具體計算可以參考我的作業3
###############################
#TODO:forward pass
#computing the loss of the net
################################
exp_scores = np.exp(score)
probs = exp_scores / np.sum(exp_scores,axis=1,keepdims=True)
#數據損失
data_loss = -1.0/ N * np.log(probs[np.arange(N),Y]).sum()
#正則損失
reg_loss = 0.5*reg*(np.sum(W1*W1) + np.sum(W2*W2))
#總損失
loss = data_loss + reg_loss
4、計算后向傳播梯度
################################
#TODO:backward pass
#computing the gradient
################################
grads = {}
dscores = probs
dscores[np.arange(N),Y] -= 1
dscores /= N
#更新W2B2
grads['W2'] = S1.T.dot(dscores) + reg *W2
grads['b2'] = np.sum(dscores,axis = 0)
#第二層
dhidden = dscores.dot(W2.T)
dhidden[S1<=0] = 0
grads['W1'] = X.T.dot(dhidden) + reg *W1
grads['b1'] = np.sum(dhidden,axis = 0)
代碼如下：
def loss(self,X,Y=None,reg=0.0):
'''
計算損失函數
'''
W1, b1 = self.params['W1'], self.params['b1']
W2, b2 = self.params['W2'], self.params['b2']
N, D = X.shape
##############################
#Computing the class scores of the input
##############################
Z1 = X.dot(W1) + b1#第一層
S1 = np.maximum(0,Z1)#隱藏層激活函數
score = S1.dot(W2) + b2#輸出層
if Y is None:
return score
loss = None
###############################
#TODO:forward pass
#computing the loss of the net
################################
exp_scores = np.exp(score)
probs = exp_scores / np.sum(exp_scores,axis=1,keepdims=True)
#數據損失
data_loss = -1.0/ N * np.log(probs[np.arange(N),Y]).sum()
#正則損失
reg_loss = 0.5*reg*(np.sum(W1*W1) + np.sum(W2*W2))
#總損失
loss = data_loss + reg_loss
################################
#TODO:backward pass
#computing the gradient
################################
grads = {}
dscores = probs
dscores[np.arange(N),Y] -= 1
dscores /= N
#更新W2B2
grads['W2'] = S1.T.dot(dscores) + reg *W2
grads['b2'] = np.sum(dscores,axis = 0)
#第二層
dhidden = dscores.dot(W2.T)
dhidden[S1<=0] = 0
grads['W1'] = X.T.dot(dhidden) + reg *W1
grads['b1'] = np.sum(dhidden,axis = 0)
return loss,grads
2.2 訓練函數
訓練參數依然是
學習率learning_rate
正則系數reg
訓練步數num_iters
每次訓練的采樣數量batch_size
1、進入循環中，首先采樣一定數據，batch_inx = np.random.choice(num_train, batch_size)
代表從0-》num_train中隨機產生batch_size 個數，這些數據其實反應這采樣樣本的索引
值，然后我們用X_batch = X[batch_inx,:]可以獲取到該索引所對應的數據
for it in range(num_iters):
X_batch = None
y_batch = None
#########################################################################
# TODO: Create a random minibatch of training data and labels, storing #
# them in X_batch and y_batch respectively. #
#########################################################################
batch_inx = np.random.choice(num_train, batch_size)
X_batch = X[batch_inx,:]
y_batch = y[batch_inx]
2、取樣數據后需要計算損失值和梯度。
# Compute loss and gradients using the current minibatch
loss, grads = self.loss(X_batch, Y=y_batch, reg=reg)
loss_history.append(loss)
3、計算完損失之后，需要根據梯度值去更新參數W1、W2、b1、b2。
梯度反映着它的最大變化方向，如果梯度是正的表示增長，我們應該反方向去調控，所以在其基礎上
減去學習率乘以梯度值。
#########################################################################
# TODO: Use the gradients in the grads dictionary to update the #
# parameters of the network (stored in the dictionary self.params) #
# using stochastic gradient descent. You'll need to use the gradients #
# stored in the grads dictionary defined above. #
#########################################################################
self.params['W1'] -= learning_rate * grads['W1']
self.params['b1'] -= learning_rate * grads['b1']
self.params['W2'] -= learning_rate * grads['W2']
self.params['b2'] -= learning_rate * grads['b2']
4、實時驗證
在神經網絡訓練中我們加入一個實時驗證，沒訓練一次，我們比較以下訓練集與預測值的真實程度，
驗證集與預測值的真實程度。在最后時可以將這條曲線繪制觀測一下。
# Every epoch, check train and val accuracy and decay learning rate.
if it % iterations_per_epoch == 0:
# Check accuracy
train_acc = (self.predict(X_batch) == y_batch).mean()
val_acc = (self.predict(X_val) == y_val).mean()
train_acc_history.append(train_acc)
val_acc_history.append(val_acc)
# Decay learning rate
learning_rate *= learning_rate_decay
最終總代碼如下所示：
def train(self, X, y, X_val, y_val,
learning_rate=1e-3, learning_rate_decay=0.95,
reg=1e-5, num_iters=100,
batch_size=200, verbose=False):
"""
Train this neural network using stochastic gradient descent.
Inputs:
- X: A numpy array of shape (N, D) giving training data.
- y: A numpy array f shape (N,) giving training labels; y[i] = c means that
X[i] has label c, where 0 <= c < C.
- X_val: A numpy array of shape (N_val, D) giving validation data.
- y_val: A numpy array of shape (N_val,) giving validation labels.
- learning_rate: Scalar giving learning rate for optimization.
- learning_rate_decay: Scalar giving factor used to decay the learning rate
after each epoch.
- reg: Scalar giving regularization strength.
- num_iters: Number of steps to take when optimizing.
- batch_size: Number of training examples to use per step.
- verbose: boolean; if true print progress during optimization.
"""
self.hyper_params = {}
self.hyper_params['learning_rate'] = learning_rate
self.hyper_params['reg'] = reg
self.hyper_params['batch_size'] = batch_size
self.hyper_params['hidden_size'] = self.params['W1'].shape[1]
self.hyper_params['num_iter'] = num_iters
num_train = X.shape[0]
iterations_per_epoch = max(num_train / batch_size, 1)
# Use SGD to optimize the parameters in self.model
loss_history = []
train_acc_history = []
val_acc_history = []
for it in range(num_iters):
X_batch = None
y_batch = None
#########################################################################
# TODO: Create a random minibatch of training data and labels, storing #
# them in X_batch and y_batch respectively. #
#########################################################################
batch_inx = np.random.choice(num_train, batch_size)
X_batch = X[batch_inx,:]
y_batch = y[batch_inx]
#########################################################################
# END OF YOUR CODE #
#########################################################################
# Compute loss and gradients using the current minibatch
loss, grads = self.loss(X_batch, Y=y_batch, reg=reg)
loss_history.append(loss)
#########################################################################
# TODO: Use the gradients in the grads dictionary to update the #
# parameters of the network (stored in the dictionary self.params) #
# using stochastic gradient descent. You'll need to use the gradients #
# stored in the grads dictionary defined above. #
#########################################################################
self.params['W1'] -= learning_rate * grads['W1']
self.params['b1'] -= learning_rate * grads['b1']
self.params['W2'] -= learning_rate * grads['W2']
self.params['b2'] -= learning_rate * grads['b2']
#########################################################################
# END OF YOUR CODE #
#########################################################################
if verbose and it % 100 == 0:
print ('iteration %d / %d: loss %f' % (it, num_iters, loss))
# Every epoch, check train and val accuracy and decay learning rate.
if it % iterations_per_epoch == 0:
# Check accuracy
train_acc = (self.predict(X_batch) == y_batch).mean()
val_acc = (self.predict(X_val) == y_val).mean()
train_acc_history.append(train_acc)
val_acc_history.append(val_acc)
# Decay learning rate
learning_rate *= learning_rate_decay
return {
'loss_history': loss_history,
'train_acc_history': train_acc_history,
'val_acc_history': val_acc_history,
}
訓練時間可能稍微較長，等待一段時間后可以看到如下結果
2.3 predict函數
預測和之前類似，將數據帶入損失，找分數最大值即可
def predict(self, X):
y_pred = None
scores = self.loss(X)
y_pred = np.argmax(scores, axis=1)
return y_pred
訓練結果如下所示
2.4 可視化結果
訓練完之后我們可以進行可視化觀察，我們把訓練時的loss顯示出來，還有實時比較的誤差拿出來看看。
測試代碼如下：
#step1 數據裁剪
#數據量太大，我們重新整理數據，提取一部分訓練數據、測試數據、驗證數據
num_training = 49000#訓練集數量
num_validation = 1000#驗證集數量
num_test = 1000#測試集數量
num_dev = 500
Data = load_CIFAR10()
CIFAR10_Data = './'
X_train,Y_train,X_test,Y_test = Data.load_CIFAR10(CIFAR10_Data)#load the data
#從訓練集中截取一部分數據作為驗證集
mask = range(num_training,num_training + num_validation)
X_val = X_train[mask]
Y_val = Y_train[mask]
#訓練集前一部分數據保存為訓練集
mask = range(num_training)
X_train = X_train[mask]
Y_train = Y_train[mask]
#訓練集數量太大，我們實驗只要一部分作為開發集
mask = np.random.choice(num_training,num_dev,replace = False)
X_dev = X_train[mask]
Y_dev = Y_train[mask]
#測試集也太大，變小
mask = range(num_test)
X_test = X_test[mask]
Y_test = Y_test[mask]
#step2 數據預處理
#所有數據准變為二位數據，方便處理
X_train = np.reshape(X_train,(X_train.shape[0],-1))
X_val = np.reshape(X_val,(X_val.shape[0],-1))
X_test = np.reshape(X_test,(X_test.shape[0],-1))
X_dev = np.reshape(X_dev,(X_dev.shape[0],-1))
print('Traing data shape', X_train.shape)
print('Validation data shape',X_val.shape)
print('Test data shape',X_test.shape)
print('Dev data shape',X_dev.shape)
#step3訓練數據
input_size = 32*32*3
hidden_size = 50
num_classes = 10
net = TwoLayerNet(input_size,hidden_size,num_classes)
#訓練
sta = net.train(X_train,Y_train,X_val,Y_val,num_iters=1000,batch_size=200,learning_rate=4e-4,learning_rate_decay=0.95,reg=0.7,verbose=True)
#step4預測數據
val = (net.predict(X_val) == Y_val).mean()
print(val)
#step5可視化效果
plt.subplot(2,1,1)
plt.plot(sta['loss_history'])
plt.ylabel('loss')
plt.xlabel('Iteration')
plt.title('Loss_History')
plt.subplot(2,1,2)
plt.plot(sta['train_acc_history'],label = 'train')
plt.plot(sta['val_acc_history'],label = 'val')
plt.xlabel('epoch')
plt.ylabel('Classfication accuracy')
plt.show()
