一、RNN
全稱為Recurrent Neural Network,意為循環神經網絡，用於處理序列數據。
序列數據是指在不同時間點上收集到的數據，反映了某一事物、現象等隨時間的變化狀態或程度。即數據之間有聯系。
RNN的特點：1，,層間神經元也有連接（主要為隱層）；2，共享參數
其結構如上圖所示，數據為順序處理，在處理長序列數據時，極易導致梯度消失問題。
二、LSTM
LSTM為長短期記憶，是一種變種的RNN，在RNN的基礎上引入了細胞狀態，根據細胞狀態可決定哪些狀態應該保留下來，哪些狀態應該被遺忘。
LSTM可一定程度上解決梯度消失問題。
由上圖可知，在RNN的基礎上，增加了一路輸入和輸出，增加的這一路就是細胞狀態。
由上一時刻的輸出和當前時刻的輸入，經過sigmod函數之后，趨近於0被遺忘的多，趨近於1被遺忘的少。
由上一時刻的輸出和當前時刻的輸入，經過sigmod函數之后，決定哪些內容應該被記住，被記住的內容並不是上一時刻的輸出和當前時刻的輸入，而是需要經過tanh函數。
程序：應用LSTM訓練mnist數據集
import os import torch import torch.nn as nn import torch.utils.data as Data from torch.autograd import Variable import torchvision.datasets as dsets import matplotlib.pyplot as plt import torchvision.transforms as transforms # torch.manual_seed(1) # reproducible # Hyper Parameters EPOCH = 1 # train the training data n times, to save time, we just train 1 epoch BATCH_SIZE = 64 LR = 0.01 # learning rate DOWNLOAD_MNIST = False #已下載好數據集，就設置為False，否則為TRUE TIME_STEP=28 #可理解為輸入圖像維度 INPUT_SIZE=28 # Mnist digits dataset if not(os.path.exists('./mnist/')) or not os.listdir('./mnist/'): # not mnist dir or mnist is empyt dir DOWNLOAD_MNIST = True train_data = dsets.MNIST( root='./mnist/', train=True, # this is training data transform=transforms.ToTensor(), # Converts a PIL.Image or numpy.ndarray to # torch.FloatTensor of shape (C x H x W) and normalize in the range [0.0, 1.0] download=DOWNLOAD_MNIST, ) # plot one example # print(train_data.train_data.size()) # (60000, 28, 28) # print(train_data.train_labels.size()) # (60000) # plt.imshow(train_data.train_data[0].numpy(), cmap='gray') # plt.title('%i' % train_data.train_labels[0]) # plt.show() # Data Loader for easy mini-batch return in training, the image batch shape will be (50, 1, 28, 28) train_loader = Data.DataLoader(dataset=train_data, batch_size=BATCH_SIZE, shuffle=True) # pick 2000 samples to speed up testing test_data = dsets.MNIST(root='./mnist/', train=False,transform=transforms.ToTensor()) test_x = test_data.test_data.type(torch.FloatTensor)[:2000]/255. # shape from (2000, 28, 28) to (2000, 1, 28, 28), value in range(0,1) test_y = test_data.test_labels.numpy()[:2000] class RNN(nn.Module): def __init__(self): super(RNN, self).__init__() self.rnn = nn.LSTM( input_size=INPUT_SIZE, hidden_size=64, num_layers=1, batch_first=True ) self.out=nn.Linear(64,10) def forward(self,x): r_out,(h_n,h_c)=self.rnn(x,None) out=self.out(r_out[:,-1,:]) #數據格式為[batch,time_step,input],因此輸出參考的是最后時刻的數據 return out rnn=RNN() print(rnn) # net architecture optimizer = torch.optim.Adam(rnn.parameters(), lr=LR) # optimize all cnn parameters loss_func = nn.CrossEntropyLoss() # the target label is not one-hotted for epoch in range(EPOCH): for step, (x, y) in enumerate(train_loader): # gives batch data, normalize x when iterate train_loader b_x=Variable(x.view(-1,28,28)) b_y=Variable(y) output = rnn(b_x) # cnn output loss = loss_func(output, b_y) # cross entropy loss optimizer.zero_grad() # clear gradients for this training step loss.backward() # backpropagation, compute gradients optimizer.step() # apply gradients if step % 50 == 0: test_output = rnn(test_x) pred_y = torch.max(test_output, 1)[1].data.numpy().squeeze() accuracy =float((pred_y==test_y).astype(int).sum())/float(test_y.size) print('Epoch: ', epoch, '| train loss: %.4f' % loss.data.numpy(), '| test accuracy: %.2f' % accuracy) # print 10 predictions from test data test_output = rnn(test_x[:10].view(-1,28,28)) pred_y = torch.max(test_output, 1)[1].data.numpy().squeeze() print(pred_y, 'prediction number') print(test_y[:10], 'real number')
運行結果為：
