Task5.PyTorch實現L1，L2正則化以及Dropout

1.了解知道Dropout原理　　

　　深度學習網路中，參數多，可能出現過擬合及費時問題。為了解決這一問題，通過實驗，在2012年，Hinton在其論文《Improving neural networks by preventing co-adaptation of feature detectors》中提出Dropout。證明了其能有效解決過擬合的能力。

dropout 是指在深度學習網絡的訓練過程中，按照一定的概率將一部分神經網絡單元暫時從網絡中丟棄，相當於從原始的網絡中找到一個更瘦的網絡示意圖如下：

　　其實現是以某種概率分布使得一些神經元為0，一些為1.這樣在有N個神經元的神經網絡中，其參數搭配可能有2^N種。
具體介紹 見論文（我也不是很懂 實現得見）
適用情況：
1 Dropout主要用在數據量不夠，容易過擬合，需要dropout。

 

L1及L2可以使得結構化風險最小
其中：
L1的參數具有稀疏性（具有更多的0或1）
L2的參數趨近於分散化 ，其參數值趨向於選擇更簡單（趨於0的參數），因此比較平滑

2.用代碼實現正則化(L1、L2、Dropout）

 

L1范數

 

　　L1范數是參數矩陣W中元素的絕對值之和，L1范數相對於L0范數不同點在於，L0范數求解是NP問題，而L1范數是L0范數的最優凸近似，求解較為容易。L1常被稱為LASSO.

regularization_loss = 0
for param in model.parameters():
    regularization_loss += torch.sum(abs(param))

for epoch in range(EPOCHS):
    y_pred = model(x_train)
    classify_loss = criterion(y_pred, y_train.float().view(-1, 1))
    loss = classify_loss + 0.001 * regularization_loss  # 引入L1正則化項

L2范數

　　L2范數是參數矩陣W中元素的平方之和，這使得參數矩陣中的元素更稀疏，與前兩個范數不同的是，它不會讓參數變為0，而是使得參數大部分都接近於0。L1追求稀疏化，從而丟棄了一部分特征（參數為0），而L2范數只是使參數盡可能為0，保留了特征。L2被稱為Rigde.

 

criterion  = torch.nn.BCELoss() #定義損失函數
optimizer = torch.optim.SGD(model.parameters(),lr = 0.01, momentum=0, dampening=0,weight_decay=0) #weight_decay 表示使用L2正則化

3.Dropout的numpy實現

import numpy as np

X = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1] ])

y = np.array([[0,1,1,0]]).T

alpha,hidden_dim,dropout_percent,do_dropout = (0.5,4,0.2,True)

synapse_0 = 2*np.random.random((3,hidden_dim)) - 1

synapse_1 = 2*np.random.random((hidden_dim,1)) - 1

for j in xrange(60000):

    layer_1 = (1/(1+np.exp(-(np.dot(X,synapse_0)))))

    if(do_dropout):

        layer_1 *= np.random.binomial([np.ones((len(X),hidden_dim))],1-dropout_percent)[0] * (1.0/(1-dropout_percent))

    layer_2 = 1/(1+np.exp(-(np.dot(layer_1,synapse_1))))

    layer_2_delta = (layer_2 - y)*(layer_2*(1-layer_2))

    layer_1_delta = layer_2_delta.dot(synapse_1.T) * (layer_1 * (1-layer_1))

    synapse_1 -= (alpha * layer_1.T.dot(layer_2_delta))

    synapse_0 -= (alpha * X.T.dot(layer_1_delta))

4.完整代碼

 

import torch
from torch import nn
from torch.autograd import Variable
import torch.nn.functional as F
import torch.nn.init as init
import math
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
import numpy as np
import pandas as pd
%matplotlib inline

# 導入數據
data = pd.read_csv(r'C:\Users\betty\Desktop\pytorch學習\data.txt')
x, y = data.ix[:,:8],data.ix[:,-1]

#測試集為30%，訓練集為80%
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=0)

x_train = Variable(torch.from_numpy(np.array(x_train)).float())
y_train = Variable(torch.from_numpy(np.array(y_train).reshape(-1, 1)).float())    

x_test = Variable(torch.from_numpy(np.array(x_test)).float())
y_test= Variable(torch.from_numpy(np.array(y_test).reshape(-1,1)).float())    


print(x_train.data.shape)
print(y_train.data.shape)

print(x_test.data.shape)
print(y_test.data.shape)

class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.l1 = torch.nn.Linear(8, 200)
        self.l2 = torch.nn.Linear(200, 50)
        self.l3 = torch.nn.Linear(50, 1)

    def forward(self, x):
        out1 = F.relu(self.l1(x))
        out2 = F.dropout(out1, p= 0.5)
        out3 = F.relu(self.l2(out2))
        out4 = F.dropout(out3, p=0.5)
        y_pred = F.sigmoid(self.l3(out3))
        return y_pred

model = Model()

criterion = torch.nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=0.1)

Loss=[]
for epoch in range(2000):
        y_pred = model(x_train)
        loss = criterion(y_pred, y_train)
        if epoch % 400 == 0:
            print("epoch =", epoch, "loss", loss.item())
            Loss.append(loss.item())
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

# 模型評估
def label_flag(data):
    for i in range(len(data)):
        if(data[i]>0.5):
            data[i] = 1.0
        else:
            data[i] = 0.0
    return data

y_pred = label_flag(y_pred)  
print(classification_report(y_train.detach().numpy(), y_pred.detach().numpy()))

# 測試
y_test_pred = model(x_test)
y_test_pred = label_flag(y_test_pred)       
print(classification_report(y_test.detach().numpy(), y_test_pred.detach().numpy()))

 

數據集下載鏈接：鏈接：https://pan.baidu.com/s/1LrJktjVQ1OM9mYt_cuE-FQ
提取碼：hatv

 

 

原文鏈接：https://blog.csdn.net/wehung/article/details/89283583