對於2個變量的樣本回歸分析，L2和L1正則化基本相同，僅僅正則化項不同
LASSO回歸為在損失函數加入\(||\omega||_1\) ,\(\omega\) 的1范數 而 嶺回歸為\(||\omega||_2^2\)，\(\omega\) 的2范數
*矩陣、向量范數
*L1正則化（嶺回歸）
LASSO Regression
Loss Function
\[J(\omega)= (X \omega - Y)^T(X \omega - Y) + \lambda ||\omega||_1 \]
\(||\omega||_1\)導數不連續，采用坐標下降法求\(\omega\)
坐標下降法推導過程
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
M = 3 #變量個數+1 變量加 偏移項b， 一個3個參數
N = 50 #樣本個數
#隨機生成兩個屬性的N個樣本
feature1 = np.random.rand(N)*10
feature2 = np.random.rand(N)*10
splt = np.ones((1, N))
#
temp_X1 = np.row_stack((feature1, feature2))
temp_X = np.vstack((temp_X1, splt))
X_t = np.mat(temp_X)
X = X_t.T
temp_Y = np.random.rand(N)*10
Y_t = np.mat(temp_Y)
Y = Y_t.T
#畫樣本散點圖
fig = plt.figure()
ax1 = Axes3D(fig)
ax1.scatter(feature1, feature2, temp_Y)
######
def errors(X, Y, Omega) :
err = (X*Omega - Y).T*(X*Omega - Y)
return err
#坐標下降算法
def lasso_regression(X, Y, lambd, threshold):
#
Omega = np.mat(np.zeros((M, 1)))
err = errors(X, Y, Omega)
counts = 0 #統計迭代次數
# 使用坐標下降法優化回歸系數Omega
while err > threshold:
counts += 1
for k in range(M):
# 計算常量值z_k和p_k
z_k = (X[:, k].T*X[:, k])[0, 0]
p_k = 0
for i in range(N):
p_k += X[i, k]*(Y[i, 0] - sum([X[i, j]*Omega[j, 0] for j in range(M) if j != k]))
if p_k < -lambd/2:
w_k = (p_k + lambd/2)/z_k
elif p_k > lambd/2:
w_k = (p_k - lambd/2)/z_k
else:
w_k = 0
Omega[k, 0] = w_k
err_prime = errors(X, Y, Omega)
delta = abs(err_prime - err)[0, 0]
err = err_prime
print('Iteration: {}, delta = {}'.format(counts, delta))
if delta < threshold:
break
return Omega
#求Omega
lambd = 10.0
threshold = 0.1
Omega = lasso_regression(X, Y, lambd, threshold)
#畫分回歸平面
xx = np.linspace(0,10, num=50)
yy = np.linspace(0,10, num=50)
xx_1, yy_1 = np.meshgrid(xx, yy)
Omega_h = np.array(Omega.T)
zz_1 = Omega_h[0, 0]*xx_1 + Omega_h[0, 1]*yy_1 + Omega_h[0, 2]
ax1.plot_surface(xx_1, yy_1, zz_1, alpha= 0.6, color= "r")
plt.show()
效果
