梯度下降法VS隨機梯度下降法 (Python的實現)

# -*- coding: cp936 -*-
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt


# 構造訓練數據
x = np.arange(0., 10., 0.2)
m = len(x)                                      # 訓練數據點數目
x0 = np.full(m, 1.0)                 
input_data = np.vstack([x0, x]).T               # 將偏置b作為權向量的第一個分量
target_data = 2 * x + 5 + np.random.randn(m)


# 兩種終止條件
loop_max = 10000   # 最大迭代次數(防止死循環)
epsilon = 1e-3     

# 初始化權值
np.random.seed(0)
w = np.random.randn(2)
#w = np.zeros(2)

alpha = 0.001      # 步長(注意取值過大會導致振盪,過小收斂速度變慢)
diff = 0.           
error = np.zeros(2) 
count = 0          # 循環次數
finish = 0         # 終止標志
# -------------------------------------------隨機梯度下降算法----------------------------------------------------------
'''
while count < loop_max:
    count += 1

    # 遍歷訓練數據集，不斷更新權值
    for i in range(m):  
        diff = np.dot(w, input_data[i]) - target_data[i]  # 訓練集代入,計算誤差值

        # 采用隨機梯度下降算法,更新一次權值只使用一組訓練數據
        w = w - alpha * diff * input_data[i]

        # ------------------------------終止條件判斷-----------------------------------------
        # 若沒終止，則繼續讀取樣本進行處理，如果所有樣本都讀取完畢了,則循環重新從頭開始讀取樣本進行處理。

    # ----------------------------------終止條件判斷-----------------------------------------
    # 注意：有多種迭代終止條件，和判斷語句的位置。終止判斷可以放在權值向量更新一次后,也可以放在更新m次后。
    if np.linalg.norm(w - error) < epsilon:     # 終止條件：前后兩次計算出的權向量的絕對誤差充分小  
        finish = 1
        break
    else:
        error = w
print 'loop count = %d' % count,  '\tw:[%f, %f]' % (w[0], w[1])
'''


# -----------------------------------------------梯度下降法-----------------------------------------------------------
while count < loop_max:
    count += 1

    # 標准梯度下降是在權值更新前對所有樣例匯總誤差，而隨機梯度下降的權值是通過考查某個訓練樣例來更新的
    # 在標准梯度下降中，權值更新的每一步對多個樣例求和，需要更多的計算
    sum_m = np.zeros(2)
    for i in range(m):
        dif = (np.dot(w, input_data[i]) - target_data[i]) * input_data[i]
        sum_m = sum_m + dif     # 當alpha取值過大時,sum_m會在迭代過程中會溢出

    w = w - alpha * sum_m       # 注意步長alpha的取值,過大會導致振盪
    #w = w - 0.005 * sum_m      # alpha取0.005時產生振盪,需要將alpha調小
    
    # 判斷是否已收斂
    if np.linalg.norm(w - error) < epsilon:
        finish = 1
        break
    else:
        error = w
print 'loop count = %d' % count,  '\tw:[%f, %f]' % (w[0], w[1])

# check with scipy linear regression 
slope, intercept, r_value, p_value, slope_std_error = stats.linregress(x, target_data)
print 'intercept = %s slope = %s' %(intercept, slope)
    
plt.plot(x, target_data, 'k+')
plt.plot(x, w[1] * x + w[0], 'r')
plt.show()

 

 

The Learning Rate

An important consideration is the learning rate µ, which determines by how much we change the weights w at each step. If µ is too small, the algorithm will take a long time to converge .

Conversely, if µ is too large, we may end up bouncing around the error surface out of control - the algorithm diverges. This usually ends with an overflow error in the computer's floating-point arithmetic.

 

Batch vs. Online Learning

Above we have accumulated the gradient contributions for all data points in the training set before updating the weights. This method is often referred to as batch learning. An alternative approach is online learning, where the weights are updated immediately after seeing each data point. Since the gradient for a single data point can be considered a noisy approximation to the overall gradient, this is also called stochastic gradient descent.

Online learning has a number of advantages:

• it is often much faster, especially when the training set is redundant (contains many similar data points),
• it can be used when there is no fixed training set (new data keeps coming in),
• it is better at tracking nonstationary environments (where the best model gradually changes over time),
• the noise in the gradient can help to escape from local minima (which are a problem for gradient descent in nonlinear models)

These advantages are, however, bought at a price: many powerful optimization techniques (such as: conjugate and second-order gradient methods, support vector machines, Bayesian methods, etc.)  are batch methods that cannot be used online.A compromise between batch and online learning is the use of "mini-batches": the weights are updated after every n data points, where n is greater than 1 but smaller than the training set size.

 

 

參考：http://www.tuicool.com/articles/MRbee2i

     https://www.willamette.edu/~gorr/classes/cs449/linear2.html

     http://www.bogotobogo.com/python/python_numpy_batch_gradient_descent_algorithm.php