CS231N作業1 詳細實錄（1）：環境准備+Knn

 

一、准備工作

1. 安裝Ipython Notebook^[4]
2. 設置遠程訪問服務器上Ipython^[2,5],我直接用的[5]中的方法，可以直接在本地瀏覽器上使用服務器上的notebook。
   1. 在工作目錄下輸入命令 jupyter notebook --ip 0.0.0.0
   2. 會輸出一個同token,在瀏覽器上輸入服務器ip和token組成的URL，例如：http://192.168.2.175:8888/tree?token=b63ad6d9c7ee1877bd2bdeb5060ceb9e4e250644a311e948
3. 學習IPython Notebook教程^[7，8]
4. 建立課程作業的anaconda環境^[6]
   • 要設置虛擬環境，請運行（在終端中）conda create -n cs231n python=3.6 anaconda 創建一個叫做的環境cs231n。
   • 要激活並進入環境，請運行 source activate cs231n
   • 要退出，您只需關閉窗口或運行即可 source deactivate cs231n
   • 請注意，每次要處理作業時，都應該運行 source activate cs231n
     
5. 下載數據：

cd cs231n/datasets

./get_datasets.sh

 

 

Q1：k-最近鄰分類器（20分）

 

IPython Notebook knn.ipynb將引導您完成kNN分類器的實現。

 

Step1： 在Jupyter中打開knn.ipnb

出現問題：

Kernel崩潰，如下圖所示，Ipython瀏覽器里報錯：“The kernel has died, and the automatic restart has failed. It is possible the kernel cannot be restarted. If you are not able to restart the kernel, you will still be able to save the notebook, but running code will no longer work until the notebook is reopened.”

 

嘗試用這個方案解決這個問題：《解決Anaconda下“The kernel has died, and the automatic restart has failed.” 的問題》，即conda更新相關庫：

 

conda upgrade notebook
conda upgrade jupyter

 

再次打開 knn.ipynb 時，不再報錯，問題解決。

 

問題描述

 

　　翻譯：

　　完成工作表（包括其輸出和工作表之外的任何支持代碼），並與您的作業一起提交。詳細信息請參閱課程網站上的作業頁面。

　　kNN分類器包括兩個階段：

• • 　　在訓練期間，分類器獲取訓練數據並簡單地記住它
  • 　　在測試期間，kNN通過與所有訓練圖像進行比較並且轉移k個最相似訓練示例的標簽來對每個測試圖像進​​行分類
  • 　　k的值是交叉驗證的

　　在本練習中，您將實現這些步驟並理解基本的圖像分類流程、交叉驗證，並提高編寫高效矢量化代碼的熟練程度。

 

Step 1: 環境參數設置

代碼段1 ：導入工具庫

# Run some setup code for this notebook.

import random
import numpy as np
from cs231n.data_utils import load_CIFAR10
import matplotlib.pyplot as plt

from __future__ import print_function

 

理解：

• matplotlib是一個 Python 的 2D繪圖庫。通過 Matplotlib，開發者可以僅需要幾行代碼，便可以生成繪圖，直方圖，功率譜，條形圖，錯誤圖，散點圖等。

• 對於image檢測的Python的第一行便出現了：from __future__ import print_function原來這是為了在老版本的Python中兼顧新特性的一種方法。具體地，從python2.1開始以后, 當一個新的語言特性首次出現在發行版中時候, 如果該新特性與以前舊版本python不兼容,^【9】 則該特性將會被默認禁用. 如果想啟用這個新特性, 則必須使用 "from __future__import *" 語句進行導入.Python 2.7可以通過 import __future__ 來將2.7版本的print語句移除，讓你可以Python3.x的print()功能函數的形式。例如：

from __future__ import print_function  
print('hello', end='\t') 

　　

代碼段2：設置在notebook中顯示圖像的默認參數

# This is a bit of magic to make matplotlib figures appear inline in the notebook
# rather than in a new window.
%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

理解：

• plt.rcParams是配置參數的方法，這里inline方式實現在notebook內顯示圖像，接下來設置顯示的默認參數^【10】：
• 通過給figsize參數賦值，設置顯示圖像的最大范圍
• 通過給interpolation賦值，設置差值方式
• 通過給cmap參數賦值'gray'，設置圖像在灰度空間內。

 

代碼段3：其他設置（讓notebook加載更多外部python模塊）

# Some more magic so that the notebook will reload external python modules;
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

 

 

 

Step 2：數據的獲取和處理

（1）加載原始CIFAR-10數據集

# Load the raw CIFAR-10 data.
cifar10_dir = 'cs231n/datasets/cifar-10-batches-py'

# Cleaning up variables to prevent loading data multiple times (which may cause memory issue)
try:
   del X_train, y_train
   del X_test, y_test
   print('Clear previously loaded data.')
except:
   pass

X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)

# As a sanity check, we print out the size of the training and test data.
print('Training data shape: ', X_train.shape)
print('Training labels shape: ', y_train.shape)
print('Test data shape: ', X_test.shape)
print('Test labels shape: ', y_test.shape)

 

這里使用本課程已經有的加載數據的方法加載Cifar-10數據庫，首先輸入數據庫的路徑，然后用load_CIFAR10方法自動取出訓練-標記數據集和測試-標記數據集。

這里為了避免重復加載訓練數據、測試數據集，用一個try-except方法處理異常。

最后做一個完成性檢查，運行這個notebook單元，按下 CTRL+Enter 得到完整性檢查結果：

 

 （2）可視化部分樣本，顯示部分訓練數據樣本的圖像

# Visualize some examples from the dataset.
# We show a few examples of training images from each class.
classes = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']
num_classes = len(classes)
samples_per_class = 7
for y, cls in enumerate(classes):
    idxs = np.flatnonzero(y_train == y)
    idxs = np.random.choice(idxs, samples_per_class, replace=False)
    for i, idx in enumerate(idxs):
        plt_idx = i * num_classes + y + 1
        plt.subplot(samples_per_class, num_classes, plt_idx)
        plt.imshow(X_train[idx].astype('uint8'))
        plt.axis('off')
        if i == 0:
            plt.title(cls)
plt.show()

該單元的運行輸出：

 

（3）子采樣：使代碼運行更快

# Subsample the data for more efficient code execution in this exercise
num_training = 5000
mask = list(range(num_training))
X_train = X_train[mask]
y_train = y_train[mask]

num_test = 500
mask = list(range(num_test))
X_test = X_test[mask]
y_test = y_test[mask]

從50000張圖片中取出500張作為訓練樣本，從10000張圖片中選出500作為練習用的測試樣本。

 

 （4）將三維數據轉為向量

# Reshape the image data into rows
X_train = np.reshape(X_train, (X_train.shape[0], -1))
X_test = np.reshape(X_test, (X_test.shape[0], -1))
print(X_train.shape, X_test.shape)

輸出：(5000, 3072) (500, 3072)

 

Step 3：分類

（1）初始化KNN分類器

from cs231n.classifiers import KNearestNeighbor

# Create a kNN classifier instance. 
# Remember that training a kNN classifier is a noop: 
# the Classifier simply remembers the data and does no further processing 
classifier = KNearestNeighbor()
classifier.train(X_train, y_train)

 

（2）計算距離

　　接下來，我們利用KNN分類器對測試數據進行分類。回憶之前我們講過的，分類步驟包含兩小步：

1. 計算出所有測試數據和所有訓練數據各自的距離；
2. 對每一個測試樣本，我們找出k個最近的樣本，投票得出標簽。

 

　　那么緊接着，我們先計算出這個距離矩陣。假設有 Ntr 個訓練樣本和 Nte 個測試樣本，計算出的距離矩陣大小就是Nte*Ntr。其中，(i,j)坐標上的元素就是第i個測試樣本和第j個訓練樣本的距離。

　　我們可以借助numpy提供的矩陣計算方法方便這個矩陣距離計算，但因為我對於矩陣計算的數學知識真的。。差。。。。

　　所以，我查閱了網上的資料來完成一層循環和無循環的計算的實現代碼。

　　最后，在Knn.py中也計算了三個計算方法的時間。真的，巧妙地用numpy矩陣計算工具簡化循環數目，可以極大地減少時間！！！

　　讓人深深感受到numpy矩陣計算工具的神奇。要是我數學好些，或許就能玩的六很多。

　　

 

　　以下是我對這個問題的實現：

 

1. 兩層循環計算距離：

首先，打開 cs231n/classifiers/k_nearest_neighbor.py文件， 實現compute_distances_two_loops函數。該函數在所有訓練和測試樣本上，利用一個雙層循環計算出距離矩陣。

  def compute_distances_two_loops(self, X):
    """
    Compute the distance between each test point in X and each training point
    in self.X_train using a nested loop over both the training data and the 
    test data.

    Inputs:
    - X: A numpy array of shape (num_test, D) containing test data.

    Returns:
    - dists: A numpy array of shape (num_test, num_train) where dists[i, j]
      is the Euclidean distance between the ith test point and the jth training
      point.
    """
    num_test = X.shape[0]
    print(X.shape)
    num_train = self.X_train.shape[0]
    dists = np.zeros((num_test, num_train))
    for i in range(num_test):
        print(i)
        for j in range(num_train):
        #####################################################################
        # TODO:                                                               #
        # Compute the l2 distance between the ith test point and the jth    #
        # training point, and store the result in dists[i, j]. You should   #
        # not use a loop over dimension.                                    #
        #####################################################################
            d = np.zeros(X.shape[1])
            for p in range(X.shape[1]):
                d[p] = np.square( X[i,p]-self.X_train[j,p])
            dists[i,j] = np.sqrt(np.sum(d))
        #####################################################################
        #                       END OF YOUR CODE                            #
        #####################################################################
    return dists

　　解釋：

• 　　因為不知道什么時候程序跑得完，我加上了一個print(i)，在測試Knn.py的時候，就可以知道現在正在計算第幾張測試圖片的距離了。事實上，我覺得這一點非常必要，因為我在運行knn.py時，單位一直顯示的是“IN【*】”，說明這個單位還在被執行。就灰常滴茫然。所以我在循環里加入了這個輸出，提示目前的進度。也看得出這個雙層循環結構真的非常ineffficient。
•        for p循環里，計算的是圖片每個像素pixel相對應的訓練圖片中的pixel元素的距離。
•        為了實現這個東東，我還復習了一遍numpy，建議還看不懂的同學也學習下。
•        實現完以后，運行knn.py中的單元：

# Open cs231n/classifiers/k_nearest_neighbor.py and implement
# compute_distances_two_loops.

# Test your implementation:
dists = classifier.compute_distances_two_loops(X_test)

輸出是這樣的：

 

　　后來，我意識到對像素進行循環遍歷也非常耗費時間！！！

　　所以參考了別人的代碼，修改如下：

　　

 def cal_dists_two_loop(self, X):
        """
        Calculate the distance with two for-loop
        Input:
        - X: A numpy array of shape (num_test, D) containing the test data
          consisting of num_test samples each of dimension D.
        Return:
        - dists: The distance between test X and train X
        """
        num_test = X.shape[0]
        num_train = self.X_train.shape[0]
        dists = np.zeros((num_test, num_train))
        for i in range(num_test):
            for j in range(num_train):
                dists[i][j] = np.sqrt(np.sum(np.square(X[i] - self.X_train[j])))

        return dists   

 

• 一層循環計算距離：

 　　實現compute_distances_one_loop：

　　

   def cal_dists_one_loop(self, X):
        """
        Calculate the distance with one for-loop
        Input:
        - X: A numpy array of shape (num_test, D) containing the test data
          consisting of num_test samples each of dimension D.
        Return:
        - dists: The distance between test X and train X
        """
        num_test = X.shape[0]
        num_train = self.X_train.shape[0]
        dists = np.zeros((num_test, num_train))
        for i in range(num_test):
            dists[i] = np.sqrt(np.sum(np.square(self.X_train - X[i]), axis=1))

        return dists

 

• 無循環地計算距離

  def compute_distances_no_loops(self, X):
    """
    Compute the distance between each test point in X and each training point
    in self.X_train using no explicit loops.

    Input / Output: Same as compute_distances_two_loops
    """
    num_test = X.shape[0]
    num_train = self.X_train.shape[0]
    dists = np.zeros((num_test, num_train)) 
    #########################################################################
    # TODO:                                                                 #
    # Compute the l2 distance between all test points and all training      #
    # points without using any explicit loops, and store the result in      #
    # dists.                                                                #
    #                                                                       #
    # You should implement this function using only basic array operations; #
    # in particular you should not use functions from scipy.                #
    #                                                                       #
    # HINT: Try to formulate the l2 distance using matrix multiplication    #
    #       and two broadcast sums.                                         #
    #########################################################################
    d1 = np.multiply(np.dot(X, self.X_train.T), -2)    # shape (num_test, num_train)
    d2 = np.sum(np.square(X), axis=1, keepdims=True)    # shape (num_test, 1)
    d3 = np.sum(np.square(self.X_train), axis=1)    # shape (1, num_train)
    dists = np.sqrt(d1 + d2 + d3)
    #########################################################################
    #                         END OF YOUR CODE                              #
    #########################################################################
    return dists

 

• 計算每種距離計算方法的耗時：

代碼是教程中已經自帶的，直接運行。

# Let's compare how fast the implementations are
def time_function(f, *args):
    """
    Call a function f with args and return the time (in seconds) that it took to execute.
    """
    import time
    tic = time.time()
    f(*args)
    toc = time.time()
    return toc - tic

two_loop_time = time_function(classifier.compute_distances_two_loops, X_test)
print('Two loop version took %f seconds' % two_loop_time)

one_loop_time = time_function(classifier.compute_distances_one_loop, X_test)
print('One loop version took %f seconds' % one_loop_time)

no_loop_time = time_function(classifier.compute_distances_no_loops, X_test)
print('No loop version took %f seconds' % no_loop_time)

# you should see significantly faster performance with the fully vectorized implementation

輸出：

 

 

 

（4）把距離可視化：

# We can visualize the distance matrix: each row is a single test example and
# its distances to training examples
plt.imshow(dists, interpolation='none')
plt.show()

 

問題1：注意到距離矩陣中的結構化模式，其中某些行或列可會更亮。 （請注意，默認黑色表示距離較近，而白色表示距離較遠。）

明顯亮行背后的原因？ 又是什么導致亮列？

你的答案：

 

 

 Step 4: 預測

(1)預測標簽

 　　實現k_nearest_neighbor.py，

 

  def predict_labels(self, dists, k=1):
    """
    Given a matrix of distances between test points and training points,
    predict a label for each test point.

    Inputs:
    - dists: A numpy array of shape (num_test, num_train) where dists[i, j]
      gives the distance betwen the ith test point and the jth training point.

    Returns:
    - y: A numpy array of shape (num_test,) containing predicted labels for the
      test data, where y[i] is the predicted label for the test point X[i].  
    """
    num_test = dists.shape[0]
    y_pred = np.zeros(num_test)

          # A list of length k storing the labels of the k nearest neighbors to
      # the ith test point.
    for i in range(num_test):
      #########################################################################
      # TODO:                                                                 #
      # Use the distance matrix to find the k nearest neighbors of the ith    #
      # testing point, and use self.y_train to find the labels of these       #
      # neighbors. Store these labels in closest_y.                           #
      # Hint: Look up the function numpy.argsort.                             #
      #########################################################################
        closest_y = []
        a = np.argsort(dists[i])[:k]
        closet_y = self.y_train[a]
        np.sort(closet_y)
      #########################################################################
      # TODO:                                                                 #
      # Now that you have found the labels of the k nearest neighbors, you    #
      # need to find the most common label in the list closest_y of labels.   #
      # Store this label in y_pred[i]. Break ties by choosing the smaller     #
      # label.                                                                #
      #########################################################################
       
        num_same = 0
        num_temp = 0
        for j in range(k):
            if(j == 0):
                num_same = num_temp = 1
                y_pred[i] = closet_y[0]
            else:
                if(closet_y[j] == closet_y[j-1]):
                    num_temp = num_temp + 1
                elif(closet_y[j] != closet_y[j-1]):
                    num_temp = 1
                if num_temp > num_same:
                    num_same = num_temp
                    y_pred[i] = closet_y[j]
                        
      #########################################################################
      #                           END OF YOUR CODE                            # 
      #########################################################################

    return y_pred

　　在knn.py中測試：

# Now implement the function predict_labels and run the code below:
# We use k = 1 (which is Nearest Neighbor).
y_test_pred = classifier.predict_labels(dists, k=1)

# Compute and print the fraction of correctly predicted examples
num_correct = np.sum(y_test_pred == y_test)
accuracy = float(num_correct) / num_test
print('Got %d / %d correct => accuracy: %f' % (num_correct, num_test, accuracy))

y_test_pred = classifier.predict_labels(dists, k=5)
num_correct = np.sum(y_test_pred == y_test)
accuracy = float(num_correct) / num_test
print('Got %d / %d correct => accuracy: %f' % (num_correct, num_test, accuracy))

 

 Step 5: 交叉驗證

Cross-validation

我們已經實現了K近鄰分類器，在我們之前的代碼中，我們默認k=5.現在我們通過交叉驗證來決定k這個超參數的最優值。

We have implemented the k-Nearest Neighbor classifier but we set the value k = 5 arbitrarily. We will now determine the best value of this hyperparameter with cross-validation.

 

num_folds = 5
k_choices = [1, 3, 5, 8, 10, 12, 15, 20, 50, 100]

X_train_folds = []
y_train_folds = []
################################################################################
# TODO:                                                                        #
# Split up the training data into folds. After splitting, X_train_folds and    #
# y_train_folds should each be lists of length num_folds, where                #
# y_train_folds[i] is the label vector for the points in X_train_folds[i].     #
# Hint: Look up the numpy array_split function.                                #
################################################################################
# Your code
X_train_folds = np.array_split(X_train,num_folds)
y_train_folds = np.array_split(y_train,num_folds)
################################################################################
#                                 END OF YOUR CODE                             #
################################################################################

# A dictionary holding the accuracies for different values of k that we find
# when running cross-validation. After running cross-validation,
# k_to_accuracies[k] should be a list of length num_folds giving the different
# accuracy values that we found when using that value of k.
k_to_accuracies = {}


################################################################################
# TODO:                                                                        #
# Perform k-fold cross validation to find the best value of k. For each        #
# possible value of k, run the k-nearest-neighbor algorithm num_folds times,   #
# where in each case you use all but one of the folds as training data and the #
# last fold as a validation set. Store the accuracies for all fold and all     #
# values of k in the k_to_accuracies dictionary.                               #
################################################################################
for k in k_choices:
    accuracies = []
    for i in range(num_folds):
        Xtr = X_train_folds[:i] + X_train_folds[i+1:]
        ytr = y_train_folds[:i] + y_train_folds[i+1:]
        Xcv = X_train_folds[i]
        ycv = y_train_folds[i]
        classifier.train(Xtr,ytr)
        ycv_pred = classifier.predict(Xcv, k = k , num_loops = 1)
        accuracy = np.mean(ycv_pred == ycv)
        accuracies.append(accuracy)
    k_accuracy[k] = accuracies
################################################################################
#                                 END OF YOUR CODE                             #
################################################################################

# Print out the computed accuracies
for k in sorted(k_to_accuracies):
    for accuracy in k_to_accuracies[k]:
        print('k = %d, accuracy = %f' % (k, accuracy))

 

 

參考資料：

1. 《 cs231n 課程作業 Assignment 1 》https://blog.csdn.net/zhangxb35/article/details/55223825
2. 《遠程訪問（雲）服務器上ipython設置》https://blog.csdn.net/u011531010/article/details/60580807
3. 《Linux安裝遠程ipython notebook》https://blog.csdn.net/suzyu12345/article/details/51037905
4. 《Jupyter Notebook介紹、安裝及使用教程》https://www.jianshu.com/p/91365f343585
5. 《本機直接遠程連接服務器Jupyter notebook》https://blog.csdn.net/danlei94/article/details/74049975?utm_source=itdadao&utm_medium=referral

6. 《課程設置說明

   》http://cs231n.github.io/setup-instructions/
7. 《Ipython教程（英文）》http://cs231n.github.io/ipython-tutorial/
8. 《Jupyter notebook入門教程（上）》https://blog.csdn.net/red_stone1/article/details/72858962

9. 《關於python 中的__future__模塊（from __future__ import ***）》https://blog.csdn.net/wonengguwozai/article/details/75267858

10. Caffe學習1-圖像識別與數據可視化：https://blog.csdn.net/cfyzcc/article/details/51318200

11. 斯坦福CS231n項目實戰（一）：k最近鄰（kNN）分類算法:https://blog.csdn.net/red_stone1/article/details/79238866
    

12. 交叉驗證（簡書）：https://www.jianshu.com/p/201a164e1b35