主成分分析(PCA) vs 多元判別式分析(MDA)
PCA和MDA都是線性變換的方法,二者關系密切。在PCA中,我們尋找數據集中最大化方差的成分,在MDA中,我們對類間最大散布的方向更感興趣。
一句話,通過PCA,我們將整個數據集(不帶類別標簽)映射到一個子空間中,在MDA中,我們致力於找到一個能夠最好區分各類的最佳子集。粗略來講,PCA是通過尋找方差最大的軸(在一類中,因為PCA把整個數據集當做一類),在MDA中,我們還需要最大化類間散布。
在通常的模式識別問題中,MDA往往在PCA后面。
PCA的主要算法如下:
其中協方差矩陣的分解可以通過按對稱矩陣的特征向量來,也可以通過分解矩陣的SVD來實現,而在Scikit-learn中,也是采用SVD來實現PCA算法的。
本文將用三種方法來實現PCA算法,一種是原始算法,即上面所描述的算法過程,具體的計算方法和過程,可以參考:A tutorial on Principal Components Analysis, Lindsay I Smith. 一種是帶SVD的原始算法,在Python的Numpy模塊中已經實現了SVD算法,並且將特征值從大從小排列,省去了對特征值和特征向量重新排列這一步。最后一種方法是用Python的Scikit-learn模塊實現的PCA類直接進行計算,來驗證前面兩種方法的正確性。
用以上三種方法來實現PCA的完整的Python如下:
import numpy as np from sklearn.decomposition import PCA import sys #returns choosing how many main factors def index_lst(lst, component=0, rate=0): #component: numbers of main factors #rate: rate of sum(main factors)/sum(all factors) #rate range suggest: (0.8,1) #if you choose rate parameter, return index = 0 or less than len(lst) if component and rate: print('Component and rate must choose only one!') sys.exit(0) if not component and not rate: print('Invalid parameter for numbers of components!') sys.exit(0) elif component: print('Choosing by component, components are %s......'%component) return component else: print('Choosing by rate, rate is %s ......'%rate) for i in range(1, len(lst)): if sum(lst[:i])/sum(lst) >= rate: return i return 0 def main(): # test data mat = [[-1,-1,0,2,1],[2,0,0,-1,-1],[2,0,1,1,0]] # simple transform of test data Mat = np.array(mat, dtype='float64') print('Before PCA transforMation, data is:\n', Mat) print('\nMethod 1: PCA by original algorithm:') p,n = np.shape(Mat) # shape of Mat t = np.mean(Mat, 0) # mean of each column # substract the mean of each column for i in range(p): for j in range(n): Mat[i,j] = float(Mat[i,j]-t[j]) # covariance Matrix cov_Mat = np.dot(Mat.T, Mat)/(p-1) # PCA by original algorithm # eigvalues and eigenvectors of covariance Matrix with eigvalues descending U,V = np.linalg.eigh(cov_Mat) # Rearrange the eigenvectors and eigenvalues U = U[::-1] for i in range(n): V[i,:] = V[i,:][::-1] # choose eigenvalue by component or rate, not both of them euqal to 0 Index = index_lst(U, component=2) # choose how many main factors if Index: v = V[:,:Index] # subset of Unitary matrix else: # improper rate choice may return Index=0 print('Invalid rate choice.\nPlease adjust the rate.') print('Rate distribute follows:') print([sum(U[:i])/sum(U) for i in range(1, len(U)+1)]) sys.exit(0) # data transformation T1 = np.dot(Mat, v) # print the transformed data print('We choose %d main factors.'%Index) print('After PCA transformation, data becomes:\n',T1) # PCA by original algorithm using SVD print('\nMethod 2: PCA by original algorithm using SVD:') # u: Unitary matrix, eigenvectors in columns # d: list of the singular values, sorted in descending order u,d,v = np.linalg.svd(cov_Mat) Index = index_lst(d, rate=0.95) # choose how many main factors T2 = np.dot(Mat, u[:,:Index]) # transformed data print('We choose %d main factors.'%Index) print('After PCA transformation, data becomes:\n',T2) # PCA by Scikit-learn pca = PCA(n_components=2) # n_components can be integer or float in (0,1) pca.fit(mat) # fit the model print('\nMethod 3: PCA by Scikit-learn:') print('After PCA transformation, data becomes:') print(pca.fit_transform(mat)) # transformed data main()
運行以上代碼,輸出結果為:
這說明用以上三種方法來實現PCA都是可行的。這樣我們就能理解PCA的具體實現過程啦~~有興趣的讀者可以用其它語言實現一下哈。
本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。