數據挖掘實戰（二）—— 類不平衡問題_信用卡欺詐檢測

寫在jupyter里面比較漂亮：

https://douzujun.github.io/page/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0%E7%AC%94%E8%AE%B0/%E7%B1%BB%E4%B8%8D%E5%B9%B3%E8%A1%A1%E9%97%AE%E9%A2%98_%E4%BF%A1%E7%94%A8%E5%8D%A1%E6%AC%BA%E8%AF%88%E6%A3%80%E6%B5%8B.html

In [34]:
import pandas as pd import matplotlib.pylab as plt import numpy as np %matplotlib inline In [35]: data = pd.read_csv('creditcard.csv') data.head() # 0 - 正常的樣本，1 - 有問題的數據

Out[35]:

	Time	V1	V2	V3	V4	V5	V6	V7	V8	V9	...	V21	V22	V23	V24	V25	V26	V27	V28	Amount	Class
0	0.0	-1.359807	-0.072781	2.536347	1.378155	-0.338321	0.462388	0.239599	0.098698	0.363787	...	-0.018307	0.277838	-0.110474	0.066928	0.128539	-0.189115	0.133558	-0.021053	149.62	0
1	0.0	1.191857	0.266151	0.166480	0.448154	0.060018	-0.082361	-0.078803	0.085102	-0.255425	...	-0.225775	-0.638672	0.101288	-0.339846	0.167170	0.125895	-0.008983	0.014724	2.69	0
2	1.0	-1.358354	-1.340163	1.773209	0.379780	-0.503198	1.800499	0.791461	0.247676	-1.514654	...	0.247998	0.771679	0.909412	-0.689281	-0.327642	-0.139097	-0.055353	-0.059752	378.66	0
3	1.0	-0.966272	-0.185226	1.792993	-0.863291	-0.010309	1.247203	0.237609	0.377436	-1.387024	...	-0.108300	0.005274	-0.190321	-1.175575	0.647376	-0.221929	0.062723	0.061458	123.50	0
4	2.0	-1.158233	0.877737	1.548718	0.403034	-0.407193	0.095921	0.592941	-0.270533	0.817739	...	-0.009431	0.798278	-0.137458	0.141267	-0.206010	0.502292	0.219422	0.215153	69.99	0

5 rows × 31 columns

In [36]:
count_classes = pd.value_counts(data['Class'], sort=True).sort_index() count_classes.plot(kind = 'bar', alpha=0.5) plt.title('Fraud class histogram') plt.xlabel('Class') plt.ylabel('Frequency') # 此時無缺陷的樣本數非常多，有缺陷的樣本非常的少，可以明顯看到 樣本分布非常不平衡 Out[36]: <matplotlib.text.Text at 0xfc64470>

 

 

樣本不均衡解決方案

• 下采樣 (兩者一樣小)

  • 從0的樣本中選取 跟 1的樣本一樣小

• 過采樣 (兩者一樣多)

  • 1的樣本中采取生成策略，生成的數據和 0號樣本一樣多

標准化

• 比如Amount，數據量區間很大，會對模型有一個“數據大就重要的”誤區; 所以要進行歸一化，或者標准化
• 把他們區間放在 [-1,1] 或者[0, 1]區間上

In [37]:
# 這個sklearn庫: 預處理操作 => 標准化的模塊
from sklearn.preprocessing import StandardScaler # fit_transform(): 對數據進行一個變化了; 變化好的列成為一個新屬性添加到 data data['normAmount'] = StandardScaler().fit_transform(data['Amount'].reshape(-1, 1)) data = data.drop(['Time', 'Amount'], axis=1) data.head()

Out[37]:

	V1	V2	V3	V4	V5	V6	V7	V8	V9	V10	...	V21	V22	V23	V24	V25	V26	V27	V28	Class	normAmount
0	-1.359807	-0.072781	2.536347	1.378155	-0.338321	0.462388	0.239599	0.098698	0.363787	0.090794	...	-0.018307	0.277838	-0.110474	0.066928	0.128539	-0.189115	0.133558	-0.021053	0	0.244964
1	1.191857	0.266151	0.166480	0.448154	0.060018	-0.082361	-0.078803	0.085102	-0.255425	-0.166974	...	-0.225775	-0.638672	0.101288	-0.339846	0.167170	0.125895	-0.008983	0.014724	0	-0.342475
2	-1.358354	-1.340163	1.773209	0.379780	-0.503198	1.800499	0.791461	0.247676	-1.514654	0.207643	...	0.247998	0.771679	0.909412	-0.689281	-0.327642	-0.139097	-0.055353	-0.059752	0	1.160686
3	-0.966272	-0.185226	1.792993	-0.863291	-0.010309	1.247203	0.237609	0.377436	-1.387024	-0.054952	...	-0.108300	0.005274	-0.190321	-1.175575	0.647376	-0.221929	0.062723	0.061458	0	0.140534
4	-1.158233	0.877737	1.548718	0.403034	-0.407193	0.095921	0.592941	-0.270533	0.817739	0.753074	...	-0.009431	0.798278	-0.137458	0.141267	-0.206010	0.502292	0.219422	0.215153	0	-0.073403

5 rows × 30 columns

下采樣實現

In [38]:
X = data.ix[:, data.columns != 'Class'] y = data.ix[:, data.columns == 'Class'] # Number of data points in the minority class # 樣本的個數 number_records_fraud = len(data[data.Class == 1]) # 樣本的索引, 所有值為1的 索引 fraud_indices = np.array(data[data.Class == 1].index) # Picking the indices of the normal classes # 值為0 的 數據的索引 normal_indices = data[data.Class == 0].index # 使兩個樣本一樣少 # Out of the indices we picked, randomly select 'x' number (number_records_fraud) # 隨機選取樣本中的數據, 隨機選擇: np.random.choice(Class為0的數據索引=>樣本, 選擇多少數據量, 是否選擇代替=false) random_normal_indices = np.random.choice(normal_indices, number_records_fraud, replace=False) random_normal_indices = np.array(random_normal_indices) # Appending the 2 indices (連接缺陷數據索引 和 隨機選取的Class=0的數據索引) under_sample_indices = np.concatenate([fraud_indices, random_normal_indices]) # Under sample dataset 下采樣 (選取該索引下的數據) under_sample_data = data.iloc[under_sample_indices, :] X_undersample = under_sample_data.ix[:, under_sample_data.columns != 'Class'] y_undersample = under_sample_data.ix[:, under_sample_data.columns == 'Class'] # Showing Ratio print('Percentage oif normal transactions: ', len(under_sample_data[under_sample_data.Class == 0]) / len(under_sample_data)) print('Percentage oif fraud transactions: ', len(under_sample_data[under_sample_data.Class == 1]) / len(under_sample_data)) print('Total number of transactions in resampled data: ', len(under_sample_data)) Percentage oif normal transactions: 0.5 Percentage oif fraud transactions: 0.5 Total number of transactions in resampled data: 984 In [39]: # 交叉驗證， train_test_split: 切分數據 from sklearn.cross_validation import train_test_split # Whole dataset, 將數據切割成訓練集0.7 和測試集 0.3 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 0) print('Number transactions train dataset: ', len(X_train)) print('Number transactions test dataset: ', len(X_test)) print('Total number of transactions: ', len(X_train) + len(X_test)) # Undersampled dataset(對下采樣數據集相同操作) # 只用下采樣模型進行訓練 # 最終用原始數據集進行測試 X_train_undersample, X_test_undersample, y_train_undersample, y_test_undersample = train_test_split(X_undersample, y_undersample, test_size = 0.3, random_state = 0) print('') print("Number transactions train dataset: ", len(X_train_undersample)) print("Number transactions test dataset: ", len(X_test_undersample)) print("Total number of transactions: ", len(X_train_undersample) + len(X_test_undersample)) Number transactions train dataset: 199364 Number transactions test dataset: 85443 Total number of transactions: 284807 Number transactions train dataset: 688 Number transactions test dataset: 296 Total number of transactions: 984 

模型評估標准

• 在類不平衡問題中，用精度來衡量指標是騙人的！沒有意義(1000個人，全部預測為正常， 0個癌症) 精度 = TPTotalTPTotal

• 這里使用Recall(召回率, 查全率) = TPTP+FNTPTP+FN, 1000個人(990個正常,10個癌症),如果檢測出0個病人 010=0,檢測出2個病人210=0.2010=0,檢測出2個病人210=0.2

• 檢測任務上，通常用Recall作為模型的評估標准

In [40]:
# Recall = TP / (TP + FN)
from sklearn.linear_model import LogisticRegression # KFold->做幾倍的交叉驗證 from sklearn.cross_validation import KFold, cross_val_score # confusion_matrix: 混淆矩陣 from sklearn.metrics import confusion_matrix, recall_score, classification_report

正則化懲罰

• 設置正則化懲罰項
• 希望當前模型的泛化能力(更穩定一些)， 不僅滿足訓練數據，還要在測試數據上盡可能滿足
• 浮動的差異小 ===> 過擬合的風險小
• 懲罰 θθ (一組分布大A，一組分布小B) => 大力度懲罰A模型，小力度懲罰B模型
• L2正則化L2正則化: loss函數(越低越好) + 12W2(懲罰項)12W2(懲罰項) => 計算哪個模型loss值小, 分析哪個模型更好
• L1正則化L1正則化: loss+|W|loss+|W|
• λL2:設置懲罰力度λλL2:設置懲罰力度λ

In [41]:
def printing_Kfold_score(x_train_data, y_train_data): # 切分成5部分，把原始訓練集進行切分 fold = KFold(len(y_train_data), 5, shuffle=False) # Different C parameters (正則化懲罰項) # 希望當前模型的泛化能力(更穩定一些) ， 不僅滿足訓練數據，還要在測試數據上盡可能滿足 # 浮動的差異小 ===> 過擬合的風險小 c_param_range = [0.01, 0.1, 1, 10, 100] # C_parameter => lambda results_table = pd.DataFrame(index = range(len(c_param_range), 2), columns = ['C_parameter', 'Mean recall score']) results_table['C_parameter'] = c_param_range # the k-fold will give 2 lists: train_indics = indices[0], test_indices = indices[1] j = 0 # 查看哪一個C值比較好 for c_param in c_param_range: print('-----------------------------------------') print('C parameter: ', c_param) print('-----------------------------------------') print('') recall_accs = [] # 每次交叉驗證的結果 for iteration, indices in enumerate(fold, start=1): # Call the logistic regresstion model with a certain C parameter, "L1懲罰 or L2懲罰" lr = LogisticRegression(C = c_param, penalty='l1') # Use the training data to fit the model. In the case, we use the portion of the fold to train # with indices[0]. We then predict on the portion assigned as the test cross validation with indices. # 進行訓練, 交叉驗證里的數據(訓練集)-->建立一個模型  lr.fit(x_train_data.iloc[indices[0], :], y_train_data.iloc[indices[0], :].values.ravel()) # Predict values using the test indices in the training data # 進行預測, 用交叉驗證中的驗證集 再進行一個驗證(測試)的操作 y_pred_undersample = lr.predict(x_train_data.iloc[indices[1], :].values) # Calculate the recall score and append it to a list for recall scores representing the current xx # 計算當前模型的recall (indices[1]: 測試集) recall_acc = recall_score(y_train_data.iloc[indices[1], :].values, y_pred_undersample) recall_accs.append(recall_acc) print("Iterator ", iteration, ': recall score = ', recall_acc) # The mean value of those recall scores is the metric we want to save and get hold of. results_table.ix[j, 'Mean recall score'] = np.mean(recall_accs) j += 1 print('') print('Mean recall score ', np.mean(recall_accs)) print('') best_c = results_table.loc[results_table['Mean recall score'].idxmax()]['C_parameter'] # Finally, we can check which C parameter is the best amongst the chosen. print('*********************************************************************************') print('Best model to choose from cross validation is with C parameter = ', best_c) print('*********************************************************************************') return best_c In [42]: best_c = printing_Kfold_score(X_train_undersample, y_train_undersample)

 [out]:

-----------------------------------------
C parameter:  0.01
----------------------------------------- Iterator 1 : recall score = 0.931506849315 Iterator 2 : recall score = 0.917808219178 Iterator 3 : recall score = 1.0 Iterator 4 : recall score = 0.959459459459 Iterator 5 : recall score = 0.954545454545 Mean recall score 0.9526639965 ----------------------------------------- C parameter: 0.1 ----------------------------------------- Iterator 1 : recall score = 0.835616438356 Iterator 2 : recall score = 0.86301369863 Iterator 3 : recall score = 0.915254237288 Iterator 4 : recall score = 0.932432432432 Iterator 5 : recall score = 0.893939393939 Mean recall score 0.888051240129 ----------------------------------------- C parameter: 1 ----------------------------------------- Iterator 1 : recall score = 0.849315068493 Iterator 2 : recall score = 0.890410958904 Iterator 3 : recall score = 0.949152542373 Iterator 4 : recall score = 0.945945945946 Iterator 5 : recall score = 0.893939393939 Mean recall score 0.905752781931 ----------------------------------------- C parameter: 10 ----------------------------------------- Iterator 1 : recall score = 0.86301369863 Iterator 2 : recall score = 0.876712328767 Iterator 3 : recall score = 0.949152542373 Iterator 4 : recall score = 0.945945945946 Iterator 5 : recall score = 0.909090909091 Mean recall score 0.908783084961 ----------------------------------------- C parameter: 100 ----------------------------------------- Iterator 1 : recall score = 0.86301369863 Iterator 2 : recall score = 0.876712328767 Iterator 3 : recall score = 0.949152542373 Iterator 4 : recall score = 0.945945945946 Iterator 5 : recall score = 0.939393939394 Mean recall score 0.914843691022 ********************************************************************************* Best model to choose from cross validation is with C parameter = 0.01 ********************************************************************************* 

混淆矩陣

• 如何繪制混淆矩陣
• 混淆矩陣用途

In [43]:
def plot_confusion_matrix(cm, classes, title = 'Confusion matrix', cmap = plt.cm.Blues): ''' This function prints and plots the confusion matrix ''' plt.imshow(cm, interpolation='nearest', cmap=cmap) plt.title(title) plt.colorbar() tick_marks = np.arange(len(classes)) plt.xticks(tick_marks, classes, rotation=0) plt.yticks(tick_marks, classes) thresh = cm.max() / 2. for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])): plt.text(j, i, cm[i, j], horizontalalignment='center', color='white' if cm[i, j] > thresh else 'black') plt.tight_layout() plt.ylabel('True label') plt.xlabel('Perdicted label') In [44]: import itertools lr = LogisticRegression(C = best_c, penalty='l1') lr.fit(X_train_undersample, y_train_undersample.values.ravel()) y_pred_undersample = lr.predict(X_test_undersample.values) # Compulter confusion matrix cnf_matrix = confusion_matrix(y_test_undersample, y_pred_undersample) np.set_printoptions(precision=2) print("Recal metric in the testing dataset: ", cnf_matrix[1, 1]/(cnf_matrix[1, 0] + cnf_matrix[1, 1])) # Plot non-normalized confusion matrix class_names = [0, 1] plt.figure() plot_confusion_matrix(cnf_matrix, classes=class_names, title='Confusion matrix') plt.show() Recal metric in the testing dataset: 0.931972789116

 

In [45]:
lr = LogisticRegression(C = best_c, penalty='l1') lr.fit(X_train_undersample, y_train_undersample.values.ravel()) y_pred = lr.predict(X_test.values) # COmputer confusion matrix cnf_matrix = confusion_matrix(y_test, y_pred) np.set_printoptions(precision=2) print('Recall metric in the testing dataset: ', cnf_matrix[1, 1] / (cnf_matrix[1, 0] + cnf_matrix[1, 1])) # Plot non-normalized confusion matrix class_names = [0, 1] plt.figure() plot_confusion_matrix(cnf_matrix, classes=class_names, title='Confusion matrix') plt.show() # Recall = TP / (TP + FN) # 7101 ==> 誤殺了那么多數據

Recall metric in the testing dataset:  0.918367346939

 

 

注: 下采樣伴隨着誤殺的問題，精度不高

In [46]:
# 什么都不做的話
# best_c = printing_Kfold_score(X_train, y_train)
 

Logical 回歸

• 閾值對結果的影響

In [47]:
lr = LogisticRegression(C = 0.01, penalty='l1') lr.fit(X_train_undersample, y_train_undersample.values.ravel()) y_pred_undersample_proba = lr.predict_proba(X_test_undersample.values) # 設置不同的閾值 thresholds = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9] plt.figure(figsize=(10, 10)) j = 1 for i in thresholds: y_test_undersample_high_recall = y_pred_undersample_proba[:, 1] > i plt.subplot(3, 3, j) j += 1 # Compute confusion matrix cnf_matrix = confusion_matrix(y_test_undersample, y_test_undersample_high_recall) np.set_printoptions(precision=2) print('Recal metric in the testing dataset: ', cnf_matrix[1, 1] / (cnf_matrix[1, 0] + cnf_matrix[1, 1])) # Plot non-normalized confusion matrix class_names = [0, i] plot_confusion_matrix(cnf_matrix, classes=class_names, title='Threshold >= %s' %i)

 [out]:

Recal metric in the testing dataset:  1.0
Recal metric in the testing dataset:  1.0 Recal metric in the testing dataset: 1.0 Recal metric in the testing dataset: 0.972789115646 Recal metric in the testing dataset: 0.931972789116 Recal metric in the testing dataset: 0.87074829932 Recal metric in the testing dataset: 0.823129251701 Recal metric in the testing dataset: 0.761904761905 Recal metric in the testing dataset: 0.591836734694

 

過采樣 — SMOTE樣本生成策略

• 對於少數類中每一個樣本x, 以歐氏距離為標准計算它到少數類樣本集中所有樣本的距離，得到其k近鄰。
• 根據樣本不平衡比例設置一個采樣比例以確定采樣倍率N，對於每一個少數類樣本x, 從其k近鄰中隨機選擇若干樣本，假設選擇的近鄰為 xn。
• 對於每一個隨機選出的近鄰 xn, 分別與原樣本按照如下的公式構建新的樣本。
• Xnew=X+rand(0,1)×歐式距離Xnew=X+rand(0,1)×歐式距離

In [48]:
import pandas as pd from imblearn.over_sampling import SMOTE from sklearn.ensemble import RandomForestClassifier from sklearn.metrics import confusion_matrix from sklearn.model_selection import train_test_split In [49]: credit_cards = pd.read_csv('creditcard.csv') columns = credit_cards.columns # The labels are in the last column ('Class'), Simply remove it to obtain features cloumns features_columns = columns.delete(len(columns) - 1) features = credit_cards[features_columns] labels = credit_cards['Class'] In [50]: features_train, features_test, labels_train, labels_test = train_test_split(features, labels, test_size=0.2, random_state=0) In [51]: oversampler = SMOTE(random_state=0) os_features, os_labels = oversampler.fit_sample(features_train, labels_train) In [ ]: len(os_labels[os_labels == 1])

Out[ ]:

227454

In [ ]:

os_features = pd.DataFrame(os_features)
os_labels = pd.DataFrame(os_labels) best_c = printing_Kfold_score(os_features, os_labels)

 Out[]:

-----------------------------------------
C parameter:  0.01
----------------------------------------- Iterator 1 : recall score = 0.890322580645 Iterator 2 : recall score = 0.894736842105 Iterator 3 : recall score = 0.968883479031

In [56]:

lr = LogisticRegression(C = best_c, penalty='l1')
lr.fit(os_features, os_labels.values.ravel())
y_pred = lr.predict(features_test.values) # Compute confusion matrix cnf_matrix = confusion_matrix(labels_test, y_pred) np.set_printoptions(precision=2) print("Recall metric in the testing dataset: ", cnf_matrix[1, 1] / (cnf_matrix[1, 0] + cnf_matrix[1, 1])) # Plot non-nonmalized confusion matrix class_names = [0, 1] plt.figure() plot_confusion_matrix(cnf_matrix, classes = class_names, title = 'Confusion matrix') plt.show()

 

Recall metric in the testing dataset:  0.910891089109

 

 

過采樣--> Recal 低一些，模型整體的精度更高一些

• 精度 = (TP + TN) / Total

In [ ]:

 

In [ ]: