機器學習篇：sklearn.model_selection

本文轉載自查看原文 2021-12-19 15:38 1029 知識整理

sklearn提供了許多包來進行機器學習，只是很多不去了解的話，到使用的時候就會手忙腳亂根本不會去用，所以這里整理一下，這里整理的順序是個人想要了解的順序。

在一開始對這個工具毫無概念的話，可以嘗試閱讀：User Guide，一般瀏覽器（如谷歌）提供頁面翻譯成中文的方法，當對某些概念不明確，可換回英文再看看。

0、整體

2、sklearn.model_selection

sklearn有很完善的官方文檔(sklearn.model_selection)以及使用指南(3. Model selection and evaluation)，所以這里只是個人學習的記錄，也是跟着官方文檔進行了解。

2.1 Splitter Functions 拆分器功能

2.1.1 train_test_split 拆分訓練集測試集

# train_test_split
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification

SEED = 666
X,y = make_classification(n_samples=100, 
                          n_features=20, 
                          shuffle=True, 
                          random_state=SEED)

print("拆分前：",X.shape,y.shape)
X_train,y_train,X_test,y_test = train_test_split(X,y,test_size=0.25,random_state=SEED)
print("拆分后：",X_train.shape,y_train.shape,X_test.shape,y_test.shape)

2.1.2 check_cv 簡單進行五折拆分數據集

check_cv返回的是一個KFold實例
check_cv拆分后的順序是沒有打亂的，譬如100個樣本拆分五折會默認分成五份，其下標固定為(0,19)(20,39)(40,59),(60,79)(80,99)

# check_cv
from sklearn.model_selection import check_cv
from sklearn.datasets import make_classification

SEED = 666
X,y = make_classification(n_samples=100, 
                          n_features=20, 
                          shuffle=True, 
                          random_state=SEED)

print("拆分前：",X.shape,y.shape)
aKFold = check_cv(cv=5, y=y, classifier=False) #返回的是一個KFold實例
for train_index, test_index in aKFold.split(X):
    # train_index, test_index返回的是下標
    #print("%s %s" % (train_index, test_index))
    X_train,y_train,X_test,y_test = X[train_index],y[train_index],X[test_index],y[test_index]
    print("拆分后：",X_train.shape,y_train.shape,X_test.shape,y_test.shape)

2.2 Splitter Classes 拆分器類

這里有15個數據集拆分器，為了靈活地應對各種拆分需求，各種拓展看着我頭疼，甚至一度懷疑我這樣子是不是在浪費時間。有時候其實只有在有應用需求的時候才會明白為什么需要這個拆分器。所以進行以下的分類，從簡單的開始。

2.2.1 K折拆分--KFold

默認是五折拆分，不打亂順序，不放回
shuffle=True后則是不固定的五折拆分，需要設置隨機種子random_state以進行復現

# KFold 
#K折交叉驗證，即將原數據集分成K份，每一折將其中一份作為測試集，其他k-1份作為訓練集

# 隨機的多折拆分（默認五折拆分）,shuffle=True會打亂訓練集測試集

from sklearn.model_selection import KFold
from sklearn.datasets import make_classification

SEED = 666
X,y = make_classification(n_samples=100, 
                          n_features=20, 
                          shuffle=True, 
                          random_state=SEED)

print("拆分前：",X.shape,y.shape)
aKFold = KFold(n_splits=5, shuffle=True, random_state=SEED) #返回的是一個KFold實例,shuffle=True則不是固定的下標
for train_index, test_index in aKFold.split(X):
    # train_index, test_index返回的是下標
    print("%s %s" % (train_index, test_index))
    X_train,y_train,X_test,y_test = X[train_index],y[train_index],X[test_index],y[test_index]
    print("拆分后：",X_train.shape,y_train.shape,X_test.shape,y_test.shape)

2.2.2 K折拆分--GroupKFold

GroupKFold(n_splits=5)：返回一個GroupKFold實例
GroupKFold.get_n_splits(self, X=None, y=None, groups=None)：返回拆分的折數
split(self, X, y=None, groups=None)，返回拆分結果index的迭代器，會根據傳入的第三個參數groups來拆分數據集X,y，使得拆分后分類比例不變

# GroupKFold

# 簡單的多折拆分（默認五折拆分）,需要傳入groups，會根據傳入groups使得每個groups在訓練集測試集的比例不變，與Stratified類似


from sklearn.model_selection import GroupKFold
import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
#help(GroupKFold)

SEED = 666
X,y = make_classification(n_samples=100, 
                          n_features=20, 
                          shuffle=True, 
                          n_classes=2,
                          n_clusters_per_class=1,
                          n_informative =18,
                          weights=[0.1,0.9],
                          random_state=SEED)

print("拆分前：",X.shape,y.shape)
print("拆分前的數據")
print(pd.DataFrame(y).value_counts())


group_kfold = GroupKFold(n_splits=2) #n_splits要與傳入groups的分類數相符
#group_kfold.get_n_splits(X, y, y)

# 會根據傳入的第三個參數groups來拆分數據集X,y，傳入了分類標簽y所以會將二分類數據按照0，1拆開
for train_index, test_index in group_kfold.split(X, y, groups = y):
    print("拆分--------------------------------------------------")
    print("訓練集數據:\n",pd.DataFrame(y[train_idx]).value_counts())
    print("測試集數據:\n",pd.DataFrame(y[test_idx]).value_counts())
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    print("拆分后：",X_train.shape,y_train.shape,X_test.shape,y_test.shape)

2.2.3 K折拆分--StratifiedKFold

生成測試集，使所有包含相同的類分布，或盡可能接近。
是不變的類別標簽：重貼標簽到不應該改變所產生的指標。y = ["Happy", "Sad"]``y = [1, 0]
保留數據集排序中的順序依賴性，當 shuffle=False：某些測試集中來自類 k 的所有樣本在 y 中是連續的，或者在 y 中被來自除 k 之外的類的樣本分隔。
生成測試集，其中最小和最大最多相差一個樣本。

# StratifiedKFold

# 相比於KFold，在進行split的時候需要傳入y，並且會根據y的分類，保證分類后y在各個數據集中比例不變,類似於GroupKFold（基於參數groups）

import numpy as np
import pandas as pd
from sklearn.model_selection import *
from sklearn.datasets import make_classification


SEED = 666

X,y = make_classification(n_samples=200, 
                          n_features=20, 
                          shuffle=True, 
                          n_classes=3,
                          n_clusters_per_class=1,
                          n_informative =18,
                          random_state=SEED)

skf = StratifiedKFold(n_splits=5, shuffle=False, random_state=None)

print("拆分前的數據")
print(pd.DataFrame(y).value_counts())
for train_idx, test_idx in skf.split(X,y):
    print("拆分--------------------------------------------------")
    print("訓練集數據:\n",pd.DataFrame(y[train_idx]).value_counts())
    print("測試集數據:\n",pd.DataFrame(y[test_idx]).value_counts())

2.2.4 K折拆分--StratifiedGroupKFold

# StratifiedGroupKFold

# 在進行split的時候需要傳入X,y和groups，觀察其結果，似乎只取決於傳入的group，group的長度取決於X、y的長度，分類數最好與n_splits相同


import numpy as np
import pandas as pd
from sklearn.model_selection import *
from sklearn.datasets import make_classification


SEED = 666

X,y = make_classification(n_samples=30, 
                          n_features=20, 
                          shuffle=True, 
                          n_classes=2,
                          n_clusters_per_class=1,
                          n_informative =18,
                          random_state=SEED)

sgk = StratifiedGroupKFold(n_splits=3, shuffle=False, random_state=None)

print("拆分前的數據")
print(pd.DataFrame(y).value_counts())
groups =  np.hstack((np.zeros(10),np.ones(10),np.ones(10)+1)) 

for train_idx, test_idx in sgk.split(X,y,groups):
    print("TRAIN:", train_idx, "TEST:", test_idx)
    print("訓練集數據:\n",pd.DataFrame(y[train_idx]).value_counts())
    print("測試集數據:\n",pd.DataFrame(y[test_idx]).value_counts())

2.2.5 K折拆分--RepeatedKFold

# RepeatedKFold

#重復n_repeats次n_splits折的KFold拆分，最后拆分的次數應該是n_splits*n_repeats

import numpy as np
import pandas as pd
from sklearn.model_selection import *
from sklearn.datasets import make_classification


SEED = 666

X,y = make_classification(n_samples=20, 
                          n_features=20, 
                          shuffle=True, 
                          n_classes=6,
                          n_clusters_per_class=1,
                          n_informative =18,
                          weights=[0.1,0.5,0.1,0.1,0.1,0.1],
                          random_state=SEED)


#重復n_repeats次n_splits折的拆分，最后拆分的次數應該是n_splits*n_repeats
rkf = RepeatedKFold(n_splits=4, n_repeats=2, random_state=666)




for train_idx, test_idx in rkf.split(X):
    print("TRAIN:", train_idx, "TEST:", test_idx)

2.2.6 K折拆分--RepeatedStratifiedKFold

# RepeatedStratifiedKFold

# 重復n_repeats次n_splits折的StratifiedKFold拆分，最后拆分的次數應該是n_splits*n_repeats

import numpy as np
import pandas as pd
from sklearn.model_selection import *
from sklearn.datasets import make_classification


SEED = 666

X,y = make_classification(n_samples=30, 
                          n_features=20, 
                          shuffle=True, 
                          n_classes=2,
                          n_clusters_per_class=1,
                          n_informative =18,
                          random_state=SEED)

#
rskf = RepeatedStratifiedKFold(n_splits=3, n_repeats=2, random_state=SEED)

print("拆分前的數據")
print(pd.DataFrame(y).value_counts())


for train_idx, test_idx in rskf.split(X,y):
    print("訓練集數據:\n",pd.DataFrame(y[train_idx]).value_counts())
    print("測試集數據:\n",pd.DataFrame(y[test_idx]).value_counts())

2.2.7 隨機拆分--ShuffleSplit

# ShuffleSplit
# 相比於K折拆分，ShuffleSplit可指定拆分數據集的次數及每次拆分數據集的測試集比例
# 可指定拆分次數和測試集比例，需要指定random_state才可以復現數據

import numpy as np
import pandas as pd
from sklearn.model_selection import *
from sklearn.datasets import make_classification


SEED = 666

X,y = make_classification(n_samples=100, 
                          n_features=20, 
                          shuffle=True, 
                          n_classes=6,
                          n_clusters_per_class=1,
                          n_informative =18,
                          weights=[0.1,0.5,0.1,0.1,0.1,0.1],
                          random_state=SEED)


#
ss = ShuffleSplit(n_splits=10, test_size=None, train_size=None, random_state=None)
print("拆分前的數據")
print(pd.DataFrame(y).value_counts())



# 完全是按照groups的參數進行的拆分
for train_idx, test_idx in ss.split(X, y):
    print("拆分--------------------------------------------------")
    print("訓練集數據:\n",pd.DataFrame(y[train_idx]).value_counts())
    print("測試集數據:\n",pd.DataFrame(y[test_idx]).value_counts())

2.2.8 隨機拆分--GroupShuffleSplit

# GroupShuffleSplit

# 可指定拆分次數和測試集比例，需要傳入groups，按照分組拆分

import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
from sklearn.model_selection import GroupShuffleSplit
from sklearn.datasets import make_classification


SEED = 666

X,y = make_classification(n_samples=100, 
                          n_features=20, 
                          shuffle=True, 
                          n_classes=4,
                          n_clusters_per_class=1,
                          n_informative =18,
                          weights=[0.1,0.6,0.2,0.1],
                          random_state=SEED)


#
gss = GroupShuffleSplit(n_splits=5, test_size=0.2, train_size=None, random_state=SEED)
print("拆分前的數據")
print(pd.DataFrame(y).value_counts())

# 完全是按照groups的參數進行的拆分
for train_idx, test_idx in gss.split(X, y, groups=y):
    print("拆分--------------------------------------------------")
    print("訓練集數據:\n",pd.DataFrame(y[train_idx]).value_counts())
    print("測試集數據:\n",pd.DataFrame(y[test_idx]).value_counts())

2.2.9 隨機拆分--StratifiedShuffleSplit

# StratifiedShuffleSplit

# 可指定拆分次數和測試集比例，需要傳入X、y，在划分后的數據集中y標簽比例相似

import numpy as np
import pandas as pd
from sklearn.model_selection import *
from sklearn.datasets import make_classification


SEED = 666

X,y = make_classification(n_samples=200, 
                          n_features=20, 
                          shuffle=True, 
                          n_classes=3,
                          n_clusters_per_class=1,
                          n_informative =18,
                          random_state=SEED)

skf = StratifiedShuffleSplit(n_splits=3, test_size=None, train_size=None, random_state=SEED)

print("拆分前的數據")
print(pd.DataFrame(y).value_counts())
for train_idx, test_idx in skf.split(X,y):
    print("拆分--------------------------------------------------")
    print("訓練集數據:\n",pd.DataFrame(y[train_idx]).value_counts())
    print("測試集數據:\n",pd.DataFrame(y[test_idx]).value_counts())

2.2.10 留一法-- LeaveOneOut

#### LeaveOneOut
import numpy as np
import pandas as pd
from sklearn.model_selection import *
from sklearn.datasets import make_classification


SEED = 666

X,y = make_classification(n_samples=100, 
                          n_features=20, 
                          shuffle=True, 
                          n_classes=6,
                          n_clusters_per_class=1,
                          n_informative =18,
                          weights=[0.1,0.5,0.1,0.1,0.1,0.1],
                          random_state=SEED)


#
logo = LeaveOneOut()
print("拆分前的數據")
print(pd.DataFrame(y).value_counts())



# 完全是按照groups的參數進行的拆分
for train_idx, test_idx in logo.split(X, y):
    print("拆分--------------------------------------------------")
    print("訓練集數據:\n",pd.DataFrame(y[train_idx]).value_counts())
    print("測試集數據:\n",pd.DataFrame(y[test_idx]).value_counts())

2.2.11 留一法-- LeaveOneGroupOut

# LeaveOneGroupOut
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
from sklearn.model_selection import LeaveOneGroupOut
from sklearn.datasets import make_classification


SEED = 666

X,y = make_classification(n_samples=100, 
                          n_features=20, 
                          shuffle=True, 
                          n_classes=6,
                          n_clusters_per_class=1,
                          n_informative =18,
                          weights=[0.1,0.5,0.1,0.1,0.1,0.1],
                          random_state=SEED)


#
logo = LeaveOneGroupOut()
print("拆分前的數據")
print(pd.DataFrame(y).value_counts())




for train_idx, test_idx in logo.split(X, y, groups=y):
    print("拆分--------------------------------------------------")
    print("訓練集數據:\n",pd.DataFrame(y[train_idx]).value_counts())
    print("測試集數據:\n",pd.DataFrame(y[test_idx]).value_counts())

2.2.12 留一法-- LeavePOut

#### LeavePOut
import numpy as np
import pandas as pd
from sklearn.model_selection import *
from sklearn.datasets import make_classification


SEED = 666

X,y = make_classification(n_samples=100, 
                          n_features=20, 
                          shuffle=True, 
                          n_classes=6,
                          n_clusters_per_class=1,
                          n_informative =18,
                          weights=[0.1,0.5,0.1,0.1,0.1,0.1],
                          random_state=SEED)


#
logo = LeavePOut(10)
print("拆分前的數據")
print(pd.DataFrame(y).value_counts())



# 完全是按照groups的參數進行的拆分
for train_idx, test_idx in logo.split(X, y):
    print("拆分--------------------------------------------------")
    print("訓練集數據:\n",pd.DataFrame(y[train_idx]).value_counts())
    print("測試集數據:\n",pd.DataFrame(y[test_idx]).value_counts())

2.2.13 留一法-- LeavePGroupsOut

#  LeavePGroupsOut

import numpy as np
import pandas as pd
from sklearn.model_selection import *
from sklearn.datasets import make_classification


SEED = 666

X,y = make_classification(n_samples=100, 
                          n_features=20, 
                          shuffle=True, 
                          n_classes=6,
                          n_clusters_per_class=1,
                          n_informative =18,
                          weights=[0.1,0.5,0.1,0.1,0.1,0.1],
                          random_state=SEED)


#
logo = LeavePGroupsOut(2)
print("拆分前的數據")
print(pd.DataFrame(y).value_counts())




for train_idx, test_idx in logo.split(X, y, groups=y):
    print("拆分--------------------------------------------------")
    print("訓練集數據:\n",pd.DataFrame(y[train_idx]).value_counts())
    print("測試集數據:\n",pd.DataFrame(y[test_idx]).value_counts())

2.2.14 指定拆分--PredefinedSplit

# PredefinedSplit

#根據提前指定的分類來划分數據集，譬如說test_fold包含三類0、1、2，那么會拆分三次，每一次其中一類作為測試集，（-1對應的index永遠在訓練集）

import numpy as np
import pandas as pd
from sklearn.model_selection import *
from sklearn.datasets import make_classification


SEED = 666

X,y = make_classification(n_samples=100, 
                          n_features=20, 
                          shuffle=True, 
                          n_classes=6,
                          n_clusters_per_class=1,
                          n_informative =18,
                          weights=[0.1,0.5,0.1,0.1,0.1,0.1],
                          random_state=SEED)


#
test_fold = np.hstack((np.zeros(20),np.ones(40),np.ones(10)+1,np.zeros(30)-1)) #分為三類0，1，2,設置為-1的樣本永遠包含在測試集中
print(test_fold)
pres = PredefinedSplit(test_fold)



for train_idx, test_idx in pres.split():
    print("TRAIN:", train_idx, "TEST:", test_idx)

2.2.15 時間窗口拆分--TimeSeriesSplit

時間序列的拆分。

# TimeSeriesSplit
# 時間序列拆分，類似於滑動窗口模式，以前n個樣本作為訓練集，第n+1個樣本作為測試集
import numpy as np
import pandas as pd
from sklearn.model_selection import *



X = np.array([[1, 2], [3, 4], [1, 2], [3, 4], [1, 2], [3, 4]])
y = np.array([1, 2, 3, 4, 5, 6])

tss = TimeSeriesSplit(n_splits=5,  max_train_size=None, test_size=None, gap=0)

for train_idx, test_idx in tss.split(X):
    print("TRAIN:", train_idx, "TEST:", test_idx)
'''
TRAIN: [0] TEST: [1]
TRAIN: [0 1] TEST: [2]
TRAIN: [0 1 2] TEST: [3]
TRAIN: [0 1 2 3] TEST: [4]
TRAIN: [0 1 2 3 4] TEST: [5]
'''

2.2.x 附件筆記

# 15個
#---------------------------K折驗證------------------------------------
#K折交叉驗證，即將原數據集分成K份，每一折將其中一份作為測試集，其他k-1份作為訓練集

# 隨機的多折拆分（默認五折拆分）,shuffle=True會打亂訓練集測試集
KFold(n_splits=5, shuffle=True, random_state=SEED)
for train_index, test_index in aKFold.split(X)


# 簡單的多折拆分（默認五折拆分）,需要傳入groups，會根據傳入groups使得每個groups在訓練集測試集的比例不變
GroupKFold(n_splits=5)
for train_index, test_index in group_kfold.split(X, y, groups = y)

# 相比於KFold，在進行split的時候需要傳入y，並且會根據y的分類，保證分類后y在各個數據集中比例不變,類似於GroupKFold（基於參數groups）
StratifiedKFold(n_splits=5, shuffle=False, random_state=None)

# 在進行split的時候需要傳入X,y和groups，觀察其結果，似乎只取決於傳入的group，group的長度取決於X、y的長度，分類數最好與n_splits相同
StratifiedGroupKFold(n_splits=3, shuffle=False, random_state=None)

#重復n_repeats次n_splits折的KFold拆分，最后拆分的次數應該是n_splits*n_repeats
RepeatedKFold(n_splits=4, n_repeats=2, random_state=666)

# 重復n_repeats次n_splits折的StratifiedKFold拆分，最后拆分的次數應該是n_splits*n_repeats
RepeatedStratifiedKFold(n_splits=3, n_repeats=2, random_state=SEED)



# -------------------ShuffleSplit----------------------------------
# 相比於K折拆分，ShuffleSplit可指定拆分數據集的次數及每次拆分數據集的測試集比例
# 可指定拆分次數和測試集比例，需要指定random_state才可以復現數據
ShuffleSplit(n_splits=5, test_size=0.25, train_size=None, random_state=666)

# 可指定拆分次數和測試集比例，需要傳入X、y，在划分后的數據集中y標簽比例相似
StratifiedShuffleSplit(n_splits=3, test_size=None, train_size=None, random_state=SEED)


# 可指定拆分次數和測試集比例，需要傳入groups，按照分組拆分
GroupShuffleSplit(n_splits=5, test_size=None, train_size=None, random_state=None)
for train_idx, test_idx in gss.split(X, y=None, groups=y)

# -------------------------留一法-----------------------------------------
# 留一法及其拓展留P法，即指定1(或者P)個樣本（或組）作為測試集，其他樣本（或組）做為訓練集，拆分數由樣本數決定，不必指定
# 隨機拆分的留一法，每次只會保留一個樣本作為測試集，樣本數為n則默認進行n-1次拆分
LeaveOneOut()
for train_idx, test_idx in logo.split(X)

# 按組拆分的留一法，按照傳入的groups分組，然后根據分組進行留一拆分
LeaveOneGroupOut()
for train_idx, test_idx in logo.split(X, y, groups=y):

 # 留一法的拓展，LeavePOut(1)與LeaveOneOut()是一樣的
LeavePOut(p)

# 留一法的拓展，LeavePGroupsOut(1)與LeaveOneGroupOut()是一樣的
LeavePGroupsOut(p)


#--------------------------指定拆分-----------------------------------------
#根據提前指定的分類來划分數據集，譬如說test_fold包含三類0、1、2，那么會拆分三次，每一次其中一類作為測試集，（-1對應的index永遠在訓練集）
test_fold = np.hstack((np.zeros(20),np.ones(40),np.ones(10)+1,np.zeros(30)-1)) #分為三類0，1，2,設置為-1的樣本永遠包含在測試集中
pres = PredefinedSplit(test_fold)


# -------------------------時間序列拆分-----------------------------------
# 時間序列拆分，類似於滑動窗口模式，以前n個樣本作為訓練集，第n+1個樣本作為測試集
TimeSeriesSplit(n_splits=5,  max_train_size=None, test_size=None, gap=0)

2.3 Model validation 模型驗證

2.3.1 cross_val_score

cross_val_score是最簡單的模型驗證的方法，可以傳入需要驗證的模型estimator，數據集X，數據標簽列y，可自定義交叉驗證數據集拆分規則cv，也可以自定義返回的分數計算方式scoring。

scoring可自定義，或者參照3.3.1. 該scoring參數：定義模型評估規則傳入對應分數計算的名稱，這里列出來了常用的一些指標。

分類問題中，除了准確率、精確率、召回率，二分類常用'f1'，多分類常用'f1_micro'和'f1_macro'。

結果只會返回一個numpy.ndarray，即模型驗證得分。

import numpy as np
import pandas as pd
from sklearn.model_selection import *
from sklearn.svm import *
from sklearn.datasets import load_iris


iris_data_bunch = load_iris()
X = iris_data_bunch.data
y = iris_data_bunch.target

'''
cross_val_score(
    estimator, #一個支撐了fit方法的estimator
    X, #數據特征集
    y=None, #數據標簽列
    groups=None, #groups參數用於傳遞給拆分數據集的split方法
    scoring=None, #可自定義scorer(estimator, X, y)方法，或者某個字符串，參考官方文檔
    cv=None, #數據集拆分參數，默認 KFold或StratifiedKFold策略(是否傳入y、groups)，可自己傳入對應的拆分器后者自定義拆分器
    n_jobs=None, 
    verbose=0, 
    fit_params=None, 
    pre_dispatch='2*n_jobs', 
    error_score=nan
)
'''

clf = SVC(kernel='linear', C=1, random_state=666)
scores = cross_val_score(clf, X, y, cv=5,scoring='f1_micro')
# 輸出五折交叉驗證的每一折的分數,numpy.ndarray
scores

2.3.2 cross_validate 函數和多指標評估

該cross_validate功能的區別在於cross_val_score兩個方面：

它允許指定多個評估指標。
除了測試分數之外，它還返回一個包含擬合時間、分數時間（以及可選的訓練分數和擬合估計量）的字典。

分類問題的評價指標是准確率，那么回歸算法的評價指標就是MSE，RMSE，MAE、R-Squared，scroing依舊參照3.3.1. 該scoring參數：定義模型評估規則。

import numpy as np
import pandas as pd
from sklearn.model_selection import *
from sklearn.svm import *
from sklearn.datasets import load_diabetes

diabetes_data_bunch = load_diabetes()
X = diabetes_data_bunch.data
y = diabetes_data_bunch.target


'''
cross_validate(
    estimator, #一個支撐了fit方法的estimator
    X, #數據特征集
    y=None, #數據標簽列
    groups=None, #groups參數用於傳遞給拆分數據集的split方法
    scoring=None, #可自定義scorer(estimator, X, y)方法，或者某個字符串，參考官方文檔，【可傳入多個分數】
    cv=None, #數據集拆分參數，默認 KFold或StratifiedKFold策略(是否傳入y、groups)，可自己傳入對應的拆分器后者自定義拆分器
    n_jobs=None, 
    verbose=0, 
    fit_params=None, 
    pre_dispatch='2*n_jobs', 
    return_train_score=False, #是否返回訓練集的分數
    return_estimator=False, #是否返回每一折訓練后的模型
    error_score=nan
)

'''

scoring = ['neg_mean_squared_error','neg_root_mean_squared_error','neg_mean_absolute_error','r2']

clf = SVR(kernel='linear', C=1)

scores = cross_validate(clf, X, y, cv=5,scoring=scoring,return_train_score=True,return_estimator=True)

scores

2.3.3 cross_val_predict 通過交叉驗證獲得預測結果

該功能cross_val_predict適用於：

從不同模型獲得的預測的可視化。
模型混合：當一個監督估計器的預測用於在集成方法中訓練另一個估計器時。

import numpy as np
import pandas as pd
from sklearn.model_selection import *
from sklearn.svm import *
from sklearn.datasets import load_iris
iris_data_bunch = load_iris()
X = iris_data_bunch.data
y = iris_data_bunch.target

'''
cross_val_predict(
    estimator, #一個支撐了fit方法的estimator
    X, #數據特征集
    y=None, #數據標簽列
    groups=None, #groups參數用於傳遞給拆分數據集的split方法
    cv=None, #數據集拆分參數，默認 KFold或StratifiedKFold策略(是否傳入y、groups)，可自己傳入對應的拆分器后者自定義拆分器
    n_jobs=None, 
    verbose=0, 
    fit_params=None, 
    pre_dispatch='2*n_jobs', 
    method='predict' #{'predict', 'predict_proba', 'predict_log_proba','decision_function'}
)
'''

clf = SVC(kernel='linear', C=1, random_state=666)
y_pred = cross_val_predict(clf, X, y, cv=5)
# 輸出對訓練集的預測結果
y_pred

2.3.4 validation_curve 驗證曲線

繪制驗證曲線有助於觀察隨着參數變化，訓練集測試集分數的變化。

import numpy as np
from sklearn.model_selection import validation_curve
from sklearn.datasets import load_iris
from sklearn.linear_model import Ridge

'''
validation_curve(
    estimator, 
    X, 
    y, 
    param_name, 
    param_range, 
    groups=None, 
    cv=None, 
    scoring=None, 
    n_jobs=None, 
    pre_dispatch='all', 
    verbose=0, 
    error_score=nan, 
    fit_params=None
)
'''
np.random.seed(0)
X, y = load_iris(return_X_y=True)
indices = np.arange(y.shape[0])
np.random.shuffle(indices)
X, y = X[indices], y[indices]

param_name = 'alpha'
param_range = np.logspace(-10, 1, 10)

train_scores, valid_scores = validation_curve(
    Ridge(), X, y, param_name=param_name,param_range=param_range,
    cv=5)

print('參數：',param_range)

print('train_scores:',np.average(train_scores, axis=1))

print("valid_scores:",np.average(valid_scores, axis=1))

import matplotlib.pyplot as plt
plt.plot(param_range,np.average(train_scores, axis=1))
plt.plot(param_range,np.average(valid_scores, axis=1))
plt.show()

2.3.5 learning_curve 學習曲線

learning_curve 繪制隨着訓練數據變化的訓練結果情況。

from sklearn.model_selection import learning_curve
from sklearn.svm import SVC
from sklearn.datasets import load_iris


np.random.seed(0)
X, y = load_iris(return_X_y=True)
indices = np.arange(y.shape[0])
np.random.shuffle(indices)
X, y = X[indices], y[indices]

train_sizes = [x for x in range(10,120)]

train_sizes, train_scores, valid_scores = learning_curve(
    SVC(kernel='linear'), X, y, train_sizes=train_sizes, cv=5,scoring='f1_micro')

#print('訓練數據量:',train_sizes)

#print('train_scores:',np.average(train_scores, axis=1))
#print('valid_scores:',np.average(train_scores, axis=1))

import matplotlib.pyplot as plt
plt.plot(train_sizes,np.average(train_scores, axis=1),label='train_scores')
plt.plot(train_sizes,np.average(valid_scores, axis=1),label='valid_scores')
plt.legend(loc="right")
plt.show()

2.4 Hyper-parameter optimizers 超參數優化器

scikit-learn 中提供了兩種通用的參數搜索方法：對於給定的值，GridSearchCV詳盡地考慮所有參數組合，同時RandomizedSearchCV可以從具有指定分布的參數空間中采樣給定數量的候選者。這兩個工具都有連續的減半對應物 HalvingGridSearchCV和HalvingRandomSearchCV，可以更快地找到一個好的參數組合。

2.4.1 GridSearchCV 窮舉網格搜索

使用例子參考cross_val_score 和 GridSearchCV 上的多指標評估演示GridSearchCV

# GridSearchCV 
'''
GridSearchCV(estimator, 
             param_grid, #參數字典
             scoring=None, #評估指標
             n_jobs=None,
             refit=True, 
             cv=None, #交叉驗證折數
             verbose=0, 
             pre_dispatch='2*n_jobs', 
             error_score=nan, 
             return_train_score=False
            )
'''
# Author: Raghav RV <rvraghav93@gmail.com>
# License: BSD

import numpy as np
from matplotlib import pyplot as plt

from sklearn.datasets import make_hastie_10_2
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import make_scorer
from sklearn.metrics import accuracy_score
from sklearn.tree import DecisionTreeClassifier


# 二分類問題
X, y = make_hastie_10_2(n_samples=8000, random_state=666)

# The scorers can be either one of the predefined metric strings or a scorer
# callable, like the one returned by make_scorer
scoring = {"AUC": "roc_auc", "Accuracy": make_scorer(accuracy_score)}

# Setting refit='AUC', refits an estimator on the whole dataset with the
# parameter setting that has the best cross-validated AUC score.
# That estimator is made available at ``gs.best_estimator_`` along with
# parameters like ``gs.best_score_``, ``gs.best_params_`` and
# ``gs.best_index_``
gs = GridSearchCV(
    DecisionTreeClassifier(random_state=42),
    param_grid={"min_samples_split": range(2, 403, 10)},
    scoring=scoring,
    refit="AUC",
    return_train_score=True,
)
gs.fit(X, y)
results = gs.cv_results_

# 繪制結果
plt.figure(figsize=(13, 13))
plt.title("GridSearchCV evaluating using multiple scorers simultaneously", fontsize=16)

plt.xlabel("min_samples_split")
plt.ylabel("Score")

# 挪動坐標軸
ax = plt.gca()
ax.set_xlim(0, 402)
ax.set_ylim(0.73, 1)

# Get the regular numpy array from the MaskedArray
X_axis = np.array(results["param_min_samples_split"].data, dtype=float)

for scorer, color in zip(sorted(scoring), ["g", "k"]):
    for sample, style in (("train", "--"), ("test", "-")):
        sample_score_mean = results["mean_%s_%s" % (sample, scorer)]
        sample_score_std = results["std_%s_%s" % (sample, scorer)]
        ax.fill_between(
            X_axis,
            sample_score_mean - sample_score_std,
            sample_score_mean + sample_score_std,
            alpha=0.1 if sample == "test" else 0,
            color=color,
        )
        ax.plot(
            X_axis,
            sample_score_mean,
            style,
            color=color,
            alpha=1 if sample == "test" else 0.7,
            label="%s (%s)" % (scorer, sample),
        )

    best_index = np.nonzero(results["rank_test_%s" % scorer] == 1)[0][0]
    best_score = results["mean_test_%s" % scorer][best_index]

    # Plot a dotted vertical line at the best score for that scorer marked by x
    ax.plot(
        [
            X_axis[best_index],
        ]
        * 2,
        [0, best_score],
        linestyle="-.",
        color=color,
        marker="x",
        markeredgewidth=3,
        ms=8,
    )

    # Annotate the best score for that scorer
    ax.annotate("%0.2f" % best_score, (X_axis[best_index], best_score + 0.005))

plt.legend(loc="best")
plt.grid(False)
plt.show()

2.4.2 RandomizedSearchCV 隨機參數優化

雖然使用參數設置網格是目前最廣泛使用的參數優化方法，但其他搜索方法具有更有利的特性。 RandomizedSearchCV實現對參數的隨機搜索，其中每個設置都是從可能參數值的分布中采樣的。與詳盡搜索相比，這有兩個主要好處：

可以獨立於參數的數量和可能的值來選擇預算。
添加不影響性能的參數不會降低效率。

指定應該如何采樣參數是使用字典完成的，非常類似於為指定參數GridSearchCV。此外，使用n_iter參數指定計算預算，即采樣候選或采樣迭代的數量。對於每個參數，可以指定可能值的分布或離散選擇列表（將被均勻采樣）

tip:scipy.stats模塊，該模塊包含了許多有用的分布進行采樣的參數，例如expon，gamma， uniform或randint。

例子參考：比較用於超參數估計的隨機搜索和網格搜索比較了隨機搜索和網格搜索

# RandomizedSearchCV
'''
RandomizedSearchCV(estimator, 
                   param_distributions,  
                   n_iter=10, 
                   scoring=None, 
                   n_jobs=None, 
                   refit=True, 
                   cv=None, 
                   verbose=0, 
                   pre_dispatch='2*n_jobs', 
                   random_state=None, 
                   error_score=nan, 
                   return_train_score=False
)
'''
import numpy as np

from time import time
import scipy.stats as stats
from sklearn.utils.fixes import loguniform

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.datasets import load_digits
from sklearn.linear_model import SGDClassifier

# get some data
X, y = load_digits(return_X_y=True)

# build a classifier
clf = SGDClassifier(loss="hinge", penalty="elasticnet", fit_intercept=True)


# Utility function to report best scores
def report(results, n_top=3):
    for i in range(1, n_top + 1):
        candidates = np.flatnonzero(results["rank_test_score"] == i)
        for candidate in candidates:
            print("Model with rank: {0}".format(i))
            print(
                "Mean validation score: {0:.3f} (std: {1:.3f})".format(
                    results["mean_test_score"][candidate],
                    results["std_test_score"][candidate],
                )
            )
            print("Parameters: {0}".format(results["params"][candidate]))
            print("")


# specify parameters and distributions to sample from
param_dist = {
    "average": [True, False],
    "l1_ratio": stats.uniform(0, 1),
    "alpha": loguniform(1e-4, 1e0),
}

# run randomized search
n_iter_search = 20
random_search = RandomizedSearchCV(
    clf, param_distributions=param_dist, n_iter=n_iter_search
)

start = time()
random_search.fit(X, y)
print(
    "RandomizedSearchCV took %.2f seconds for %d candidates parameter settings."
    % ((time() - start), n_iter_search)
)
report(random_search.cv_results_)

# use a full grid over all parameters
param_grid = {
    "average": [True, False],
    "l1_ratio": np.linspace(0, 1, num=10),
    "alpha": np.power(10, np.arange(-4, 1, dtype=float)),
}

# run grid search
grid_search = GridSearchCV(clf, param_grid=param_grid)
start = time()
grid_search.fit(X, y)

print(
    "GridSearchCV took %.2f seconds for %d candidate parameter settings."
    % (time() - start, len(grid_search.cv_results_["params"]))
)
report(grid_search.cv_results_)

2.4.3 HalvingGridSearchCV 連續減半

HalvingGridSearchCV和 HalvingRandomSearchCV估計器仍處於試驗階段：它們的預測和它們的 API 可能會在沒有任何棄用周期的情況下發生變化。要使用它們，您需要顯式導入enable_halving_search_cv：

例子參考：

網格搜索和連續減半的比較

# 2.4.3 HalvingGridSearchCV 連續減半
'''
HalvingGridSearchCV(
    estimator, 
    param_grid, 
    factor=3, 
    resource='n_samples', 
    max_resources='auto', 
    min_resources='exhaust', 
    aggressive_elimination=False, 
    cv=5, 
    scoring=None, 
    refit=True, 
    error_score=nan, 
    return_train_score=True, 
    random_state=None, 
    n_jobs=None, 
    verbose=0
)
'''
from time import time

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

from sklearn.svm import SVC
from sklearn import datasets
from sklearn.model_selection import GridSearchCV
from sklearn.experimental import enable_halving_search_cv  # noqa
from sklearn.model_selection import HalvingGridSearchCV



rng = np.random.RandomState(0)
X, y = datasets.make_classification(n_samples=1000, random_state=rng)

gammas = [1e-1, 1e-2, 1e-3, 1e-4, 1e-5, 1e-6, 1e-7]
Cs = [1, 10, 100, 1e3, 1e4, 1e5]
param_grid = {"gamma": gammas, "C": Cs}

clf = SVC(random_state=rng)

tic = time()
gsh = HalvingGridSearchCV(
    estimator=clf, param_grid=param_grid, factor=2, random_state=rng
)
gsh.fit(X, y)
gsh_time = time() - tic

tic = time()
gs = GridSearchCV(estimator=clf, param_grid=param_grid)
gs.fit(X, y)
gs_time = time() - tic

def make_heatmap(ax, gs, is_sh=False, make_cbar=False):
    """Helper to make a heatmap."""
    results = pd.DataFrame.from_dict(gs.cv_results_)
    results["params_str"] = results.params.apply(str)
    if is_sh:
        # SH dataframe: get mean_test_score values for the highest iter
        scores_matrix = results.sort_values("iter").pivot_table(
            index="param_gamma",
            columns="param_C",
            values="mean_test_score",
            aggfunc="last",
        )
    else:
        scores_matrix = results.pivot(
            index="param_gamma", columns="param_C", values="mean_test_score"
        )

    im = ax.imshow(scores_matrix)

    ax.set_xticks(np.arange(len(Cs)))
    ax.set_xticklabels(["{:.0E}".format(x) for x in Cs])
    ax.set_xlabel("C", fontsize=15)

    ax.set_yticks(np.arange(len(gammas)))
    ax.set_yticklabels(["{:.0E}".format(x) for x in gammas])
    ax.set_ylabel("gamma", fontsize=15)

    # Rotate the tick labels and set their alignment.
    plt.setp(ax.get_xticklabels(), rotation=45, ha="right", rotation_mode="anchor")

    if is_sh:
        iterations = results.pivot_table(
            index="param_gamma", columns="param_C", values="iter", aggfunc="max"
        ).values
        for i in range(len(gammas)):
            for j in range(len(Cs)):
                ax.text(
                    j,
                    i,
                    iterations[i, j],
                    ha="center",
                    va="center",
                    color="w",
                    fontsize=20,
                )

    if make_cbar:
        fig.subplots_adjust(right=0.8)
        cbar_ax = fig.add_axes([0.85, 0.15, 0.05, 0.7])
        fig.colorbar(im, cax=cbar_ax)
        cbar_ax.set_ylabel("mean_test_score", rotation=-90, va="bottom", fontsize=15)


fig, axes = plt.subplots(ncols=2, sharey=True)
ax1, ax2 = axes

make_heatmap(ax1, gsh, is_sh=True)
make_heatmap(ax2, gs, make_cbar=True)

ax1.set_title("Successive Halving\ntime = {:.3f}s".format(gsh_time), fontsize=15)
ax2.set_title("GridSearch\ntime = {:.3f}s".format(gs_time), fontsize=15)

plt.show()

2.4.4 HalvingRandomSearchCV 連續減半

連續減半迭代

# HalvingRandomSearchCV 連續減半

import pandas as pd
from sklearn import datasets
import matplotlib.pyplot as plt
from scipy.stats import randint
import numpy as np

from sklearn.experimental import enable_halving_search_cv  # noqa
from sklearn.model_selection import HalvingRandomSearchCV
from sklearn.ensemble import RandomForestClassifier

rng = np.random.RandomState(0)

X, y = datasets.make_classification(n_samples=700, random_state=rng)

clf = RandomForestClassifier(n_estimators=20, random_state=rng)

param_dist = {
    "max_depth": [3, None],
    "max_features": randint(1, 11),
    "min_samples_split": randint(2, 11),
    "bootstrap": [True, False],
    "criterion": ["gini", "entropy"],
}

rsh = HalvingRandomSearchCV(
    estimator=clf, param_distributions=param_dist, factor=2, random_state=rng
)
rsh.fit(X, y)

results = pd.DataFrame(rsh.cv_results_)
results["params_str"] = results.params.apply(str)
results.drop_duplicates(subset=("params_str", "iter"), inplace=True)
mean_scores = results.pivot(
    index="iter", columns="params_str", values="mean_test_score"
)
ax = mean_scores.plot(legend=False, alpha=0.6)

labels = [
    f"iter={i}\nn_samples={rsh.n_resources_[i]}\nn_candidates={rsh.n_candidates_[i]}"
    for i in range(rsh.n_iterations_)
]

ax.set_xticks(range(rsh.n_iterations_))
ax.set_xticklabels(labels, rotation=45, multialignment="left")
ax.set_title("Scores of candidates over iterations")
ax.set_ylabel("mean test score", fontsize=15)
ax.set_xlabel("iterations", fontsize=15)
plt.tight_layout()
plt.show()

2.4.5 ParameterGrid 參數網格

用於生成參數組合的方法。

# 2.4.5 ParameterGrid 參數網格
# ParameterGrid(param_grid)

from sklearn.model_selection import ParameterGrid
param_grid = {'a': [1, 2], 'b': [True, False]}
print(list(ParameterGrid(param_grid)) == (
   [{'a': 1, 'b': True}, {'a': 1, 'b': False},
    {'a': 2, 'b': True}, {'a': 2, 'b': False}]))

grid = [{'kernel': ['linear']}, {'kernel': ['rbf'], 'gamma': [1, 10]}]

print(list(ParameterGrid(grid)) == [{'kernel': 'linear'},
                              {'kernel': 'rbf', 'gamma': 1},
                              {'kernel': 'rbf', 'gamma': 10}])

print(ParameterGrid(grid)[1] == {'kernel': 'rbf', 'gamma': 1})

2..4.6 ParameterSampler 參數生成器

# 2..4.6 ParameterSampler 參數生成器

'''
 ParameterSampler(param_distributions, n_iter, *, random_state=None)
'''

from sklearn.model_selection import ParameterSampler
from scipy.stats.distributions import expon
import numpy as np
rng = np.random.RandomState(666)
param_grid = {'a':[1, 2], 'b': expon()}
param_list = list(ParameterSampler(param_grid, n_iter=4,
                                   random_state=rng))
rounded_list = [dict((k, round(v, 6)) for (k, v) in d.items())
                for d in param_list]

print(param_list) 
print(rounded_list)

2.X 參考

2.x2 附件：全部api

SPLITTER CLASSES 拆分器類
model_selection.KFold	K折交叉驗證器
model_selection.GroupKFold	具有非重疊組的 K 折迭代器變體。
model_selection.ShuffleSplit	隨機排列交叉驗證器
model_selection.GroupShuffleSplit	Shuffle-Group(s)-Out 交叉驗證迭代器
model_selection.LeaveOneOut	留一法交叉驗證器
model_selection.LeaveOneGroupOut	留一組交叉驗證器
model_selection.LeavePOut	留P法交叉驗證器
model_selection.LeavePGroupsOut	留P組交叉驗證器
model_selection.PredefinedSplit	預定義的拆分交叉驗證器
model_selection.RepeatedKFold	重復 K 折交叉驗證器。
model_selection.RepeatedStratifiedKFold	重復分層 K 折交叉驗證器。
model_selection.StratifiedKFold	分層 K 折交叉驗證器。
model_selection.StratifiedShuffleSplit	分層 ShuffleSplit 交叉驗證器
model_selection.StratifiedGroupKFold	具有非重疊組的分層 K-Folds 迭代器變體。
model_selection.TimeSeriesSplit	時間序列交叉驗證器

Splitter Functions 拆分器功能
model_selection.check_cv	用於構建交叉驗證器的輸入檢查器實用程序
model_selection.train_test_split	將數組或矩陣拆分為隨機訓練和測試子集

Hyper-parameter optimizers 超參數優化器
model_selection.GridSearchCV	對估計器的指定參數值進行詳盡搜索。
model_selection.HalvingGridSearchCV	使用連續減半搜索指定的參數值。
model_selection.ParameterGrid	參數網格，每個參數具有離散數量的值。
model_selection.ParameterSampler	從給定分布中采樣的參數生成器。
model_selection.RandomizedSearchCV	對超參數的隨機搜索。
model_selection.HalvingRandomSearchCV	對超參數的隨機搜索。

Model validation 模型驗證
model_selection.cross_validate	通過交叉驗證評估指標並記錄擬合/得分時間。
model_selection.cross_val_predict	為每個輸入數據點生成交叉驗證的估計
model_selection.cross_val_score	通過交叉驗證評估分數
model_selection.learning_curve	學習曲線。
model_selection.permutation_test_score	評估具有排列的交叉驗證分數的重要性
model_selection.validation_curve	驗證曲線。

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 sklearn.model_selection 機器學習筆記：sklearn.model_selection.train_test_split切分訓練、測試集 sklearn.model_selection 的train_test_split方法和參數機器學習第4篇：sklearn 最鄰近算法概述機器學習分類實例(sklearn)——SVM Python機器學習庫sklearn的安裝機器學習算法的sklearn實現機器學習-特征工程-Feature generation 和 Feature selection 機器學習之特征選擇（Feature Selection）機器學習模型優化 ---- Model Optimization