python利用決策樹進行特征選擇

本文轉載自查看原文 2018-03-23 18:49 7782 機器學習

python利用決策樹進行特征選擇（注釋部分為繪圖功能），最后輸出特征排序：

import numpy as np
import tflearn
from tflearn.layers.core import dropout
from tflearn.layers.normalization import batch_normalization
from tflearn.data_utils import to_categorical
from sklearn.model_selection import train_test_split
import sys 
import pandas as pd
from pandas import Series,DataFrame
import matplotlib.pyplot as plt 

data_train = pd.read_csv("feature_with_dnn_todo2.dat")
print(data_train.info())
import matplotlib.pyplot as plt
print(data_train.columns)

"""
for col in data_train.columns[1:]:
    fig = plt.figure(figsize=(20, 16), dpi=8)
    fig.set(alpha=0.2)
    plt.figure()
    data_train[data_train.label == 0.0][col].plot()
    data_train[data_train.label == 1.0][col].plot()
    data_train[data_train.label == 2.0][col].plot()
    data_train[data_train.label == 3.0][col].plot()
    data_train[data_train.label == 4.0][col].plot()
    plt.xlabel(u"sample data id")
    plt.ylabel(col) 
    plt.title(col)
    plt.legend((u'white', u'cdn',u'tunnel', u"msad", "todo"),loc='best')
    plt.show()
"""    

from sklearn.ensemble import ExtraTreesClassifier
X = data_train.iloc[:,1:] 
y = data_train['label'].tolist()


print(X.columns)

X = X.values.tolist()
print(X[-3:])
print("-------------")
print(y[-3:])

# preprocess data
from sklearn.preprocessing import StandardScaler
#X = StandardScaler().fit_transform(X)
from sklearn.preprocessing import MinMaxScaler
X = MinMaxScaler().fit_transform(X)
from sklearn.preprocessing import Normalizer
#X=Normalizer().fit_transform(X)

# abnormal data process
"""
for i,n in enumerate(y):
    if n == 4.0:
        y[i]=0
"""

import collections
print(collections.Counter(y))

print("just change to 2 classify !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
for i,n in enumerate(y):
    if n != 2.0:
        y[i]=0
    else:
        y[i]=1
print(collections.Counter(y))
#print(X.shape)

from imblearn.over_sampling import SMOTE 
X, y = SMOTE().fit_sample(X, y)
print(sorted(collections.Counter(y).items()))


import sys
clf = ExtraTreesClassifier()
print(dir(clf))
X_new = clf.fit(X, y) 
print (clf.feature_importances_ )
names = [u'flow_cnt', u'len(srcip_arr)', u'len(dstip_arr)', u'subdomain_num',
       u'uniq_subdomain_ratio', u'np.average(dns_request_len_arr)',
       u'np.average(dns_reply_len_arr)', u'np.average(subdomain_tag_num_arr)',
       u'np.average(subdomain_len_arr)',
       u'np.average(subdomain_weird_len_arr)',
       u'np.average(subdomain_entropy_arr)', u'A_rr_type_ratio',
       u'incommon_rr_type_rato', u'valid_ipv4_ratio', u'uniq_valid_ipv4_ratio',
       u'request_reply_ratio', u'np.max(dns_request_len_arr)',
       u'np.max(dns_reply_len_arr)', u'np.max(subdomain_tag_num_arr)',
       u'np.max(subdomain_len_arr)', u'np.max(subdomain_weird_len_arr)',
       u'np.max(subdomain_entropy_arr)', u'avg_distance', u'std_distance']
print "Features sorted by their score:"
print sorted(zip(clf.feature_importances_, names), reverse=True)

其中，

from imblearn.over_sampling import SMOTE 
X, y = SMOTE().fit_sample(X, y) print(sorted(collections.Counter(y).items()))

是使用smote算法補齊樣本不均衡的情況。
加如下代碼可以看score！
from sklearn.cross_validation import cross_val_score
scores = cross_val_score(clf, X, y)
print(scores.mean())

官方文檔：

1.13. Feature selection

The classes in the sklearn.feature_selection module can be used for feature selection/dimensionality reduction on sample sets, either to improve estimators’ accuracy scores or to boost their performance on very high-dimensional datasets.

1.13.1. Removing features with low variance

VarianceThreshold is a simple baseline approach to feature selection. It removes all features whose variance doesn’t meet some threshold. By default, it removes all zero-variance features, i.e. features that have the same value in all samples.

As an example, suppose that we have a dataset with boolean features, and we want to remove all features that are either one or zero (on or off) in more than 80% of the samples. Boolean features are Bernoulli random variables, and the variance of such variables is given by

$\mathrm{Var}[X] = p(1 - p)$

so we can select using the threshold .8 * (1 - .8):

 
           >>> 
           >>> from sklearn.feature_selection import VarianceThreshold >>> X = [[0, 0, 1], [0, 1, 0], [1, 0, 0], [0, 1, 1], [0, 1, 0], [0, 1, 1]] >>> sel = VarianceThreshold(threshold=(.8 * (1 - .8))) >>> sel.fit_transform(X) array([[0, 1],  [1, 0],  [0, 0],  [1, 1],  [1, 0],  [1, 1]])

As expected, VarianceThreshold has removed the first column, which has a probability $p = 5/6 > .8$ of containing a zero.

1.13.2. Univariate feature selection

Univariate feature selection works by selecting the best features based on univariate statistical tests. It can be seen as a preprocessing step to an estimator. Scikit-learn exposes feature selection routines as objects that implement the transform method:

SelectKBest removes all but the $k$ highest scoring features

SelectPercentile removes all but a user-specified highest scoring percentage of features

using common univariate statistical tests for each feature: false positive rate SelectFpr, false discovery rate SelectFdr, or family wise error SelectFwe.

GenericUnivariateSelect allows to perform univariate feature selection with a configurable strategy. This allows to select the best univariate selection strategy with hyper-parameter search estimator.

For instance, we can perform a $\chi^2$ test to the samples to retrieve only the two best features as follows:

 
           >>> 
           >>> from sklearn.datasets import load_iris >>> from sklearn.feature_selection import SelectKBest >>> from sklearn.feature_selection import chi2 >>> iris = load_iris() >>> X, y = iris.data, iris.target >>> X.shape (150, 4) >>> X_new = SelectKBest(chi2, k=2).fit_transform(X, y) >>> X_new.shape (150, 2)

These objects take as input a scoring function that returns univariate scores and p-values (or only scores for SelectKBest and SelectPercentile):

For regression: f_regression, mutual_info_regression

For classification: chi2, f_classif, mutual_info_classif

The methods based on F-test estimate the degree of linear dependency between two random variables. On the other hand, mutual information methods can capture any kind of statistical dependency, but being nonparametric, they require more samples for accurate estimation.

Feature selection with sparse data

If you use sparse data (i.e. data represented as sparse matrices), chi2, mutual_info_regression, mutual_info_classif will deal with the data without making it dense.

Warning

Beware not to use a regression scoring function with a classification problem, you will get useless results.

Examples:

1.13.3. Recursive feature elimination

Given an external estimator that assigns weights to features (e.g., the coefficients of a linear model), recursive feature elimination (RFE) is to select features by recursively considering smaller and smaller sets of features. First, the estimator is trained on the initial set of features and the importance of each feature is obtained either through a coef_ attribute or through a feature_importances_ attribute. Then, the least important features are pruned from current set of features.That procedure is recursively repeated on the pruned set until the desired number of features to select is eventually reached.

RFECV performs RFE in a cross-validation loop to find the optimal number of features.

Examples:

Recursive feature elimination: A recursive feature elimination example showing the relevance of pixels in a digit classification task.
Recursive feature elimination with cross-validation: A recursive feature elimination example with automatic tuning of the number of features selected with cross-validation.

1.13.4. Feature selection using SelectFromModel

SelectFromModel is a meta-transformer that can be used along with any estimator that has a coef_ or feature_importances_ attribute after fitting. The features are considered unimportant and removed, if the corresponding coef_ or feature_importances_ values are below the provided threshold parameter. Apart from specifying the threshold numerically, there are built-in heuristics for finding a threshold using a string argument. Available heuristics are “mean”, “median” and float multiples of these like “0.1*mean”.

For examples on how it is to be used refer to the sections below.

Examples

Feature selection using SelectFromModel and LassoCV: Selecting the two most important features from the Boston dataset without knowing the threshold beforehand.

1.13.4.1. L1-based feature selection

Linear models penalized with the L1 norm have sparse solutions: many of their estimated coefficients are zero. When the goal is to reduce the dimensionality of the data to use with another classifier, they can be used along with feature_selection.SelectFromModel to select the non-zero coefficients. In particular, sparse estimators useful for this purpose are the linear_model.Lasso for regression, and of linear_model.LogisticRegression and svm.LinearSVC for classification:

 
            >>> 
            >>> from sklearn.svm import LinearSVC >>> from sklearn.datasets import load_iris >>> from sklearn.feature_selection import SelectFromModel >>> iris = load_iris() >>> X, y = iris.data, iris.target >>> X.shape (150, 4) >>> lsvc = LinearSVC(C=0.01, penalty="l1", dual=False).fit(X, y) >>> model = SelectFromModel(lsvc, prefit=True) >>> X_new = model.transform(X) >>> X_new.shape (150, 3)

With SVMs and logistic-regression, the parameter C controls the sparsity: the smaller C the fewer features selected. With Lasso, the higher the alpha parameter, the fewer features selected.

Examples:

sphx_glr_auto_examples_text_document_classification_20newsgroups.py: Comparison of different algorithms for document classification including L1-based feature selection.

L1-recovery and compressive sensing

For a good choice of alpha, the Lasso can fully recover the exact set of non-zero variables using only few observations, provided certain specific conditions are met. In particular, the number of samples should be “sufficiently large”, or L1 models will perform at random, where “sufficiently large” depends on the number of non-zero coefficients, the logarithm of the number of features, the amount of noise, the smallest absolute value of non-zero coefficients, and the structure of the design matrix X. In addition, the design matrix must display certain specific properties, such as not being too correlated.

There is no general rule to select an alpha parameter for recovery of non-zero coefficients. It can by set by cross-validation (LassoCV or LassoLarsCV), though this may lead to under-penalized models: including a small number of non-relevant variables is not detrimental to prediction score. BIC (LassoLarsIC) tends, on the opposite, to set high values of alpha.

Reference Richard G. Baraniuk “Compressive Sensing”, IEEE Signal Processing Magazine [120] July 2007 http://users.isr.ist.utl.pt/~aguiar/CS_notes.pdf

1.13.4.2. Tree-based feature selection

Tree-based estimators (see the sklearn.tree module and forest of trees in the sklearn.ensemble module) can be used to compute feature importances, which in turn can be used to discard irrelevant features (when coupled with the sklearn.feature_selection.SelectFromModel meta-transformer):

 
            >>> 
            >>> from sklearn.ensemble import ExtraTreesClassifier >>> from sklearn.datasets import load_iris >>> from sklearn.feature_selection import SelectFromModel >>> iris = load_iris() >>> X, y = iris.data, iris.target >>> X.shape (150, 4) >>> clf = ExtraTreesClassifier() >>> clf = clf.fit(X, y) >>> clf.feature_importances_ array([ 0.04..., 0.05..., 0.4..., 0.4...]) >>> model = SelectFromModel(clf, prefit=True) >>> X_new = model.transform(X) >>> X_new.shape  (150, 2)

Examples:

Feature importances with forests of trees: example on synthetic data showing the recovery of the actually meaningful features.
Pixel importances with a parallel forest of trees: example on face recognition data.





參考：https://blog.csdn.net/code_caq/article/details/74066899

sklearn中實現如下：

from sklearn.datasets import load_boston from sklearn.ensemble import RandomForestRegressor import numpy as np #Load boston housing dataset as an example boston = load_boston() X = boston["data"] print type(X),X.shape Y = boston["target"] names = boston["feature_names"] print names rf = RandomForestRegressor() rf.fit(X, Y) print "Features sorted by their score:" print sorted(zip(map(lambda x: round(x, 4), rf.feature_importances_), names), reverse=True)

結果如下：

Features sorted by their score: [(0.5104, 'RM'), (0.2837, 'LSTAT'), (0.0812, 'DIS'), (0.0303, 'CRIM'), (0.0294, 'NOX'), (0.0176, 'PTRATIO'), (0.0134, 'AGE'), (0.0115, 'B'), (0.0089, 'TAX'), (0.0077, 'INDUS'), (0.0051, 'RAD'), (0.0006, 'ZN'), (0.0004, 'CHAS')]


from：https://blog.csdn.net/lming_08/article/details/39210409

RandomForest algorithm ：

有兩個class，分別處理分類和回歸，RandomForestClassifier and RandomForestRegressor classes。樣本提取時允許replacement（a bootstrap sample），在隨機選取的部分（而不是全部的）features上進行划分，與原論文的vote方法不同，scikit-learn通過平均每個分類器的預測概率（averaging their probabilistic prediction）來生成最終結果。

Extremely Randomized Trees ：

有兩個class，分別處理分類和回歸， ExtraTreesClassifier and ExtraTreesRegressor classes。默認使用所有樣本，但划分時features隨機選取部分。

給個比較例子：

>>> from sklearn.cross_validation import cross_val_score >>> from sklearn.datasets import make_blobs >>> from sklearn.ensemble import RandomForestClassifier >>> from sklearn.ensemble import ExtraTreesClassifier >>> from sklearn.tree import DecisionTreeClassifier >>> X, y = make_blobs(n_samples=10000, n_features=10, centers=100, ... random_state=0) >>> clf = DecisionTreeClassifier(max_depth=None, min_samples_split=1, ... random_state=0) >>> scores = cross_val_score(clf, X, y) >>> scores.mean() 0.97... >>> clf = RandomForestClassifier(n_estimators=10, max_depth=None, ... min_samples_split=1, random_state=0) >>> scores = cross_val_score(clf, X, y) >>> scores.mean() 0.999... >>> clf = ExtraTreesClassifier(n_estimators=10, max_depth=None, ... min_samples_split=1, random_state=0) >>> scores = cross_val_score(clf, X, y) >>> scores.mean() > 0.999 True

幾點說明：

1）參數：最主要的調節參數是 n_estimators and max_features ，經驗最好數據是，回歸問題設置 max_features=n_features ，分類問題設置max_features=sqrt(n_features)(n_features是數據集的features個數).；設置max_depth=None 並且結合min_samples_split=1 (i.e., when fully developing the trees)經常導致好的結果；但切記，最好的參數還是通過CV調出來的。

2）默認機制：random forests, bootstrap samples are used by default (bootstrap=True) while the default strategy for extra-trees is to use the whole dataset (bootstrap=False).

3）並行：設置n_jobs=k 保證使用機器的k個cores；設置n_jobs=-1 使用所有可用的cores。

4）特征重要性評估：一個決策樹，節點在越高的分支，相應的特征對最終預測結果的contribute越大。這里的大，是指影響輸入數據集的比例比較大（the fraction of the input samples is large）。所以，對於某一個randomized tree，可以通過 The expected fraction of the samples they contribute to can thus be used as an estimate of the relative importance of the features.，然后對於 n_estimators 個randomized tree，通過averaging those expected activity rates over several randomized trees，達到區分特征重要性、特征選擇的目的。但上面的敘述沒什么X用，屬性 feature_importances_ 已經保留了該重要性記錄。。。。

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 決策樹之特征選擇決策樹之特征選擇算法（ID3、C4.5、CART）通俗講解決策樹：如何利用有效特征進行決策分類？ python選擇最佳決策樹參數決策樹如何對連續性特征進行分段？再探決策樹算法之利用sklearn進行決策樹實戰如何進行特征選擇 python進行機器學習（二）之特征選擇特征選擇實踐---python python實現決策樹