ID3算法

一、ID3算法簡單介紹

最早起源於《羅斯昆ID3在悉尼大學。他第一次提出的ID3 1975年在一本書、機器學習、研究所碩士論文。ID3是建立了概念學習系統(CLS)算法。ID3算法是一種基於決策樹的算法。決策樹由決策結點、分支和葉子組成。決策樹中最上面的結點為根節點，每個分支是一個新的決策結點，或者是樹的葉子。每個決策結點代表一個問題或決策，通常對應於待分類對象的屬性。每一個葉子節點代表一種可能的分類結果。沿決策樹從上到下遍歷的過程中，在每個結點都會遇到一個測試，對每個結點上問題的不同的測試輸出導致不同的分支，最后會到達一個葉子節點，這個過程就是利用決策樹進行分類的過程，利用若干個變量來判斷所屬的類別。

二、ID3算法基礎--信息論

 ID3算法是一信息論為基礎，以信息熵和信息增益度為衡量指標，從而實現對數據的分類操作。下面給出一些信息論中的基本概念：

定義1：若存在n個相同概率的消息，則每個消息的概率p是1/n，一個消息傳遞的消息量為-Log2(1/n)

定義2：若有n個消息，其給定概率分布為P=(p1,p2,....pn)，則由該分布傳遞的消息量稱為P的熵，記為I(P)=-p1*Log2(p1)-p2*Log2(p2)-...-pn*Log2(pn).

定義3：若一個記錄集合T根據類別屬性的值被分成互相獨立的類C1,C2.....Ck，那么識別T的一個所屬那個類型需要的信息量為Info(T)=I(P)，其中P為C1,C2....Ck的概率分布，即P=(|C1|/|T|,|C2|/|T|,....|Ck|/|T|)。

定義4：若我們先根據非類別屬性X的值將T的值分成集合T1,T2,T3....Tn，則確定T中一個元素類的信息量可通過確定Ti的加權平均值來得到，即Info(Ti)的加權平均值為：Info(X,T)=（i=1 to n求和）((|Ti|/|T|)Info(Ti))

定義5：信息增益度是兩個信息量之間的差值，其中一個信息量是確定T的一個元素的信息量，另一個信息量是在得到一個確定屬性X的值后需要確定T一個元素的信息量，公式為:Gain(X,T) = Info(T) = Info(X,T).

ID3算法計算每個屬性的信息增益，並選擇具有最高增益的屬性作為給定集合的測試屬性。對被選擇的屬性創建一個節點，並記錄該節點的屬性標記，對該屬性的每一個值創建一個分支，並對分支進行迭代循環計算信息增益操作。

三、ID3算法步驟示例

 下面給定一個ID3算法的示例：

RID	age	income	student	credit_rating	buy_compter
1	youth	high	no	fair	no
2	youth	high	no	excellent	no
3	middle_aged	high	no	fair	yes
4	senior	medium	no	fair	yes
5	senior	low	yes	fair	yes
6	senior	low	yes	excellent	no
7	middle_aged	low	yes	excellent	yes
8	youth	medium	no	fair	no
9	youth	low	yes	fair	yes
10	senior	medium	yes	fair	yes
11	youth	medium	yes	excellent	yes
12	middle_aged	medium	no	excellent	yes
13	middle_aged	high	yes	fair	yes
14	senior	medium	no	excellent	no

 

總數據量是14條，參考屬性是age(youth[5], middle_aged[4], senior[5])，income(high[4], medium[6], low[4]), student(no[7], yes[7]), credit_rating(fair[8], excellent[6])。目標屬性是bug_computer(no[5], yes[9])，希望的結果是能夠得到一個根據age, income, student, credit_rating來推測出來buy_computer的值。假設初始數據集D，參考屬性列表A，下面給定計算步驟：

第一步：在數據集D中就目標屬性的信息熵： Info(_{buy_computer}) = -(5/14)*log2(5/14)-(9/14)*log2(9/14)=0.94

第二步：在數據集D中就參考屬性列表A中的每一個屬性計算，在該屬性值確定的條件下，確定一個bug_computer的信息熵，也就是條件熵。

　　age屬性：youth(no[3],yes[2])，middle_aged(no[0],yes[4])，senior(no[2],yes[3])，先分別計算youth、middle_aged、senior的信息熵。

　　　　Info_age(_{bug_computer|youth}) = -(3/5)*log2(3/5) - (2/5)*log2(2/5) = 0.971

　　　　Info_age(_{bug_computer|middle_aged}) = -(4/4)*log2(4/4) - (0/5)*log2(0/5) = 0

　　　　Info_age(_{bug_computer|senior}) = -(2/5)*log2(2/5) - (3/5)*log2(3/5) = 0.971

　　則Info_age(buy_computer) = 5/14*0.971 + 4/14 * 0+ 5/14 * 0.971 = 0.694

　　同理：Info_income(buy_computer) = 0.911；Info_student(buy_computer) = 0.789；Info_{credit_rating}(buy_computer) = 0.892.

第三步，計算信息增益度，該值如果越大，表示目標屬性在該參考屬性上失去的信息熵越多，那么該屬性就越應該在決策樹的上層。計算結果為：

　　Gain(age,bug_computer) = Info(buy_computer) - Info_age(buy_computer) = 0.94 - 0.694 = 0.246

　　Gain(income,bug_computer) = Info(buy_computer) - Info_icome(buy_computer) = 0.94 - 0.911 = 0.029

　　Gain(student,bug_computer) = Info(buy_computer) - Info_student(buy_computer) = 0.94 - 0.789 = 0.151

　　Gain(credit_rating,bug_computer) = Info(buy_computer) - Info_{credit_rating}(buy_computer) = 0.94 - 0.892 = 0.048

第四步，選擇信息增益度最大的屬性作為當前節點，此時是age，根據age的不同取值將初始數據集D分隔成以下情況。

　　1. age為youth的時候，子數據集是D1：

RID	income	student	credit_rating	buy_computer
1	high	no	fair	no
2	high	no	excellent	no
8	medium	no	fair	no
9	low	yes	fair	yes
11	high	yes	excellent	yes

　　2. age為middle_aged的時候，子數據集是D2：

RID	income	student	credit_rating	buy_computer
3	high	no	fair	yes
7	low	yes	excellent	yes
12	medium	no	excellent	yes
13	high	yes	fair	yes

　　3. age為senior的時候，子數據集是D3:

RID	income	student	credit_rating	buy_computer
4	medium	no	fair	yes
5	low	yes	fair	yes
6	low	yes	excellent	no
10	medium	yes	fair	yes
14	medium	no	excellent	no

 

第五步，將已經選擇的參考屬性(age)從參考屬性列表A中剔除，針對第四步中產生的子數據集D_i使用處理后的參考屬性列表A，再從第一步迭代處理。迭代結束條件為：

1. 當某種分類中，目標屬性只有一個值，如這里當age為middle_aged的時候。
2. 當分到某類的時候，目標屬性所有值中，某個值的比例達到了閾值(人為控制)，比如可以設為只要buy_computer中某個值達到90%以上，就可以結束迭代。

經過多次迭戈處理，最終會得到一個樹結構如下圖所示：

獲得規則是：

IF AGE=middle_aged, THEN buy_computer = yes

IF AGE = youth AND STUDENT = yes, THEN buy_computer = yes

IF AGE = youth AND STUDENT = no, THEN buy_computer = no

IF AGE = senior AND CREDIT_RATING = excellent, THEN buy_computer = no

IF AGE = senior AND CREDIT_RATING = fair, THEN buy_computer = yes

SO, If the instance are ("15", "youth", "medium", "yes", "fair"), the predicted value of buy_computer is "yes".

 

四、ID3算法程序實現

下面分別給出python和java兩種語言的ID3算法的實現：

Python程序：

# -*- coding: utf-8 -*-


class Node:
    '''Represents a decision tree node.
    
    '''
    def __init__(self, parent = None, dataset = None):
        self.dataset = dataset # 落在該結點的訓練實例集
        self.result = None # 結果類標簽
        self.attr = None # 該結點的分裂屬性ID
        self.childs = {} # 該結點的子樹列表，key-value pair: (屬性attr的值, 對應的子樹)
        self.parent = parent # 該結點的父親結點
        


def entropy(props):
    if (not isinstance(props, (tuple, list))):
        return None
    
    from math import log
    log2 = lambda x:log(x)/log(2) # an anonymous function
    e = 0.0
    for p in props:
        if p != 0:
            e = e - p * log2(p)
    return e


def info_gain(D, A, T = -1, return_ratio = False):
    '''特征A對訓練數據集D的信息增益 g(D,A)
    
    g(D,A)=entropy(D) - entropy(D|A)
            假設數據集D的每個元組的最后一個特征為類標簽
    T為目標屬性的ID，-1表示元組的最后一個元素為目標'''
    if (not isinstance(D, (set, list))):
        return None
    if (not type(A) is int):
        return None
    C = {} # 類別計數字典
    DA = {} # 特征A的取值計數字典
    CDA = {} # 類別和特征A的不同組合的取值計數字典
    for t in D:
        C[t[T]] = C.get(t[T], 0) + 1
        DA[t[A]] = DA.get(t[A], 0) + 1
        CDA[(t[T], t[A])] = CDA.get((t[T], t[A]), 0) + 1

    PC = map(lambda x : 1.0 * x / len(D), C.values()) # 類別的概率列表,即目標屬性的概率，信息熵
    entropy_D = entropy(tuple(PC)) # map返回的對象類型為map，需要強制類型轉換為元組


    PCDA = {} # 特征A的每個取值給定的條件下各個類別的概率（條件概率）
    for key, value in CDA.items():
        a = key[1] # 特征A
        pca = value / DA[a]
        PCDA.setdefault(a, []).append(pca)
    
    condition_entropy = 0.0
    for a, v in DA.items():
        p = v / len(D)
        e = entropy(PCDA[a])
        condition_entropy += e * p
    
    if (return_ratio):
        return (entropy_D - condition_entropy) / entropy_D
    else:
        return entropy_D - condition_entropy
    
def get_result(D, T = -1):
    '''獲取數據集D中實例數最大的目標特征T的值'''
    if (not isinstance(D, (set, list))):
        return None
    if (not type(T) is int):
        return None
    count = {}
    for t in D:
        count[t[T]] = count.get(t[T], 0) + 1
    max_count = 0
    for key, value in count.items():
        if (value > max_count):
            max_count = value
            result = key
    return result 


def devide_set(D, A):
    '''根據特征A的值把數據集D分裂為多個子集'''
    if (not isinstance(D, (set, list))):
        return None
    if (not type(A) is int):
        return None
    subset = {}
    for t in D:
        subset.setdefault(t[A], []).append(t)
    return subset


def build_tree(D, A, threshold = 0.0001, T = -1, Tree = None, algo = "ID3"):
    '''根據數據集D和特征集A構建決策樹.
    
    T為目標屬性在元組中的索引 . 目前支持ID3和C4.5兩種算法'''
    if (Tree != None and not isinstance(Tree, Node)):
        return None
    if (not isinstance(D, (set, list))):
        return None
    if (not type(A) is set):
        return None
    
    if (None == Tree):
        Tree = Node(None, D)
    subset = devide_set(D, T)
    if (len(subset) <= 1):
        for key in subset.keys():
            Tree.result = key
        del(subset)
        return Tree
    if (len(A) <= 0):
        Tree.result = get_result(D)
        return Tree
    use_gain_ratio = False if algo == "ID3" else True

    max_gain = 0
    for a in A:
        gain = info_gain(D, a, return_ratio = use_gain_ratio)
        if (gain > max_gain):
            max_gain = gain
            attr_id = a # 獲取信息增益最大的特征
    if (max_gain < threshold):
        Tree.result = get_result(D)
        return Tree
    Tree.attr = attr_id
    subD = devide_set(D, attr_id)
    del(D[:]) # 刪除中間數據,釋放內存
    Tree.dataset = None
    A.discard(attr_id) # 從特征集中排查已經使用過的特征
    for key in subD.keys():
        tree = Node(Tree, subD.get(key))
        Tree.childs[key] = tree
        build_tree(subD.get(key), A, threshold, T, tree)
    return Tree


def print_brance(brance, target):
    odd = 0
    for e in brance:
        print e,('=' if odd == 0 else '∧'),
        odd = 1 - odd
    print "target =", target


def print_tree(Tree, stack = []): 
    if (None == Tree):
        return
    if (None != Tree.result):
        print_brance(stack, Tree.result)
        return
    stack.append(Tree.attr)
    for key, value in Tree.childs.items():
        stack.append(key)
        print_tree(value, stack)
        stack.pop()
    stack.pop()
    
def classify(Tree, instance):
    if (None == Tree):
        return None
    if (None != Tree.result):
        return Tree.result
    if instance[Tree.attr] in Tree.childs:
        return classify(Tree.childs[instance[Tree.attr]], instance)
    else:
        return None

dataset = [
   ("青年", "否", "否", "一般", "否")
   ,("青年", "否", "否", "好", "否")
   ,("青年", "是", "否", "好", "是")
   ,("青年", "是", "是", "一般", "是")
   ,("青年", "否", "否", "一般", "否")
   ,("中年", "否", "否", "一般", "否")
   ,("中年", "否", "否", "好", "否")
   ,("老年", "是", "否", "非常好", "是")
   ,("老年", "否", "是", "一般", "是")
   ,("老年", "否", "是", "一般", "是")
   ,("老年", "否", "是", "一般", "是")
   ,("老年", "否", "是", "好", "是")
   ,("老年", "是", "否", "一般", "是")
   ,("老年", "是", "否", "一般", "否")
   ,("老年", "否", "否", "一般", "否")
]

s = set(range(0, len(dataset[0]) - 1))
s = set([0,1,3,4])
T = build_tree(dataset, s)
print_tree(T)
print(classify(T, ("老年", "是", "否", "一般", "否")))
print(classify(T, ("老年", "是", "否", "一般", "是")))
print(classify(T, ("老年", "是", "是", "好", "否")))
print(classify(T, ("青年", "是", "否", "好", "是")))
print(classify(T, ("中年", "是", "否", "好", "否")))

 

該python程序的訓練集不是上面給定的這個列子，輸出結果為：

0 = 青年 ∧ 1 = 否 ∧ target = 否
0 = 青年 ∧ 1 = 是 ∧ target = 是
0 = 中年 ∧ target = 否
0 = 老年 ∧ 3 = 好 ∧ target = 是
0 = 老年 ∧ 3 = 非常好 ∧ target = 是
0 = 老年 ∧ 3 = 一般 ∧ 4 = 否 ∧ target = 否
0 = 老年 ∧ 3 = 一般 ∧ 4 = 是 ∧ target = 是
否
是
是
是
否
[Finished in 0.3s]

  

Java程序，該程序的數據集是上面給定的例子，代碼及結果如下：

  2 
import java.util.ArrayList;
import java.util.Collection;
import java.util.Deque;
import java.util.HashMap;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;

public class ID3Tree {
    private List<String[]> datas;
    private List<Integer> attributes;
    private double threshold = 0.0001;
    private int targetIndex = 1;
    private Node tree;
    private Map<Integer, String> attributeMap;

    protected ID3Tree() {
        super();
    }

    public ID3Tree(List<String[]> datas, List<Integer> attributes, Map<Integer, String> attributeMap, int targetIndex) {
        this(datas, attributes, attributeMap, 0.0001, targetIndex, null);
    }

    public ID3Tree(List<String[]> datas, List<Integer> attributes, Map<Integer, String> attributeMap, double threshold, int targetIndex, Node tree) {
        super();
        this.datas = datas;
        this.attributes = attributes;
        this.attributeMap = attributeMap;
        this.threshold = threshold;
        this.targetIndex = targetIndex;
        this.tree = tree;
    }

    /**
     * 節點對象
     * 
     * @author Gerry.Liu
     *
     */
    class Node {
        private List<String[]> dataset; // 落在該節點上的訓練實訓集
        private String result; // 結果類標簽
        private int attr; // 該節點的分裂屬性ID,下標
        private Node parent; // 該節點的父節點
        private Map<String, List<Node>> childs; // 該節點的子節點集合

        public Node(List<String[]> datas, Node parent) {
            this.dataset = datas;
            this.parent = parent;
            this.childs = new HashMap<>();
        }
    }

    class KeyValue {
        private String first;
        private String second;

        public KeyValue(String first, String second) {
            super();
            this.first = first;
            this.second = second;
        }

        @Override
        public int hashCode() {
            final int prime = 31;
            int result = 1;
            result = prime * result + getOuterType().hashCode();
            result = prime * result + ((first == null) ? 0 : first.hashCode());
            result = prime * result + ((second == null) ? 0 : second.hashCode());
            return result;
        }

        @Override
        public boolean equals(Object obj) {
            if (this == obj)
                return true;
            if (obj == null)
                return false;
            if (getClass() != obj.getClass())
                return false;
            KeyValue other = (KeyValue) obj;
            if (!getOuterType().equals(other.getOuterType()))
                return false;
            if (first == null) {
                if (other.first != null)
                    return false;
            } else if (!first.equals(other.first))
                return false;
            if (second == null) {
                if (other.second != null)
                    return false;
            } else if (!second.equals(other.second))
                return false;
            return true;
        }

        private ID3Tree getOuterType() {
            return ID3Tree.this;
        }
    }

    /**
     * 根據概率計算信息熵，計算規則是：<br/>
     * entropy(p1,p2....pn) = -p1*log2(p1) -p2*log2(p2)-.....-pn*log2(pn)
     *
     * @param props
     * @return
     */
    private double entropy(List<Double> props) {
        if (props == null || props.isEmpty()) {
            return 0;
        } else {
            double result = 0;
            for (double p : props) {
                if (p > 0) {
                    result = result - p * Math.log(p) / Math.log(2);
                }
            }
            return result;
        }
    }

    /**
     * 計算概率
     * 
     * @param totalRecords
     * @param counts
     * @return
     */
    private List<Double> calcProbability(int totalRecords, Collection<Integer> counts) {
        if (totalRecords == 0 || counts == null || counts.isEmpty()) {
            return null;
        }

        List<Double> result = new ArrayList<>();
        for (int count : counts) {
            result.add(1.0 * count / totalRecords);
        }
        return result;
    }

    /**
     * 獲取信息增益Gain(datas,attribute)<br/>
     * 特征屬性attribute(A)對訓練數據集datas(D)的信息增益<br/>
     * g(D,A) = entropy(D) - entropy(D|A)<br/>
     * 
     * @param datas
     *            訓練數據集
     * @param attributeIndex
     *            特征屬性下標
     * @param targetAttributeIndex
     *            目標屬性下標
     * @return
     */
    private double infoGain(List<String[]> datas, int attributeIndex, int targetAttributeIndex) {
        if (datas == null || datas.isEmpty()) {
            return 0;
        }

        Map<String, Integer> targetAttributeCountMap = new HashMap<String, Integer>(); // 類別(目標屬性)計數
        Map<String, Integer> featureAttributesCountMap = new HashMap<>(); // 特征屬性上的取值計數
        Map<KeyValue, Integer> tfAttributeCountMap = new HashMap<>(); // 類別和特征屬性的不同組合的計數

        for (String[] arrs : datas) {
            String tv = arrs[targetAttributeIndex];
            String fv = arrs[attributeIndex];
            if (targetAttributeCountMap.containsKey(tv)) {
                targetAttributeCountMap.put(tv, targetAttributeCountMap.get(tv) + 1);
            } else {
                targetAttributeCountMap.put(tv, 1);
            }
            if (featureAttributesCountMap.containsKey(fv)) {
                featureAttributesCountMap.put(fv, featureAttributesCountMap.get(fv) + 1);
            } else {
                featureAttributesCountMap.put(fv, 1);
            }
            KeyValue key = new KeyValue(fv, tv);
            if (tfAttributeCountMap.containsKey(key)) {
                tfAttributeCountMap.put(key, tfAttributeCountMap.get(key) + 1);
            } else {
                tfAttributeCountMap.put(key, 1);
            }
        }

        int totalDataSize = datas.size();
        // 計算概率
        List<Double> probabilitys = calcProbability(totalDataSize, targetAttributeCountMap.values());
        // 計算目標屬性的信息熵
        double entropyDatas = this.entropy(probabilitys);

        // 計算條件概率
        // 第一步，計算目標屬性的各種取值，在特征屬性確定的條件下的情況
        Map<String, List<Double>> pcda = new HashMap<>();
        for (Map.Entry<KeyValue, Integer> entry : tfAttributeCountMap.entrySet()) {
            String key = entry.getKey().first;
            double pca = 1.0 * entry.getValue() / featureAttributesCountMap.get(key);
            if (pcda.containsKey(key)) {
                pcda.get(key).add(pca);
            } else {
                List<Double> list = new ArrayList<Double>();
                list.add(pca);
                pcda.put(key, list);
            }
        }
        // 第二步，針對每個特征屬性的值取信息熵，並獲取平均熵
        double conditionEntropy = 0.0;
        for (Map.Entry<String, Integer> entry : featureAttributesCountMap.entrySet()) {
            double p = 1.0 * entry.getValue() / totalDataSize;
            double e = this.entropy(pcda.get(entry.getKey()));
            conditionEntropy += e * p;
        }
        return entropyDatas - conditionEntropy;
    }

    /**
     * 獲取數據集中目標屬性中，實例值個數最大的目標特征值
     * 
     * @param datas
     * @param targetAttributeIndex
     * @return
     */
    private String getResult(List<String[]> datas, int targetAttributeIndex) {
        if (datas == null || datas.isEmpty()) {
            return null;
        } else {
            String result = "";
            Map<String, Integer> countMap = new HashMap<>();
            for (String[] arr : datas) {
                String key = arr[targetAttributeIndex];
                if (countMap.containsKey(key)) {
                    countMap.put(key, countMap.get(key) + 1);
                } else {
                    countMap.put(key, 1);
                }
            }

            int maxCount = -1;
            for (Map.Entry<String, Integer> entry : countMap.entrySet()) {
                if (entry.getValue() > maxCount) {
                    maxCount = entry.getValue();
                    result = entry.getKey();
                }
            }
            return result;
        }
    }

    /**
     * 按照特征屬性的值將數據集D分裂成為多個子集
     * 
     * @param datas
     *            數據集
     * @param attributeIndex
     *            特征屬性下標
     * @return
     */
    private Map<String, List<String[]>> devideDatas(List<String[]> datas, int attributeIndex) {
        Map<String, List<String[]>> subdatas = new HashMap<>();
        if (datas != null && !datas.isEmpty()) {
            for (String[] arr : datas) {
                String key = arr[attributeIndex];
                if (subdatas.containsKey(key)) {
                    subdatas.get(key).add(arr);
                } else {
                    List<String[]> list = new ArrayList<>();
                    list.add(arr);
                    subdatas.put(key, list);
                }
            }
        }
        return subdatas;
    }

    /**
     * 打印決策樹
     * 
     * @param tree
     * @param stock
     */
    private void printTree(Node tree, Deque<Object> stock) {
        if (tree == null) {
            return;
        }

        if (tree.result != null) {
            this.printBrance(stock, tree.result);
        } else {
            stock.push(this.attributeMap.get(tree.attr));
            for (Map.Entry<String, List<Node>> entry : tree.childs.entrySet()) {
                stock.push(entry.getKey());
                for (Node node : entry.getValue()) {
                    this.printTree(node, stock);
                }
                stock.pop();
            }
            stock.pop();
        }
    }

    /**
     * 輸出Node表示的決策樹的規則
     * 
     * @param tree
     */
    private void printBrance(Deque<Object> stock, String target) {
        StringBuffer sb = new StringBuffer();
        int odd = 0;
        for (Object e : stock) {
            sb.insert(0, odd == 0 ? "^" : "=").insert(0, e);
            // sb.append(e).append(odd == 0 ? "=" : "^");
            odd = 1 - odd;
        }
        sb.append("target=").append(target);
        System.out.println(sb.toString());
    }

    /**
     * 創建一個決策樹
     * 
     * @param datas
     * @param attributes
     * @param threshold
     * @param targetIndex
     * @param tree
     * @return
     */
    private Node buildTree(List<String[]> datas, List<Integer> attributes, double threshold, int targetIndex, Node tree) {
        if (tree == null) {
            tree = new Node(datas, null);
        }
        // 分隔數據集,返回的數據集為empty或者是有數據,不會為null
        Map<String, List<String[]>> subDatas = this.devideDatas(datas, targetIndex);
        if (subDatas.size() <= 1) {
            // 這里只會有一個key
            for (String key : subDatas.keySet()) {
                tree.result = key;
            }
        } else if (attributes == null || attributes.size() < 1) {
            // 沒有特征集，那么直接獲取最多的值
            tree.result = this.getResult(datas, targetIndex);
        } else {
            double maxGain = 0;
            int attr = 0;

            for (int attribute : attributes) {
                double gain = this.infoGain(datas, attribute, targetIndex);
                if (gain > maxGain) {
                    maxGain = gain;
                    attr = attribute;// 最大的信息增益下標
                }
            }

            if (maxGain < threshold) {
                // 達到收益條件
                tree.result = this.getResult(datas, targetIndex);
            } else {
                // 沒有達到結束條件，繼續進行
                tree.attr = attr;
                subDatas = this.devideDatas(datas, attr);
                tree.dataset = null;
                attributes.remove(Integer.valueOf(attr));
                for (String key : subDatas.keySet()) {
                    Node childTree = new Node(subDatas.get(key), tree);
                    if (tree.childs.containsKey(key)) {
                        tree.childs.get(key).add(childTree);
                    } else {
                        List<Node> childs = new ArrayList<>();
                        childs.add(childTree);
                        tree.childs.put(key, childs);
                    }
                    this.buildTree(subDatas.get(key), attributes, threshold, targetIndex, childTree);
                }
            }
        }
        return tree;
    }

    /**
     * 根據決策規則獲取推薦值
     * 
     * @param instance
     * @return
     */
    private String classify(Node tree, String[] instance) {
        if (tree == null) {
            return null;
        }
        if (tree.result != null) {
            return tree.result;
        }
        if (tree.childs.containsKey(instance[tree.attr])) {
            List<Node> nodes = tree.childs.get(instance[tree.attr]);
            for (Node node : nodes) {
                return this.classify(node, instance);
            }
        }
        return null;
    }

    /**
     * 生產決策樹
     */
    public void buildTree() {
        this.tree = new Node(this.datas, null);
        this.buildTree(datas, attributes, threshold, targetIndex, tree);
    }

    /**
     * 打印生產的規則
     */
    public void printTree() {
        this.printTree(this.tree, new LinkedList<>());
    }

    /**
     * 獲取推薦結果
     * 
     * @param instance
     * @return
     */
    public String classify(String[] instance) {
        return this.classify(this.tree, instance);
    }

    public static void main(String[] args) {
        List<String[]> dataset = new ArrayList<>();
        dataset.add(new String[] { "1", "youth", "high", "no", "fair", "no" });
        dataset.add(new String[] { "2", "youth", "high", "no", "excellent", "no" });
        dataset.add(new String[] { "3", "middle_aged", "high", "no", "fair", "yes" });
        dataset.add(new String[] { "4", "senior", "medium", "no", "fair", "yes" });
        dataset.add(new String[] { "5", "senior", "low", "yes", "fair", "yes" });
        dataset.add(new String[] { "6", "senior", "low", "yes", "excellent", "no" });
        dataset.add(new String[] { "7", "middle_aged", "low", "yes", "excellent", "yes" });
        dataset.add(new String[] { "8", "youth", "medium", "no", "fair", "no" });
        dataset.add(new String[] { "9", "youth", "low", "yes", "fair", "yes" });
        dataset.add(new String[] { "10", "senior", "medium", "yes", "fair", "yes" });
        dataset.add(new String[] { "11", "youth", "medium", "yes", "excellent", "yes" });
        dataset.add(new String[] { "12", "middle_aged", "medium", "no", "excellent", "yes" });
        dataset.add(new String[] { "13", "middle_aged", "high", "yes", "fair", "yes" });
        dataset.add(new String[] { "14", "senior", "medium", "no", "excellent", "no" });

        List<Integer> attributes = new ArrayList<>();
        attributes.add(4);
        attributes.add(1);
        attributes.add(2);
        attributes.add(3);

        Map<Integer, String> attributeMap = new HashMap<>();
        attributeMap.put(1, "Age");
        attributeMap.put(2, "Income");
        attributeMap.put(3, "Student");
        attributeMap.put(4, "Credit_rating");

        int targetIndex = 5;

        String[] instance = new String[] { "15", "youth", "medium", "yes", "fair" };

        ID3Tree tree = new ID3Tree(dataset, attributes,attributeMap, targetIndex);
        System.out.println("start build the tree");
        tree.buildTree();
        System.out.println("completed build the tree, start print the tree");
        tree.printTree();
        System.out.println("start classify.....");
        String result = tree.classify(instance);
        System.out.println(result);
    }
}

 運行java程序的結果是：

Start build the tree.....
Completed build the tree, start print the tree.....
Age=youth^Student=yes^target=yes
Age=youth^Student=no^target=no
Age=middle_aged^target=yes
Age=senior^Credit_rating=excellent^target=no
Age=senior^Credit_rating=fair^target=yes
start classify.....
yes

 五、ID3算法不足 

ID3算法運行速度較慢，只能加載內存中的數據，處理的數據集相對於其他算法較小。