决策树分类器背后的数学 (Maths behind Decision Tree Classifier)

Before we see the python implementation of the decision tree. Let’s first understand the math behind the decision tree classification. We will see how all the above-mentioned terms are used for splitting.

在我们看到决策树的python实现之前。 首先让我们了解决策树分类背后的数学原理。 我们将看到如何使用所有上述术语进行拆分。

We will use a simple dataset which contains information about students from different classes and gender and see whether they stay in the school’s hostel or not.

我们将使用一个简单的数据集，其中包含有关来自不同班级和性别的学生的信息，并查看他们是否留在学校的宿舍中。

This is how our data set looks like :

这就是我们的数据集的样子：

Let’s try and understand how the root node is selected by calcualting gini impurity. We will use the above mentioned data.

让我们尝试了解如何通过计算基尼杂质来选择根节点。 我们将使用上述数据。

We have two features which we can use for nodes: “Class” and “Gender”. We will calculate gini impurity for each of the features and then select that feature which has least gini impurity.

我们有两个可用于节点的功能：“类”和“性别”。 我们将为每个特征计算基尼杂质，然后选择基尼杂质最少的特征。

Let’s review the formula for calculating ginni impurity:

让我们回顾一下计算ginni杂质的公式：

Let’s start with class, we will try to gini impurity for all different values in “class”.

让我们从类开始，我们将尝试为“类”中的所有不同值添加杂质。

This is how our Decision tree node is selected by calculating gini impurity for each node individually. If the number of feautures increases, then we just need to repeat the same steps after the selection of the root node.

这就是通过分别计算每个节点的基尼杂质来选择我们的决策树节点的方式。 如果功能数量增加，那么我们只需要在选择根节点之后重复相同的步骤即可。

We will try and find the root nodes for the same dataset by calculating entropy and information gain.

我们将通过计算熵和信息增益来尝试找到同一数据集的根节点。

DataSet:

数据集：

We have two features and we will try to choose the root node by calculating the information gain by splitting each feature.

我们有两个功能，我们将尝试通过拆分每个功能来计算信息增益来选择根节点。

Let’ review the formula for entropy and information gain:

让我们回顾一下熵和信息增益的公式：

Let’s start with feature “class” :

让我们从功能“类”开始：

Let’ see the information gain from feature “gender” :

让我们看看从“性别”功能获得的信息：

决策树的不同算法 (Different Algorithms for Decision Tree)

• ID3 (Iterative Dichotomiser) : It is one of the algorithms used to construct decision tree for classification. It uses Information gain as the criteria for finding the root nodes and splitting them. It only accepts categorical attributes.ID3(迭代二分器)：这是用于构建决策树以进行分类的算法之一。 它使用信息增益作为查找根节点并将其拆分的标准。 它仅接受分类属性。
• C4.5 : It is an extension of ID3 algorithm, and better than ID3 as it deals both continuous and discreet values.It is also used for classfication purposes.C4.5：它是ID3算法的扩展，比ID3更好，因为它既处理连续值又处理离散值，也用于分类目的。
• Classfication and Regression Algorithm(CART) : It is the most popular algorithm used for constructing decison trees. It uses ginni impurity as the default calculation for selecting root nodes, however one can use “entropy” for criteria as well. This algorithm works on both regression as well as classfication problems. We will use this algorithm in our pyhton implementation.分类和回归算法(CART)：这是用于构建决策树的最流行算法。 它使用ginni杂质作为选择根节点的默认计算，但是也可以使用“熵”作为标准。 该算法适用于回归和分类问题。 我们将在pyhton实现中使用此算法。

Entropy and Ginni impurity can be used reversibly. It doesn’t affects the result much. Although, ginni is easier to compute than entropy, since entropy has a log term calculation. That’s why CART algorithm uses ginni as the default algorithm.

熵和Ginni杂质可以可逆地使用。 它对结果的影响不大。 尽管ginni比熵更容易计算，因为熵具有对数项计算。 这就是CART算法使用ginni作为默认算法的原因。

If we plot ginni vs entropy graph, we can see there is not much difference between them:

如果我们绘制ginni vs熵图，我们可以看到它们之间没有太大的区别：

Advantages of Decision Tree:

决策树的优势：

• It can be used for both Regression and Classification problems.它可以用于回归和分类问题。
• Decision Trees are very easy to grasp as the rules of splitting is clearly mentioned.决策树很容易掌握，因为明确提到了拆分规则。
• Complex decision tree models are very simple when visualized. It can be understood just by visualizing.可视化时，复杂的决策树模型非常简单。 仅仅通过可视化就可以理解。
• Scaling and normalization are not needed.不需要缩放和规范化。

Disadvantages of Decision Tree:

决策树的缺点：

• A small change in data can cause instability in the model because of the greedy approach.由于贪婪的方法，数据的微小变化会导致模型不稳定。
• Probability of overfitting is very high for Decision Trees.对于决策树，过度拟合的可能性非常高。
• It takes more time to train a decision tree model than other classification algorithms.与其他分类算法相比，训练决策树模型需要更多时间。