github地址：DataScicence欢迎star
集成学习4-前向分步算法与GBDT-原理与案例
集成学习3-Boosting的原理和案例
集成学习2-bagging的原理与案例分析
集成学习1-投票法的原理和案例分析

Xgboost原理与调参

XGBoost原理
- 目标函数
- 定义树模型
- 重构目标函数
- 求解w和L：
- 寻找最佳分支
Xgboost案例
- 加载数据集
- 训练模型
- 绘制特征重要性
- 调参

XGBoost原理

Xgboost的大致原理与GBDT相似，但是在部分步骤中进行了改进

目标函数

xgboost与GBDT最大的不同就是目标函数

上式中， y ^ i t − 1 \hat y_{i}^{t-1} y^it−1表示前面t-1轮中生成的加权树模型的预测结果， Ω ( f i ) 表示正则项 Ω(f_i)表示正则项 Ω(fi)表示正则项

接下来是重点，利用泰勒展开拟合目标函数：

前面t-1轮的训练误差是已知的，因此将上式改变为：

函数g和h分别是1阶和2阶导数

定义树模型

f t ( x ) = w q ( x ) 每个节点的权重 f_t(x)=w_q(x) 每个节点的权重 ft(x)=wq(x)每个节点的权重

q ( x ) 每个样本属于的节点 q(x)每个样本属于的节点 q(x)每个样本属于的节点

I j = { i ∣ q ( x i ) = j } 每个节点的样本集合 I_{j}=\left\{i \mid q\left(\mathbf{x}_{i}\right)=j\right\}每个节点的样本集合 Ij={i∣q(xi)=j}每个节点的样本集合

如上图所示， q ( x 1 ) = 1 , q ( x 2 ) = 3 , q ( x 3 ) = 1 , q ( x 4 ) = 2 , q ( x 5 ) = 3 q(x_1) = 1,q(x_2) = 3,q(x_3) = 1,q(x_4) = 2,q(x_5) = 3 q(x1)=1,q(x2)=3,q(x3)=1,q(x4)=2,q(x5)=3， I 1 = { 1 , 3 } , I 2 = { 4 } , I 3 = { 2 , 5 } I_1 = \{1,3\},I_2 = \{4\},I_3 = \{2,5\} I1={1,3},I2={4},I3={2,5}， w = ( 15 , 12 , 20 ) w = (15,12,20) w=(15,12,20)

重新定义树的复杂度：
Ω ( f K ) = γ T + 1 2 λ ∑ j = 1 T w j 2 \Omega\left(f_{K}\right) = \gamma T+\frac{1}{2} \lambda \sum_{j=1}^{T} w_{j}^{2} Ω(fK)=γT+21λj=1∑Twj2

重构目标函数

O b j ( t ) = ∑ i = 1 n [ g i f K ( x i ) + 1 2 h i f K 2 ( x i ) ] + γ T + 1 2 λ ∑ j = 1 T w j 2 = ∑ j = 1 T [ ( ∑ i ∈ I j g i ) w j + 1 2 ( ∑ i ∈ I j h i + λ ) w j 2 ] + γ T = [ G j w j + 1 2 ( H j + λ ) w j 2 ] + γ T \begin{aligned} Obj^{(t)} &=\sum_{i=1}^{n}\left[g_{i} f_{K}\left(\mathrm{x}_{i}\right)+\frac{1}{2} h_{i} f_{K}^{2}\left(\mathrm{x}_{i}\right)\right]+\gamma T+\frac{1}{2} \lambda \sum_{j=1}^{T} w_{j}^{2} \\ &=\sum_{j=1}^{T}\left[\left(\sum_{i \in I_{j}} g_{i}\right) w_{j}+\frac{1}{2}\left(\sum_{i \in I_{j}} h_{i}+\lambda\right) w_{j}^{2}\right]+\gamma T\\ &=[G_jw_j+\frac{1}{2}(H_j+λ)w^2_j]+γT \end{aligned} Obj(t)=i=1∑n[gifK(xi)+21hifK2(xi)]+γT+21λj=1∑Twj2=j=1∑T⎣⎡⎝⎛i∈Ij∑gi⎠⎞wj+21⎝⎛i∈Ij∑hi+λ⎠⎞wj2⎦⎤+γT=[Gjwj+21(Hj+λ)wj2]+γT

式中：

求解w和L：

找到令Obj最小的w:( a x 2 + b x + c ax^2+bx+c ax2+bx+c求解公式： x ∗ = − b 2 a x^*=-\frac{b}{2a} x∗=−2ab)

将 w j ∗ 代入 O b j w_j^*代入Obj wj∗代入Obj即可求得目标函数的值

所以分支后，目标函数的降低值为：

寻找最佳分支

使用精确算法或近似算法，选择每一步中使Gain最大的分支方法

Xgboost案例

加载数据集

from sklearn.datasets import load_iris
import xgboost as xgb
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score,classification_report

iris = load_iris()
X,y = iris.data,iris.target

import pandas as pd
X = pd.DataFrame(X,columns=iris.feature_names)
X.head()

	sepal length (cm)	sepal width (cm)	petal length (cm)	petal width (cm)
0	5.1	3.5	1.4	0.2
1	4.9	3.0	1.4	0.2
2	4.7	3.2	1.3	0.2
3	4.6	3.1	1.5	0.2
4	5.0	3.6	1.4	0.2

训练模型

X_train,X_test,y_train,y_test = train_test_split(X,y,test_size = 0.3)

# 算法参数
params = {'booster': 'gbtree','objective': 'multi:softmax','num_class': 3,'gamma': 0.1,'max_depth': 6,'lambda': 2,'subsample': 0.7,'colsample_bytree': 0.75,'min_child_weight': 3,'eta': 0.1,'seed': 1,'nthread': 4,
}

dtrain = xgb.DMatrix(X_train,y_train)model = xgb.XGBClassifier(**params)
model.fit(X_train,y_train)
y_pred = model.predict(X_test)
print(classification_report(y_test,y_pred))

[16:55:13] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'multi:softprob' was changed from 'merror' to 'mlogloss'. Explicitly set eval_metric if you'd like to restore the old behavior.precision    recall  f1-score   support0       1.00      1.00      1.00        131       1.00      0.86      0.93        222       0.77      1.00      0.87        10accuracy                           0.93        45macro avg       0.92      0.95      0.93        45
weighted avg       0.95      0.93      0.94        45

绘制特征重要性

xgb.plot_importance(model)

<matplotlib.axes._subplots.AxesSubplot at 0x2c2d525cc10>

调参

常用参数：

参考：机器学习集成学习之XGBoost

from sklearn.model_selection import GridSearchCV
def Tuning(cv_params, other_params,x_train_array,y_train_):model2 = xgb.XGBClassifier(**other_params)optimized_GBM = GridSearchCV(estimator=model2, param_grid=cv_params,scoring='accuracy',cv=5, n_jobs=-1)optimized_GBM.fit(x_train_array, y_train_)evalute_result = optimized_GBM.cv_results_['mean_test_score']#print('每轮迭代运行结果:{0}'.format(evalute_result))print('参数的最佳取值：{0}'.format(optimized_GBM.best_params_))print('最佳模型得分:{0}'.format(optimized_GBM.best_score_))return optimized_GBM

other_params = {'booster': 'gbtree','objective': 'multi:softmax','num_class': 3
}
cv_params = {'learning_rate':[0.01, 0.02, 0.05, 0.1, 0.15],
}
opt = Tuning(cv_params,other_params,X_train,y_train)

[17:02:24] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'multi:softprob' was changed from 'merror' to 'mlogloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
参数的最佳取值：{'learning_rate': 0.01}
最佳模型得分:0.9619047619047618

other_params = {'booster': 'gbtree','objective': 'multi:softmax','num_class': 3,'learning_rate':0.01,
}
cv_params = {'max_depth': [2,3,4,5],'min_child_weight': [0, 2, 5, 10, 20],
}
opt = Tuning(cv_params,other_params,X_train,y_train)

[17:03:16] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'multi:softprob' was changed from 'merror' to 'mlogloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
参数的最佳取值：{'max_depth': 2, 'min_child_weight': 0}
最佳模型得分:0.9619047619047618C:\Users\lipan\anaconda3\lib\site-packages\xgboost\sklearn.py:888: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].warnings.warn(label_encoder_deprecation_msg, UserWarning)

other_params = {'booster': 'gbtree','objective': 'multi:softmax','num_class': 3,'learning_rate':0.01,'max_depth': 2,'min_child_weight': 0,}
cv_params = {'subsample': [0.6, 0.7, 0.8, 0.85, 0.95],'colsample_bytree': [0.5, 0.6, 0.7, 0.8, 0.9],
}
opt = Tuning(cv_params,other_params,X_train,y_train)

C:\Users\lipan\anaconda3\lib\site-packages\xgboost\sklearn.py:888: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].warnings.warn(label_encoder_deprecation_msg, UserWarning)[17:04:37] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'multi:softprob' was changed from 'merror' to 'mlogloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
参数的最佳取值：{'colsample_bytree': 0.5, 'subsample': 0.95}
最佳模型得分:0.9619047619047618

other_params = {'booster': 'gbtree','objective': 'multi:softmax','num_class': 3,'learning_rate':0.01,'max_depth': 2,'min_child_weight': 0,'subsample': 0.95,'colsample_bytree': 0.5
}
cv_params = {'reg_alpha': [0, 0.25, 0.5, 0.75, 1],
}
opt = Tuning(cv_params,other_params,X_train,y_train)

[17:06:08] WARNING: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/learner.cc:1061: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'multi:softprob' was changed from 'merror' to 'mlogloss'. Explicitly set eval_metric if you'd like to restore the old behavior.
参数的最佳取值：{'reg_alpha': 0}
最佳模型得分:0.9619047619047618C:\Users\lipan\anaconda3\lib\site-packages\xgboost\sklearn.py:888: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].warnings.warn(label_encoder_deprecation_msg, UserWarning)

y_pred = opt.best_estimator_.predict(X_test)
print(classification_report(y_test,y_pred))

              precision    recall  f1-score   support0       1.00      1.00      1.00        131       1.00      0.86      0.93        222       0.77      1.00      0.87        10accuracy                           0.93        45macro avg       0.92      0.95      0.93        45
weighted avg       0.95      0.93      0.94        45

集成学习5-Xgboost原理与调参相关推荐

XGBoost实战与调参优化
本篇主要内容包括XGBoost的入门教学.调参优化,将之前自己遇到的问题细心的整理一遍:XGBoost参数繁多,所以如果要调参,就一定要知道每个参数的作用于意义,因此本人十分建议在实战之前对XGBoo ...
使用贝叶斯优化工具实践XGBoost回归模型调参
0. 关于调参 0.1. 超参数在机器学习的上下文中,超参数(hyper parameters)是在开始学习过程之前设置值的参数,而不是通过训练得到的参数数据.通常情况下,需要对超参数进行优化,给学 ...
XGBoost 重要参数(调参使用)
XGBoost 重要参数(调参使用) 数据比赛Kaggle,天池中最常见的就是XGBoost和LightGBM. 模型是在数据比赛中尤为重要的,但是实际上,在比赛的过程中,大部分朋友在模型上花的时间却 ...
提升深度学习模型性能及网络调参
提升深度学习模型性能及网络调参 https://www.toutiao.com/a6637086018950398472/ 图像处理与机器视觉 2018-12-25 10:42:00 深度学习有很多的 ...
如何使用hyperopt对xgboost进行自动调参
本教程重点在于传授如何使用Hyperopt对xgboost进行自动调参.但是这份代码也是我一直使用的代码模板之一,所以在其他数据集上套用该模板也是十分容易的. 同时因为xgboost,lightgbm ...
python xgboost调参_XGBoost从原理到调参
承接上文挂枝儿:再从GBDT到XGBoost!zhuanlan.zhihu.com 理解了原理,那么接下来就要开始学习怎么调参了,之前做模型的时候用xgboost比较简单粗暴跟着教程一顿乱fit,但 ...
python调参工作都是干啥的_xgboost原理及调参方法-通俗易懂版本
xgboost是各种比赛中最常使用的方法,网上介绍非常多,但是大部分看起来都比较费劲,这篇文章我将通俗的讲一下xgboost是在干什么,是怎么实现的,每一步的细节中要注意什么问题,达到理解-应用的程度 ...
XgBoost使用及调参教程
Kaggle比赛利器XGBboost 数据,决定了机器学习的上限,而算法,只是逼近这个上限. 简介如果你的机器学习集成模型表现得不是那么尽如人意,那就使用XGBoost吧.XGBoost算法现在已经 ...
集成学习-Boosting集成学习算法XGBoost
XGBoost全名叫(eXtreme Gradient Boosting)极端梯度提升,经常被用在一些项目中,其效果显著.它是大规模并行boosted tree的工具,它是目前最快最好的开源boost ...

集成学习5-Xgboost原理与调参