模型融合的理论讲解和些许实战代码

这里，我简单的说一下自己的理解，多模型的stacking在一定程度上其实就是加权融合（还有交叉验证），只是融合的参数是由次级模型优化出来的，所以我们也尽量让次级模型简单一些（线性模型），因为并没有这个必要去复杂化次级模型。

结果层面的融合，这种是最常见的融合方法，其可行的融合方法也有很多，比如根据结果的得分进行加权融合，还可以做Log，exp处理等。在做结果融合的时候，有一个很重要的条件是模型结果的得分要比较近似，然后结果的差异要比较大，这样的结果融合往往有比较好的效果提升。
特征层面的融合，这个层面其实感觉不叫融合，准确说可以叫分割，很多时候如果我们用同种模型训练，可以把特征进行切分给不同的模型，然后在后面进行模型或者结果融合有时也能产生比较好的效果。
模型层面的融合，模型层面的融合可能就涉及模型的堆叠和设计，比如加Staking层，部分模型的结果作为特征输入等，这些就需要多实验和思考了，基于模型层面的融合最好不同模型类型要有一定的差异，用同种模型不同的参数的收益一般是比较小的。

模型融合的目标

对于多种调参完成的模型进行模型融合。

1、本赛题示例

import pandas as pd
import numpy as np
import warnings
import matplotlib
import matplotlib.pyplot as plt
import seaborn as snswarnings.filterwarnings('ignore')
%matplotlib inlineimport itertools
import matplotlib.gridspec as gridspec
from sklearn import datasets
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import RandomForestClassifier
# from mlxtend.classifier import StackingClassifier
from sklearn.model_selection import cross_val_score, train_test_split
# from mlxtend.plotting import plot_learning_curves
# from mlxtend.plotting import plot_decision_regions  from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import train_test_splitfrom sklearn import linear_model
from sklearn import preprocessing
from sklearn.svm import SVR
from sklearn.decomposition import PCA,FastICA,FactorAnalysis,SparsePCAimport lightgbm as lgb
import xgboost as xgb
from sklearn.model_selection import GridSearchCV,cross_val_score
from sklearn.ensemble import RandomForestRegressor,GradientBoostingRegressorfrom sklearn.metrics import mean_squared_error, mean_absolute_error

2、建立多个模型

参数可以根据上一篇博客进行调整

def build_model_lr(x_train,y_train):reg_model = linear_model.LinearRegression()reg_model.fit(x_train,y_train)return reg_modeldef build_model_ridge(x_train,y_train):reg_model = linear_model.Ridge(alpha=0.8)#alphas=range(1,100,5)reg_model.fit(x_train,y_train)return reg_modeldef build_model_lasso(x_train,y_train):reg_model = linear_model.LassoCV()reg_model.fit(x_train,y_train)return reg_modeldef build_model_gbdt(x_train,y_train):estimator =GradientBoostingRegressor(loss='ls',subsample= 0.85,max_depth= 5,n_estimators = 100)param_grid = { 'learning_rate': [0.05,0.08,0.1,0.2],}gbdt = GridSearchCV(estimator, param_grid,cv=3)gbdt.fit(x_train,y_train)print(gbdt.best_params_)# print(gbdt.best_estimator_ )return gbdtdef build_model_xgb(x_train,y_train):model = xgb.XGBRegressor(n_estimators=120, learning_rate=0.08, gamma=0, subsample=0.8,\colsample_bytree=0.9, max_depth=5) #, objective ='reg:squarederror'model.fit(x_train, y_train)return modeldef build_model_lgb(x_train,y_train):estimator = lgb.LGBMRegressor(num_leaves=63,n_estimators = 100)param_grid = {'learning_rate': [0.01, 0.05, 0.1],}gbm = GridSearchCV(estimator, param_grid)gbm.fit(x_train, y_train)return gbm

下面这个函数可以让我们对比predict的结果分布是否一致

def Sta_inf(data):print('_min',np.min(data))print('_max:',np.max(data))print('_mean',np.mean(data))print('_ptp',np.ptp(data))print('_std',np.std(data))print('_var',np.var(data))

2.1、加权融合

# 分割数据集
x_train,x_val,y_train,y_val = train_test_split(train_X,train_Y,test_size=0.3)print('predict XGB...')
model_xgb = build_model_xgb(x_train,y_train)
val_xgb = model_xgb.predict(x_val)
subA_xgb = model_xgb.predict(X_test)print('predict lgb...')
model_lgb = build_model_lgb(x_train,y_train)
val_lgb = model_lgb.predict(x_val)
subA_lgb = model_lgb.predict(X_test)print('predict gbdt...')
model_gbdt = build_model_gbdt(x_train,y_train)
val_gbdt = model_gbdt.predict(x_val)
subA_gbdt = model_gbdt.predict(test_X)# 加权融合函数
def Weighted_method(test_pre1,test_pre2,test_pre3,w=[1/3,1/3,1/3]):Weighted_result = w[0]*pd.Series(test_pre1)+w[1]*pd.Series(test_pre2)+w[2]*pd.Series(test_pre3)return Weighted_result## Init the Weight
w = [0.3,0.4,0.3]## 测试验证集准确度
val_pre = Weighted_method(val_lgb,val_xgb,val_gbdt,w)
MAE_Weighted = mean_absolute_error(y_val,val_pre)
print('MAE of Weighted of val:',MAE_Weighted)## 预测数据部分
subA = Weighted_method(subA_lgb,subA_xgb,subA_gbdt,w)
print('Sta inf:')
print(Sta_inf(subA))

2、回归模型的融合

2.1、加权融合


from sklearn import metrics
## 生成一些简单的样本数据，test_prei 代表第i个模型的预测值
test_pre1 = [1.2, 3.2, 2.1, 6.2]
test_pre2 = [0.9, 3.1, 2.0, 5.9]
test_pre3 = [1.1, 2.9, 2.2, 6.0]# y_test_true 代表第模型的真实值
y_test_true = [1, 3, 2, 6] ## 定义结果的加权平均函数
def Mean_method(test_pre1,test_pre2,test_pre3):Mean_result = pd.concat([pd.Series(test_pre1),pd.Series(test_pre2),pd.Series(test_pre3)],axis=1).mean(axis=1)return Mean_resultMean_pre = Mean_method(test_pre1,test_pre2,test_pre3)
print('Mean_pre MAE:',metrics.mean_absolute_error(y_test_true, Mean_pre))

中位数

## 定义结果的加权平均函数
def Median_method(test_pre1,test_pre2,test_pre3):Median_result = pd.concat([pd.Series(test_pre1),pd.Series(test_pre2),pd.Series(test_pre3)],axis=1).median(axis=1)return Median_resultMedian_pre = Median_method(test_pre1,test_pre2,test_pre3)
print('Median_pre MAE:',metrics.mean_absolute_error(y_test_true, Median_pre))

2.2、stacking

from sklearn import linear_modeldef Stacking_method(train_reg1,train_reg2,train_reg3,y_train_true,test_pre1,test_pre2,test_pre3,model_L2= linear_model.LinearRegression()):model_L2.fit(pd.concat([pd.Series(train_reg1),pd.Series(train_reg2),pd.Series(train_reg3)],axis=1).values,y_train_true)Stacking_result = model_L2.predict(pd.concat([pd.Series(test_pre1),pd.Series(test_pre2),pd.Series(test_pre3)],axis=1).values)return Stacking_result
## 生成一些简单的样本数据，test_prei 代表第i个模型的预测值
train_reg1 = [3.2, 8.2, 9.1, 5.2]
train_reg2 = [2.9, 8.1, 9.0, 4.9]
train_reg3 = [3.1, 7.9, 9.2, 5.0]
# y_test_true 代表第模型的真实值
y_train_true = [3, 8, 9, 5] test_pre1 = [1.2, 3.2, 2.1, 6.2]
test_pre2 = [0.9, 3.1, 2.0, 5.9]
test_pre3 = [1.1, 2.9, 2.2, 6.0]# y_test_true 代表第模型的真实值
y_test_true = [1, 3, 2, 6] model_L2= linear_model.LinearRegression()
Stacking_pre = Stacking_method(train_reg1,train_reg2,train_reg3,y_train_true,test_pre1,test_pre2,test_pre3,model_L2)
print('Stacking_pre MAE:',metrics.mean_absolute_error(y_test_true, Stacking_pre))

3、分类模型的融合

from sklearn.datasets import make_blobs
from sklearn import datasets
from sklearn.tree import DecisionTreeClassifier
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import VotingClassifier
from xgboost import XGBClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_moons
from sklearn.metrics import accuracy_score,roc_auc_score
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import StratifiedKFold

3.1、Voting投票机制

3.1.1、硬投票

'''
硬投票：对多个模型直接进行投票，不区分模型结果的相对重要度，最终投票数最多的类为最终被预测的类。
'''
iris = datasets.load_iris()x=iris.data
y=iris.target
x_train,x_test,y_train,y_test=train_test_split(x,y,test_size=0.3)clf1 = XGBClassifier(learning_rate=0.1, n_estimators=150, max_depth=3, min_child_weight=2, subsample=0.7,colsample_bytree=0.6, objective='binary:logistic')
clf2 = RandomForestClassifier(n_estimators=50, max_depth=1, min_samples_split=4,min_samples_leaf=63,oob_score=True)
clf3 = SVC(C=0.1)# 硬投票
eclf = VotingClassifier(estimators=[('xgb', clf1), ('rf', clf2), ('svc', clf3)], voting='hard')
for clf, label in zip([clf1, clf2, clf3, eclf], ['XGBBoosting', 'Random Forest', 'SVM', 'Ensemble']):scores = cross_val_score(clf, x, y, cv=5, scoring='accuracy')print("Accuracy: %0.2f (+/- %0.2f) [%s]" % (scores.mean(), scores.std(), label))

3.1.2、软投票

'''
软投票：和硬投票原理相同，增加了设置权重的功能，可以为不同模型设置不同权重，进而区别模型不同的重要度。
'''
x=iris.data
y=iris.target
x_train,x_test,y_train,y_test=train_test_split(x,y,test_size=0.3)clf1 = XGBClassifier(learning_rate=0.1, n_estimators=150, max_depth=3, min_child_weight=2, subsample=0.8,colsample_bytree=0.8, objective='binary:logistic')
clf2 = RandomForestClassifier(n_estimators=50, max_depth=1, min_samples_split=4,min_samples_leaf=63,oob_score=True)
clf3 = SVC(C=0.1, probability=True)# 软投票
eclf = VotingClassifier(estimators=[('xgb', clf1), ('rf', clf2), ('svc', clf3)], voting='soft', weights=[2, 1, 1])
clf1.fit(x_train, y_train)for clf, label in zip([clf1, clf2, clf3, eclf], ['XGBBoosting', 'Random Forest', 'SVM', 'Ensemble']):scores = cross_val_score(clf, x, y, cv=5, scoring='accuracy')print("Accuracy: %0.2f (+/- %0.2f) [%s]" % (scores.mean(), scores.std(), label))

3.2、Stacking/Blending

'''
5-Fold Stacking
'''
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import ExtraTreesClassifier,GradientBoostingClassifier
import pandas as pd
#创建训练的数据集
data_0 = iris.data
data = data_0[:100,:]target_0 = iris.target
target = target_0[:100]#模型融合中使用到的各个单模型
clfs = [LogisticRegression(solver='lbfgs'),RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='entropy'),GradientBoostingClassifier(learning_rate=0.05, subsample=0.5, max_depth=6, n_estimators=5)]#切分一部分数据作为测试集
X, X_predict, y, y_predict = train_test_split(data, target, test_size=0.3, random_state=2020)dataset_blend_train = np.zeros((X.shape[0], len(clfs)))
dataset_blend_test = np.zeros((X_predict.shape[0], len(clfs)))#5折stacking
n_splits = 5
skf = StratifiedKFold(n_splits)
skf = skf.split(X, y)for j, clf in enumerate(clfs):#依次训练各个单模型dataset_blend_test_j = np.zeros((X_predict.shape[0], 5))for i, (train, test) in enumerate(skf):#5-Fold交叉训练，使用第i个部分作为预测，剩余的部分来训练模型，获得其预测的输出作为第i部分的新特征。X_train, y_train, X_test, y_test = X[train], y[train], X[test], y[test]clf.fit(X_train, y_train)y_submission = clf.predict_proba(X_test)[:, 1]dataset_blend_train[test, j] = y_submissiondataset_blend_test_j[:, i] = clf.predict_proba(X_predict)[:, 1]#对于测试集，直接用这k个模型的预测值均值作为新的特征。dataset_blend_test[:, j] = dataset_blend_test_j.mean(1)print("val auc Score: %f" % roc_auc_score(y_predict, dataset_blend_test[:, j]))clf = LogisticRegression(solver='lbfgs')
clf.fit(dataset_blend_train, y)
y_submission = clf.predict_proba(dataset_blend_test)[:, 1]print("Val auc Score of Stacking: %f" % (roc_auc_score(y_predict, y_submission)))

'''
Blending
'''#创建训练的数据集
#创建训练的数据集
data_0 = iris.data
data = data_0[:100,:]target_0 = iris.target
target = target_0[:100]#模型融合中使用到的各个单模型
clfs = [LogisticRegression(solver='lbfgs'),RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='entropy'),ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),#ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='entropy'),GradientBoostingClassifier(learning_rate=0.05, subsample=0.5, max_depth=6, n_estimators=5)]#切分一部分数据作为测试集
X, X_predict, y, y_predict = train_test_split(data, target, test_size=0.3, random_state=2020)#切分训练数据集为d1,d2两部分
X_d1, X_d2, y_d1, y_d2 = train_test_split(X, y, test_size=0.5, random_state=2020)
dataset_d1 = np.zeros((X_d2.shape[0], len(clfs)))
dataset_d2 = np.zeros((X_predict.shape[0], len(clfs)))for j, clf in enumerate(clfs):#依次训练各个单模型clf.fit(X_d1, y_d1)y_submission = clf.predict_proba(X_d2)[:, 1]dataset_d1[:, j] = y_submission#对于测试集，直接用这k个模型的预测值作为新的特征。dataset_d2[:, j] = clf.predict_proba(X_predict)[:, 1]print("val auc Score: %f" % roc_auc_score(y_predict, dataset_d2[:, j]))#融合使用的模型
clf = GradientBoostingClassifier(learning_rate=0.02, subsample=0.5, max_depth=6, n_estimators=30)
clf.fit(dataset_d1, y_d2)
y_submission = clf.predict_proba(dataset_d2)[:, 1]
print("Val auc Score of Blending: %f" % (roc_auc_score(y_predict, y_submission)))

3.3、mlxtend进行stacking融合

!pip install mlxtendimport warnings
warnings.filterwarnings('ignore')
import itertools
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspecfrom sklearn import datasets
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import RandomForestClassifier
from mlxtend.classifier import StackingClassifierfrom sklearn.model_selection import cross_val_score
from mlxtend.plotting import plot_learning_curves
from mlxtend.plotting import plot_decision_regions# 以python自带的鸢尾花数据集为例
iris = datasets.load_iris()
X, y = iris.data[:, 1:3], iris.targetclf1 = KNeighborsClassifier(n_neighbors=1)
clf2 = RandomForestClassifier(random_state=1)
clf3 = GaussianNB()
lr = LogisticRegression()
sclf = StackingClassifier(classifiers=[clf1, clf2, clf3], meta_classifier=lr)label = ['KNN', 'Random Forest', 'Naive Bayes', 'Stacking Classifier']
clf_list = [clf1, clf2, clf3, sclf]fig = plt.figure(figsize=(10,8))
gs = gridspec.GridSpec(2, 2)
grid = itertools.product([0,1],repeat=2)clf_cv_mean = []
clf_cv_std = []
for clf, label, grd in zip(clf_list, label, grid):scores = cross_val_score(clf, X, y, cv=3, scoring='accuracy')print("Accuracy: %.2f (+/- %.2f) [%s]" %(scores.mean(), scores.std(), label))clf_cv_mean.append(scores.mean())clf_cv_std.append(scores.std())clf.fit(X, y)ax = plt.subplot(gs[grd[0], grd[1]])fig = plot_decision_regions(X=X, y=y, clf=clf)plt.title(label)plt.show()

4、其他方法

将特征放进模型中预测，并将预测结果变换并作为新的特征加入原有特征中再经过模型预测结果（Stacking变化）¶

def Ensemble_add_feature(train,test,target,clfs):# n_flods = 5# skf = list(StratifiedKFold(y, n_folds=n_flods))train_ = np.zeros((train.shape[0],len(clfs*2)))test_ = np.zeros((test.shape[0],len(clfs*2)))for j,clf in enumerate(clfs):'''依次训练各个单模型'''# print(j, clf)'''使用第1个部分作为预测，第2部分来训练模型，获得其预测的输出作为第2部分的新特征。'''# X_train, y_train, X_test, y_test = X[train], y[train], X[test], y[test]clf.fit(train,target)y_train = clf.predict(train)y_test = clf.predict(test)## 新特征生成train_[:,j*2] = y_train**2test_[:,j*2] = y_test**2train_[:, j+1] = np.exp(y_train)test_[:, j+1] = np.exp(y_test)# print("val auc Score: %f" % r2_score(y_predict, dataset_d2[:, j]))print('Method ',j)train_ = pd.DataFrame(train_)test_ = pd.DataFrame(test_)return train_,test_from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression()data_0 = iris.data
data = data_0[:100,:]target_0 = iris.target
target = target_0[:100]x_train,x_test,y_train,y_test=train_test_split(data,target,test_size=0.3)
x_train = pd.DataFrame(x_train) ; x_test = pd.DataFrame(x_test)#模型融合中使用到的各个单模型
clfs = [LogisticRegression(),RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='entropy'),GradientBoostingClassifier(learning_rate=0.05, subsample=0.5, max_depth=6, n_estimators=5)]New_train,New_test = Ensemble_add_feature(x_train,x_test,y_train,clfs)clf = LogisticRegression()
# clf = GradientBoostingClassifier(learning_rate=0.02, subsample=0.5, max_depth=6, n_estimators=30)
clf.fit(New_train, y_train)
y_emb = clf.predict_proba(New_test)[:, 1]print("Val auc Score of stacking: %f" % (roc_auc_score(y_test, y_emb)))

二手车价格预测——模型融合相关推荐

二手车价格预测-模型stacking融合
模型融合是比赛后期一个重要的环节,大体来说有如下的类型方式. 1简单加权融合: 回归(分类概率):算术平均融合(Arithmetic mean),几何平均融合(Geometric mean): 分类: ...
数据挖掘二手车价格预测 Task05：模型融合
模型融合是kaggle等比赛中经常使用到的一个利器,它通常可以在各种不同的机器学习任务中使结果获得提升.顾名思义,模型融合就是综合考虑不同模型的情况,并将它们的结果融合到一起.模型融合主要通过几部分来 ...
二手车交易价格预测----:模型结果融合
二手车交易价格预测 --模型结果融合通过对赛题的分析,我们可以看出此类问题是对价格进行回归预测,那我们对于数据需要事先做预处理分析,这里我们采用EDA探索性数据分析来进行. 探索性数据分析是对调查, ...
Python二手车价格预测（二）—— 模型训练及可视化
系列文章目录一.Python数据分析-二手车数据获取用于机器学习二手车价格预测二.Python二手车价格预测(一)-- 数据处理文章目录系列文章目录前言一.明确任务二.模型训练 1.引入 ...
【组队学习】【24期】河北邀请赛（二手车价格预测）
河北邀请赛(二手车价格预测) 开源内容: https://github.com/datawhalechina/team-learning-data-mining/tree/master/SecondH ...
基于二手车价格预测——特征工程
特征工程特征工程分析: 第一步:异常值处理箱型图法: 第二步:特征构造第三步:数据分桶数据分桶详解删除不需要的数据特征归一化总结--特征 1.特征构造: 2.异常类型处理 3.构造新特 ...
DNN二手车价格预测完整代码
前言最近在学习深度学习,就用DNN试着跑了个天池赛二手车价格预测,特征还是用之前集成模型跑的特征,通过不断调试模型的学习率.隐藏层数量.神经元数量.优化器.激活函数.迭代次数.batchsize,K ...
数据挖掘-二手车价格预测 Task04：建模调参
数据挖掘-二手车价格预测 Task04:建模调参模型调参部分利用xgb进行五折交叉验证查看模型的参数效果 ## xgb-Model xgr = xgb.XGBRegressor(n_estimat ...
二手车价格预测task03:特征工程
二手车价格预测task03:特征工程 1.学习了operator模块operator.itemgetter()函数 2.学习了箱线图 3.了解了特征工程的方法 (内容介绍) 4.敲代码学习,加注解以 ...

二手车价格预测——模型融合

模型融合的目标

1、本赛题示例

2、建立多个模型

2.1、加权融合

2、回归模型的融合

2.1、加权融合

2.2、stacking

3、分类模型的融合

3.1、Voting投票机制

3.1.1、硬投票

3.1.2、软投票

3.2、Stacking/Blending

3.3、mlxtend进行stacking融合

4、其他方法

二手车价格预测——模型融合相关推荐

最新文章

热门文章