Rofiah Adeshina

罗菲亚·阿德希娜(Rofiah Adeshina)

Follow, Aug 28, 2020

关注 ，2020年8月28日

Outline

大纲

• Introduction介绍
• Exploratory Data Analysis探索性数据分析
• Data Preprocessing数据预处理
• Building the model and Results建立模型和结果
• Conclusion结论

Introduction

介绍

Predictive models are now used in all industries to identify and predict important metrics. In this scenario, a bank is being considered. The bank collected a huge amount of data that includes profiles of those customers who have to subscribe to term deposits and those who did not subscribe to a term deposit. The task is to come up with a robust predictive model that would help them identify customers who would or would not subscribe to their term deposits in the future.

现在，所有行业都使用预测模型来识别和预测重要指标。 在这种情况下，正在考虑一家银行。 该银行收集了大量数据，其中包括必须订阅定期存款的客户和未订阅定期存款的客户的资料。 我们的任务是提出一个强大的预测模型，该模型将帮助他们确定将来是否愿意购买定期存款的客户。

The goal is to carry out data exploration, data cleaning, feature extraction, and developing robust machine learning algorithms that would aid them in the department.

目标是进行数据探索，数据清理，特征提取，并开发强大的机器学习算法，以帮助他们在部门中工作。

Exploratory Data Analysis

探索性数据分析

The data is loaded as shown below

数据加载如下图所示

df = pd.read_csv(r’C:\Users\HP\Desktop\CV, P.Statement and others\10 Academy\week 6\bank-additional\bank-additional\bank-additional-full.csv’, sep=’;’)df.head()

The data highlights profiles of customers who have or haven’t subscribed to the bank’s term deposit. The data used is the bank additional full CSV file, with 41188 rows and 21 columns. Full details including column description can be gotten from the link above. An overview of the loaded data is shown below:

数据突出显示了已订阅或未订阅银行定期存款的客户的个人资料。 使用的数据是bank附加的完整CSV文件 ，具有41188行和21列。 可以从上面的链接中获得包括列说明在内的全部详细信息。 加载数据的概述如下所示：

The normal practice of exploratory data analysis is adopted here. By using the .info() function on the data frame, it is seen that the data has no missing values with 11 categorical columns. Also, a statistical description of the data is shown below.

探索性数据分析的常规方法在这里采用。 通过在数据帧上使用.info()函数，可以看到该数据没有包含11个分类列的缺失值。 另外，下面显示了数据的统计描述。

df.describe()

Uni-variate analysis

单变量分析

#Univariante Analysisdef univariante_plot(column, data): plot = sns.catplot(x=column, data=data, kind=’count’) plot.set_xticklabels(rotation=45) plt.show() return plotunivariante_plot('y', df)univariante_plot('job', df)univariante_plot('education', df)univariante_plot('marital', df)univariante_plot('loan', df)univariante_plot('housing', df)

Uni-variate analysis of all categorical columns shows the diverse group of customers the bank has. The categorical column “y” represents the client’s response (yes/no) to the term subscription. It is the target variable for the predictive analysis and shows a class imbalance.

所有分类列的单变量分析显示了银行拥有的不同客户群。 类别列“ y”代表客户对术语订阅的响应(是/否)。 它是预测分析的目标变量，显示类别不平衡。

# Distribution plotsdef distribution_plot(col, col_distribution): plot = sns.distplot(df[col], kde=False, color=’red’, bins=10) plt.title(col_distribution, fontsize=18) plt.xlabel(col, fontsize=16) plt.ylabel(‘Frequency’, fontsize=16) plt.show() return plotdistribution_plot('age', 'Age distribution')distribution_plot('cons.price.idx', 'Price distribution')

As part of the univariate analysis, the distribution of the numerical columns are also shown

作为单变量分析的一部分，还显示了数值列的分布

Bivariate Analysis

双变量分析

This shows how the features relate to one another

这说明了功能之间的相互关系

#bivariante Analysisdef bivariante_plot(cat, num, data, hue): plot = sns.catplot(x=cat, y=num, data=data, kind=’box’, hue=hue) plot.set_xticklabels(rotation=45) plt.show() return plotbivariante_plot('marital', 'age', df, 'y')bivariante_plot('job','age', df, 'y')

From the univariate analysis, it is seen that there is a class imbalance in the target variable “y” with our main target “yes” being the minority class. Feeding this imbalanced data to the classifier model can make it biased in favor of the majority class “no”, simply because it did not have enough data to learn about the minority. It is therefore essential to balance the classes before feeding it to the model.

从单变量分析可以看出，目标变量“ y”存在类不平衡，而我们的主要目标“是”为少数类。 将这种不平衡的数据馈送到分类器模型可能会使它偏向多数类“否”，这仅仅是因为它没有足够的数据来了解少数派。 因此，在将类输入模型之前，必须平衡类。

Data Preprocessing

数据预处理

This is data preparation for modeling and machine learning. The feature to be predicted (“target/dependent” feature) is the response to the term deposit subscription (“y” column). The other features that we use for the prediction are called the “descriptive/independent” features. There is a certain format for the data before we can perform modeling using Scikit-Learn. The following steps are followed for data preparation:

这是用于建模和机器学习的数据准备。 要预测的功能(“目标/相关”功能)是对定期存款订阅的响应(“ y”列)。 我们用于预测的其他功能称为“描述性/独立性”功能。 在使用Scikit-Learn执行建模之前，数据有某种格式。 请按照以下步骤进行数据准备：

1. Outliers and unusual values (such as a negative age) are taken care of: they are replaced with mean.离群值和异常值(例如负数年龄)将得到处理：将它们替换为均值。
2. Any categorical descriptive feature is encoded to be numeric as follows: one-hot-encoding, as this is a classification problem, the target feature is label-encoded. (in case of a binary problem, the positive class is encoded as 1).任何分类描述性特征都按如下方式编码为数字：一键编码，因为这是一个分类问题，所以目标特征是标签编码的。 (如果是二进制问题，则将正类编码为1)。
3. All descriptive features (which are all numeric at this point) are scaled.所有描述性特征(此时全部为数字)已缩放。
4. Splitting the data into test and train data set (90% train and 10% validation)将数据分为测试和训练数据集(90％训练和10％验证)
5. Dealing with the class imbalance in the target feature处理目标要素中的类不平衡

1.Taking Care of Outliers

1.照顾离群值

To deal with outliers the automatic outliers detection Isolate forest was used, however, it dropped over 4000 rows of the data set which is way too much. As an alternative, feature importance can be done and outliers of the most important features are dealt with. The function below generates box plots that help visualize these features that contain outliers.

为了处理离群值，使用了自动离群值检测“隔离林”，但是，它丢弃了4000多行数据集，这实在太多了。 作为替代方案，可以完成功能重要性，并处理最重要功能的异常值。 下面的函数生成箱形图，以帮助可视化包含离群值的这些特征。

#boxplots for outlier detectiondef outlier(col, col_distribution): chart = sns.boxplot(x=df[col]) plt.title(col_distribution, fontsize=18) plt.xlabel(col, fontsize=16) plt.show() return chartoutlier('age', 'Age distribution')outlier('campaign', 'Campaign distribution')

The function below detects the outliers and replaces them with the median.

下面的函数将检测离群值并将其替换为中位数。

#Treating the outliers def replace_outlier_with_median(dataFrame, feature): “”” a function for replacing outliers with the median, used when there’s too many outliers in a feature””” Q1 = dataFrame[feature].quantile(0.25) Q3 = dataFrame[feature].quantile(0.75) median = dataFrame[feature].quantile(0.50)IQR = Q3 — Q1upper_whisker = Q3 + (1.5 * IQR) lower_whisker = Q1 — (1.5 * IQR)dataFrame[feature] = np.where(dataFrame[feature] > upper_whisker, median, dataFrame[feature]) dataFrame[feature] = np.where(dataFrame[feature] < lower_whisker, median, dataFrame[feature])# replace in "age" and "campaign" with medianreplace_outlier_with_median(df, 'age')replace_outlier_with_median(df, 'campaign')

2. Encoding Categorical features

2.编码分类特征

The categorical columns in that are descriptive features are one hot encoded, while that of the target feature is label encoded.

描述性要素中的类别列是一种热编码，而目标要素的类别列是标签编码。

# create an object of the OneHotEncoderOHE = ce.OneHotEncoder(cols=[‘job’,’marital’,’education’,’default’,’housing’,’loan’,’contact’,’month’,’day_of_week’,’poutcome’],use_cat_names=True)# encode the categorical variablesdf = OHE.fit_transform(df)df.head()#label encoding target# creating instance of labelencoderlbe = LabelEncoder()# Assigning numerical values and storing in another columntarget = lbe.fit_transform(df.y)target =  pd.DataFrame(target, columns = ['y'])target.head()

3. Scaling data

3.缩放数据

Once all categorical descriptive features are encoded, all features in this transformed dataset will be numerical. It is always a good idea to scale these numerical descriptive features before fitting the model, as scaling is mandatory for some models especially those that consider Euclidean distance. The main idea why this is done is that some variables are often measured at different scales and would not contribute equally to model fitting and this may lead the trained model to create some bias. Here the numerical columns from the data frame are scaled

一旦所有分类描述性特征都被编码，此变换后的数据集中的所有特征将为数字。 在拟合模型之前对这些数字描述特征进行缩放总是一个好主意，因为对于某些模型，尤其是那些考虑欧几里得距离的模型，缩放是必不可少的。 这样做的主要思想是，某些变量通常以不同的比例进行测量，并且对模型拟合的贡献不尽相同，这可能导致受过训练的模型产生一些偏差。 在此缩放数据框中的数字列

# data standardization with sklearnfrom sklearn.preprocessing import StandardScalerscaler = StandardScaler()num_cols = [‘emp.var.rate’, ‘cons.price.idx’, ‘cons.conf.idx’, ‘euribor3m’, ‘nr.employed’, ‘age’, ‘pdays’,’campaign’,’previous’]Feature[num_cols].head()Feature[num_cols] = scaler.fit_transform(Feature[num_cols])Feature.head()

4. Split data into test and train

4.将数据分为测试和训练

The data is split in the ratio 9:1 using 90% for training and 10% for validation.

数据以9：1的比例分割，其中90％用于训练，而10％用于验证。

#splitting into test and trainX_train, X_test, y_train, y_test = train_test_split(Feature, target, test_size=0.10, random_state=15)print (“Training and testing split was successful.”)X_train.head()

5. Class Imbalance

5.班级失衡

In classification problems, balanced data is very important. Data is said to be imbalanced when instances of one class outnumber the other(s) by a large proportion as it is seen in the target feature “y”.

在分类问题中，平衡数据非常重要。 当在目标特征“ y”中看到一个类别的实例比另一个类别的实例大很多时，则认为数据不平衡。

Feeding imbalanced data to your classifier can make it biased in favor of the majority class, simply because it did not have enough data to learn about the minority. To deal with imbalance the oversampling method SMOTE (Synthetic Minority Over-Sampling Technique is used to generate synthetic data for the minority class.

向分类器提供不平衡的数据可能会使它偏向多数类，这仅仅是因为它没有足够的数据来了解少数。 为了解决不平衡的过采样方法SMOTE(S ynthetic 中号 inorityöVER-采样碲 chnique被用来生成用于少数类合成数据。

#dealing with class imbalancefrom imblearn.over_sampling import SMOTEsm = SMOTE(random_state = 33)X_train_new, y_train_new = sm.fit_sample(X_train, y_train.values.ravel())y_train_n = pd.DataFrame(y_train_new, columns = [“Y”])pd.Series(y_train_new).value_counts().plot.bar()

Building the model

建立模型

1. Fitting the model with k-fold cross-validation using various machine learning classifier algorithms. Determine model accuracy.使用各种机器学习分类器算法对模型进行k折交叉验证。 确定模型的准确性。
2. Prediction with validation set.带有验证集的预测。

1. Fitting the model with k-fold cross-validation using various machine learning classifier algorithms

1.使用各种机器学习分类器算法对模型进行k折交叉验证

Having split the data into test and train to fit and validate the model, K-fold cross-validation is adopted to further improve model evaluation. However, the stratified version is used as the k-fold cross-validation is not appropriate for evaluating imbalanced classifiers.

将数据分为测试和训练以拟合和验证模型后，采用K折交叉验证进一步改善了模型评估。 但是，分层版本用作k折交叉验证不适用于评估不平衡的分类器。

The reason being that the data is split into k-folds with a uniform probability distribution. This might work fine for data with a balanced class distribution, however, when the distribution is severely skewed (class imbalance), it is likely that one or more folds will have few or no examples from the minority class. This means that some or perhaps many of the model evaluations will be misleading, as the model need only predict the majority class correctly. The function below is used to implement stratified k-fold cross validation

原因是数据以均匀的概率分布分为k倍。 这对于具有均衡的类别分布的数据可能会很好，但是，当分布严重偏斜(类别不平衡)时，很可能一个或多个折叠很少或没有少数类别的示例。 这意味着某些或许多模型评估会产生误导，因为模型仅需要正确预测多数类。 以下功能用于实现分层的k倍交叉验证

from sklearn.model_selection import StratifiedKFold def model_predictor(model, x, y): scores = [] kf = StratifiedKFold(n_splits=10, shuffle=True, random_state=1) for train_index, test_index in kf.split(X_train, y_train):KX_train, KX_test = X_train.iloc[train_index], X_train.iloc[test_index] Ky_train, Ky_test = y_train.iloc[train_index], y_train.iloc[test_index]trained_model = model.fit(KX_train, Ky_train) scores.append(model.score(X = KX_test ,y = Ky_test)) return trained_model, scores

Three different machine learning algorithms were used to build predictive models and the accuracy for each considered. The logistic regression, random forest, and XGBoost machine learning algorithms were used.

三种不同的机器学习算法用于建立预测模型和每种模型的准确性。 使用了逻辑回归，随机森林和XGBoost机器学习算法。

• Logistic Regression

  逻辑回归

# Initialize logistic regression modelfrom sklearn.linear_model import LogisticRegressionlog_model = LogisticRegression()#Initialize decision treefrom sklearn.ensemble import RandomForestClassifierforest_model = RandomForestClassifier(n_estimators=10,  criterion=’entropy’,random_state=0)#initialize XGBoostimport xgboost as xgbXGB_model = xgb.XGBClassifier()#fit models, predict and determine scoreslog_model_trained_model, log_model_scores,  = model_predictor(log_model, X_train_new, y_train_n)print("Accuracy of the model is" + ":" +str(np.mean(log_model_scores)))print("std of scores computed" + ":" +str(np.std(log_model_scores)))#make predictions with validation sety_pred_log = pd.DataFrame(log_model.predict(X_test), columns=["Term Deposit Predictions"])df_log = pd.concat([y_test,y_pred_log], axis=1)df_log.head()#confusion matrixfrom sklearn.metrics import confusion_matrixfrom sklearn.metrics import classification_reportconfusion_matrix = confusion_matrix(y_test, y_pred_log)confusion_matrix

The Logistic model accuracy was 0.9008874052983294 (90% accuracy) accurate with a confusion matrix

使用混淆矩阵时，Logistic模型的准确度为0.9008874052983294(准确度为90％)

array([[3559,   65],       [ 379,  116]]precision    recall  f1-score   support           0       0.90      0.98      0.94      3624           1       0.64      0.24      0.35       495    accuracy                           0.89      4119   macro avg       0.77      0.61      0.64      4119weighted avg       0.87      0.89      0.87      4119

Though the accuracy of the model is pretty high, the confusion matrix precision shows that the major target “yes” is only predicted correctly 64% of the time.

尽管该模型的准确性很高，但是混淆矩阵的准确性表明，主要目标“是”只能在64％的时间正确预测。

• Random Forest Classifier

  随机森林分类器

This algorithm is used to train the model trying to determine if model performance improves.

该算法用于训练模型，以确定模型性能是否得到改善。

#fit models, predict and determine scoresforest_model_trained_model, forest_model_scores = model_predictor(forest_model, X_train_new, y_train_n.values.ravel())print(“Accuracy of the model is” + “:” +str(np.mean(forest_model_scores)))print(“std of scores computed” + “:” +str(np.std(forest_model_scores)))#make predictions with validation sety_pred_forest = pd.DataFrame(forest_model.predict(X_test), columns=[“Term Deposit Predictions”])print(y_pred_forest.head())y_test.head()

The Random forest model accuracy was 0.8898539627847784 (89 % accuracy) accurate with a confusion matrix

带有混淆矩阵的随机森林模型精度为0.8898539627847784(精度为89％)

[[3558   66] [ 378  117]]              precision    recall  f1-score   support           0       0.90      0.98      0.94      3624           1       0.64      0.24      0.35       495    accuracy                           0.89      4119   macro avg       0.77      0.61      0.64      4119weighted avg       0.87      0.89      0.87      4119

as was the case in logistic regression.

与逻辑回归一样。

Feature Importance

功能重要性

With the random forest algorithm, the features that contribute the most to the prediction can be known. Below shows the first 8 features that contributed the most to term deposit prediction.

利用随机森林算法，可以知道对预测贡献最大的特征。 下面显示了对定期存款预测贡献最大的前8个功能。

#plot the 7 most important features plt.figure(figsize=(10,7))feat_importances = pd.Series(forest_model.feature_importances_, index = Feature.columns)feat_importances.nlargest(10).plot(kind=’barh’);

• XGBoost

  XGBoost

The final machine learning algorithm considered was the XGBoost

最后考虑的机器学习算法是XGBoost

The XGBoost model accuracy was 0.8979199297838093 (89 % accuracy) accurate with a confusion matrix

带有混淆矩阵的XGBoost模型精度为0.8979199297838093(精度为89％)

[[3517  107] [ 355  140]]              precision    recall  f1-score   support           0       0.91      0.97      0.94      3624           1       0.57      0.28      0.38       495    accuracy                           0.89      4119   macro avg       0.74      0.63      0.66      4119weighted avg       0.87      0.89      0.87      4119

Though the precision is relatively same as that of random forest, the presicion is lower.

尽管精度与随机森林的精度相对相同，但精度较低。

Conclusion

结论

Other machine learning classification algorithms can be used to train the model and their performances determined. So far the models though have good accuracy don’t possess a good enough precision in terms of predicting a ‘yes’ for a term deposit. Further analysis can be done to improve the model such as hyperparameter tuning of the different machine learning algorithms employed, training the model with the important features from feature engineering.

其他机器学习分类算法可用于训练模型并确定其性能。 到目前为止，尽管模型具有良好的准确性，但在预测定期存款的“是”方面还没有足够好的精度。 可以进行进一步的分析来改进模型，例如对所采用的不同机器学习算法进行超参数调整，并使用特征工程中的重要特征训练模型。

• How to Deal with Imbalanced Data using SMOTE

  如何使用SMOTE处理不平衡的数据

• How to Fix k-Fold Cross-Validation for Imbalanced Classification

  如何解决k折交叉验证的不平衡分类

• 4 Automatic Outlier Detection Algorithms in Python

  Python中的4种自动离群值检测算法