数据才是王道→数据预处理与数据集构建

from IPython.display import Image
%matplotlib inline
# Added version check for recent scikit-learn 0.18 checks
from distutils.version import LooseVersion as Version
from sklearn import __version__ as sklearn_version

1.处理缺省值

import pandas as pd
from io import StringIOcsv_data = '''A,B,C,D
1.0,2.0,3.0,4.0
5.0,6.0,,8.0
10.0,11.0,12.0,'''# If you are using Python 2.7, you need
# to convert the string to unicode:
csv_data = unicode(csv_data)df = pd.read_csv(StringIO(csv_data))
df

df.isnull().sum()

2.可以直接删除缺省值多的样本或者特征

df.dropna()   #默认行
df.dropna(axis=1)
# only drop rows where all columns are NaN
df.dropna(how='all')
# drop rows that have not at least 4 non-NaN values
df.dropna(thresh=4)
# only drop rows where NaN appear in specific columns (here: 'C')
df.dropna(subset=['C'])

3.重新计算缺省值

from sklearn.preprocessing import Imputerimr = Imputer(missing_values='NaN', strategy='mean', axis=0)
imr = imr.fit(df)
imputed_data = imr.transform(df.values)
imputed_data
df.values

4.处理类别型数据

import pandas as pddf = pd.DataFrame([['green', 'M', 10.1, 'class1'],['red', 'L', 13.5, 'class2'],['blue', 'XL', 15.3, 'class1']])df.columns = ['color', 'size', 'price', 'classlabel']
df

5.序列特征映射

size_mapping = {'XL': 3,
'L': 2,
'M': 1}
df['size'] = df['size'].map(size_mapping)
df
inv_size_mapping = {v: k for k, v in size_mapping.items()}
df['size'].map(inv_size_mapping)

6.类别编码

import numpy as npclass_mapping = {label: idx for idx, label in enumerate(np.unique(df['classlabel']))}
class_mapping
df['classlabel'] = df['classlabel'].map(class_mapping)
df
inv_class_mapping = {v: k for k, v in class_mapping.items()}
df['classlabel'] = df['classlabel'].map(inv_class_mapping)
df
from sklearn.preprocessing import LabelEncoderclass_le = LabelEncoder()
y = class_le.fit_transform(df['classlabel'].values)
y
class_le.inverse_transform(y)

7.对类别型的特征用one-hot编码

X = df[['color', 'size', 'price']].valuescolor_le = LabelEncoder()
X[:, 0] = color_le.fit_transform(X[:, 0])
X

array([[1, 1, 10.1],[2, 2, 13.5],[0, 3, 15.3]], dtype=object)

from sklearn.preprocessing import OneHotEncoderohe = OneHotEncoder(categorical_features=[0])
ohe.fit_transform(X).toarray()

array([[  0. ,   1. ,   0. ,   1. ,  10.1],[  0. ,   0. ,   1. ,   2. ,  13.5],[  1. ,   0. ,   0. ,   3. ,  15.3]])

pd.get_dummies(df[['price', 'color', 'size']])

8.对连续值特征做幅度缩放(scaling)

from sklearn.preprocessing import MinMaxScaler
mms = MinMaxScaler()
X_train_norm = mms.fit_transform(X_train)
X_test_norm = mms.transform(X_test)

from sklearn.preprocessing import StandardScalerstdsc = StandardScaler()
X_train_std = stdsc.fit_transform(X_train)
X_test_std = stdsc.transform(X_test)

A visual example:

ex = pd.DataFrame([0, 1, 2, 3, 4, 5])# standardize
ex[1] = (ex[0] - ex[0].mean()) / ex[0].std(ddof=0)# Please note that pandas uses ddof=1 (sample standard deviation)
# by default, whereas NumPy's std method and the StandardScaler
# uses ddof=0 (population standard deviation)# normalize
ex[2] = (ex[0] - ex[0].min()) / (ex[0].max() - ex[0].min())
ex.columns = ['input', 'standardized', 'normalized']
ex

9.特征选择

通过L1正则化的截断性效应选择,不重要的都为0，特征矩阵变成稀疏矩阵。

from sklearn.linear_model import LogisticRegressionlr = LogisticRegression(penalty='l1', C=0.1)
lr.fit(X_train_std, y_train)
print('Training accuracy:', lr.score(X_train_std, y_train))
print('Test accuracy:', lr.score(X_test_std, y_test))

lr.intercept_

lr.coef_

10.通过随机森林对特征重要性排序

from sklearn.ensemble import RandomForestClassifierfeat_labels = df_wine.columns[1:]forest = RandomForestClassifier(n_estimators=10000,random_state=0,n_jobs=-1)forest.fit(X_train, y_train)
importances = forest.feature_importances_indices = np.argsort(importances)[::-1]for f in range(X_train.shape[1]):print("%2d) %-*s %f" % (f + 1, 30, feat_labels[indices[f]], importances[indices[f]]))plt.title('Feature Importances')
plt.bar(range(X_train.shape[1]), importances[indices],color='lightblue', align='center')plt.xticks(range(X_train.shape[1]), feat_labels[indices], rotation=90)
plt.xlim([-1, X_train.shape[1]])
plt.tight_layout()
#plt.savefig('./random_forest.png', dpi=300)
plt.show()

if Version(sklearn_version) < '0.18':X_selected = forest.transform(X_train, threshold=0.15)
else:from sklearn.feature_selection import SelectFromModelsfm = SelectFromModel(forest, threshold=0.15, prefit=True)X_selected = sfm.transform(X_train)X_selected.shape

Now, let's print the 3 features that met the threshold criterion for feature selection that we set earlier (note that this code snippet does not appear in the actual book but was added to this notebook later for illustrative purposes):

for f in range(X_selected.shape[1]):print("%2d) %-*s %f" % (f + 1, 30, feat_labels[indices[f]], importances[indices[f]]))

sklearn机器学习常用数据处理总结相关推荐

sklearn机器学习常用过程总结
由于前面对sklearn或多或少接触了一下,但是不深入,随着最近学习,我下面介绍一下机器学习常用过程. 1. 加载数据集 scikit-learn中自带了一些数据集,比如说最著名的Iris数据集. 数 ...
机器学习常用数据处理
一,自定义数据样本 from sklearn import datasets import matplotlib.pyplot as plt# create customized data,noise ...
机器学习29:Sklearn库常用分类器及效果比较
机器学习29:Sklearn库常用分类器及效果比较 1.Sklearn库常用分类器: #[1] KNN Classifier # k-近邻分类器 from sklearn.neighbors impo ...
机器学习常用的六种分类方法，Python代码详细都在这里!
机器学习常用的六种分类方法,Python代码详细都在这里! 六种常用分类方法包括两种线性分类及四种非线性分类法,分别是: 一.线性判别分析 from sklearn.model_selection i ...
《sklearn机器学习第二版》（加文海克著）学习笔记
本书附以sklearn机器学习示例程序,从调用函数的角度解释了常用的机器学习的方法,包括线性回归.逻辑回归.决策树.SVM.朴素贝叶斯.ANN.K-means.PCA.原理粗浅易懂,注重代码实践.本文 ...
机器学习常用激活函数
机器学习常用激活函数摘要: 激活函数就是神经网络输出端的一个节点,来连接两个神经网络.本文主要介绍激活函数的功能以及激活函数类型. 什么是激活函数? 激活函数就是神经网络输出端的一个节点,来连接两个 ...
HiveSQL常用数据处理语句
HiveSQL运行优化参数配置 HiveSQL常用数据处理语句 HiveSQL中复杂数据类型操作 Hive 采用了类似SQL 的查询语言 HQL(Hive Query Language),因此很容易将 ...
cfile read 最大读取限制_pandas读取表格后的常用数据处理操作
作者丨Sp4rkW来源丨凹凸数据大家好,我是Sp4rkW今天给大家讲讲pandas读取表格后的一些常用数据处理操作.这篇文章其实来源于自己的数据挖掘课程作业,通过完成老师布置的作业,感觉对于使用pyt ...
【计算机视觉】计算机视觉、模式识别、机器学习常用牛人主页链接
计算机视觉.模式识别.机器学习常用牛人主页链接牛人主页(主页有很多论文代码) Serge Belongie at UC San Diego Antonio Torralba at MIT Alexe ...

sklearn机器学习常用数据处理总结

数据才是王道→数据预处理与数据集构建

sklearn机器学习常用数据处理总结相关推荐

最新文章

热门文章