【Python学习系列二十七】pearson相关系数计算

场景：计算训练特征和目标之间的相关系数，用于判断是否加入训练。

参考代码：

# -*- coding: utf-8 -*-import pandas as pd
import time
from sklearn import tree
import numpy as np
from sklearn import metrics
from sklearn.linear_model import LinearRegression
from scipy.stats import pearsonr
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import f_regressiondef main():#加载标记数据label_ds=pd.read_csv(r"link_train_0726.txt",sep='\t',encoding='utf8',\names=['link_id','length','width','link_class','start_date','week','time_interval','time_slot','travel_time',\'avg_travel_time','sd_travel_time','inlinks_num','outlinks_num']) label_ds["link_id"] = label_ds["link_id"].astype("string")label_ds["length"] = label_ds["length"].astype("int")label_ds["width"] = label_ds["width"].astype("int")label_ds["link_class"] = label_ds["link_class"].astype("int")label_ds["start_date"] = label_ds["start_date"].astype("string")label_ds["week"] = label_ds["week"].astype("int")label_ds["time_interval"] = label_ds["time_interval"].astype("string")label_ds["time_slot"] = label_ds["time_slot"].astype("int")label_ds["travel_time"] = label_ds["travel_time"].astype("float")label_ds["avg_travel_time"] = label_ds["avg_travel_time"].astype("float")label_ds["sd_travel_time"] = label_ds["sd_travel_time"].astype("float")label_ds["inlinks_num"] = label_ds["inlinks_num"].astype("int")label_ds["outlinks_num"] = label_ds["outlinks_num"].astype("int")#加载预测数据   unlabel_ds=pd.read_csv(r"link_test_0726.txt",sep='\t',encoding='utf8',\names=['link_id','length','width','link_class','start_date','week','time_interval','time_slot',\'avg_travel_time','sd_travel_time','inlinks_num','outlinks_num']) unlabel_ds["link_id"] = unlabel_ds["link_id"].astype("string")unlabel_ds["length"] = unlabel_ds["length"].astype("int")unlabel_ds["width"] = unlabel_ds["width"].astype("int")unlabel_ds["link_class"] = unlabel_ds["link_class"].astype("int")unlabel_ds["start_date"] = unlabel_ds["start_date"].astype("string")unlabel_ds["week"] = unlabel_ds["week"].astype("int")unlabel_ds["time_interval"] = unlabel_ds["time_interval"].astype("string")unlabel_ds["time_slot"] = unlabel_ds["time_slot"].astype("int")unlabel_ds["avg_travel_time"] = unlabel_ds["avg_travel_time"].astype("float")unlabel_ds["sd_travel_time"] = unlabel_ds["sd_travel_time"].astype("float")unlabel_ds["inlinks_num"] = unlabel_ds["inlinks_num"].astype("int")unlabel_ds["outlinks_num"] = unlabel_ds["outlinks_num"].astype("int")#提取训练集、验证集、测试集train_df=label_ds.loc[(pd.to_datetime(label_ds["start_date"])<'2016-06-01')]#训练集print "训练集，有", train_df.shape[0], "行", train_df.shape[1], "列"valid_df=label_ds.loc[(pd.to_datetime(label_ds["start_date"])>='2016-06-01')]#验证集train_df.sample(frac=0.2)print "验证集，有", valid_df.shape[0], "行", valid_df.shape[1], "列"test_df=unlabel_ds#测试集print "测试集，有", test_df.shape[0], "行", test_df.shape[1], "列"#特征选择p_X=train_df['outlinks_num']#训练属性p_Y=train_df['travel_time']#目标属性p_value=pearsonr(p_X,p_Y)print p_value#选择相关性最强的k个特征参与训练#k_feature = f_regression(p_X,p_Y)#k_fearture=SelectKBest(lambda X, Y: np.array(map(lambda x:pearsonr(x, Y), X.T)).T, k=9).fit_transform(p_X, p_Y)#print k_fearture#模型训练train_X=train_df[['length','width','link_class','week','time_slot','avg_travel_time']]train_y = train_df['travel_time']#标记model_lr=LinearRegression()#tree.DecisionTreeRegressor()model_lr.fit(train_X, train_y)   #模型验证valid_X=valid_df[['length','width','link_class','week','time_slot','avg_travel_time']]valid_y=valid_df['travel_time']pre_valid_y=model_lr.predict(valid_X)abs_y=abs(pre_valid_y-valid_y)abs_error=abs_y.sum()#求和#abs_error=sum(list(abs_y))#求和print "mape:",abs_error/valid_df.shape[0]print "RMSE:",np.sqrt(metrics.mean_squared_error(valid_y, pre_valid_y))#均方差，模型评估#模型预测test_X = test_df[['length','width','link_class','week','time_slot','avg_travel_time']]  test_info = test_df[['link_id','start_date','time_interval']] test_X=test_X.fillna(0)#空值替换为0test_y=model_lr.predict(test_X) pre_test_y=pd.DataFrame(test_y,columns=['travel_time']) outset=test_info.join(pre_test_y,how='left')#输出结果 #outset["travel_time"]=outset["travel_time"].apply(lambda x: '{0:.3f}'.format(x))outset.to_csv('outit.txt',sep='#',index=False,header=None)#输出预测数据 #执行
if __name__ == '__main__':  start = time.clock()  main()end = time.clock()  print('finish all in %s' % str(end - start))

pearsonx函数的说明：https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html

scikit-learn库中：f_regression和SelectKBest用于选择最佳特征训练，可以批量给出前k个特征。

【Python学习系列二十七】pearson相关系数计算相关推荐

uniapp 学习笔记二十七购物车总价计算及弹窗交互逻辑完善
uniapp 学习笔记二十七购物车总价计算及弹窗交互逻辑完善 uniapp 学习笔记二十七购物车总价计算及弹窗交互逻辑完善 cart.vue <template><view> ...
【Python学习系列二十三】Scikit_Learn库降维方法(矩阵分解)-PCAFA
1主成分分析PCA 1.1 精确PCA和似然估计 PCA基于最大方差的正交变量分解多维数据集.在scikit-learn库中,PCA的实现是先通过fit方法计算n维的特征值和特征向量,然后通过tran ...
【Python学习系列二十一】pandas库基本操作
pandas很强大,操作参考官网:http://pandas.pydata.org/pandas-docs/stable/ 也有一份10分钟入门的材料:http://pandas.pydata.org ...
python学习笔记(二十七) -- 常用内建模块(二) Base64、MD5、SHA1、hmac
目录 base64 hashlib MD5 SHA1 hmac base64 原理 base64就是对二进制数据进行编码,比如我有6字节的二进制数据,然后每3个字节分为一组,也就是一组有3*8 = 2 ...
Python学习日记(二十七) 反射和几个内置函数
isinstance() 判断isinstance(obj,cls)中obj是否是cls类的对象 class Person:def __init__(self,name):self.name = na ...
【Python学习系列二十六】networkx库图最短路径求解
场景:基于python库networkx来求解图最短路径,相关算法基础参考 http://blog.csdn.net/fjssharpsword/article/details/52931373 ht ...
【Python学习系列二十】scikit-learn库模型持久化
场景:需要将模型保存到内存,或磁盘. 代码: # -*- coding: utf-8 -*-import pandas as pd import pickle as pkl from sklearn. ...
【Python学习系列二】Python默认编码和Eclipse环境的冲突问题
1.问题:在Eclipse+PyDev开发环境中写个 print 'Hello,python!'; 执行却提示: SyntaxError: Non-ASCII character '\xc4' 2.原 ...
python networkx进行最短路径分析_【Python学习系列二十六】networkx库图最短路径求解...
场景:基于python库networkx来求解图最短路径,相关算法基础参考 http://blog.csdn.net/fjssharpsword/article/details/52931373 ht ...

【Python学习系列二十七】pearson相关系数计算

【Python学习系列二十七】pearson相关系数计算相关推荐

最新文章

热门文章