TuShare(2)：使用TuShare，抓取股票数据并存储到数据库

1年前200

1，关于股票数据存储

股票数据比较特殊，需要做数据统计的。都需要一次进行批量查询多个数据，然后进行分析。 所以股票数据不一定要放到数据库中存储。因为一般就两个维度。 那只股票，和那天的股票信息，然后使用模型进行分析预测。 所以数据可以存储为：/data/stock/yyyy/yyyMM/yyyyMMdd.hdf5 存储的数据是hdf5： Hierarchical DataFormat，可以存储不同类型的图像和数码数据的文件格式

2，安装hdf5库

#首先安装hdf5库yum -y install hdf5 hdf5-devel pip install

unittest2 pip install--upgrade tables

3，测试读写hdf5文件

参考pytables官方文档。

# python Python 2.6.6 (r266:84292, Jul 23

2015,15:22:56) [GCC 4.4.7 20120313 (Red Hat 4.4.7-11)] on linux2

Type"help", "copyright", "credits" or "license" for more

information.>>> import

numpy as np>>> import

pandas aspd ###su >>>

a

=np.random.standard_normal((9,4))>>>

b =pd.DataFrame(a)

>>>b.columns =

[['num1','num2','num3','num4']]>>>

aarray([[-2.36198849, -1.27547933, -1.40351822, -0.6638619 ],

[1.89159066, -0.31838519, -0.2065942 , -1.02327987],

[-2.02771503,-0.81333254, -0.93644288, -0.91592467], [-1.3939496 ,

0.25899342,1.11591841, -0.7423286 ], [-1.05104415, -0.79069151,

-1.46536873,-0.01449547], [ 1.32483444, 0.32030117, -1.23575344,

0.51455106], [0.91297435, 0.43242834, 1.77235337, 1.14879289], [

0.93476429,0.18592698, 0.30198234, -0.61861642], [ 0.04462872,

-0.99275411,-0.86382085,

-1.53064223]])>>> b

num1 num2 num3num4 0 -2.361988 -1.275479 -1.403518 -0.663862 1

1.891591 -0.318385-0.206594 -1.023280 2 -2.027715 -0.813333

-0.936443 -0.915925 3-1.393950 0.258993 1.115918 -0.742329 4

-1.051044 -0.790692-1.465369 -0.014495 5 1.324834 0.320301

-1.235753 0.514551 60.912974 0.432428 1.772353 1.148793 7 0.934764

0.185927 0.301982-0.618616 8 0.044629 -0.992754 -0.863821

-1.530642>>> b.sum()

num1-1.725905 num2 -2.992993 num3 -2.921244 num4 -3.845805

dtype:float64 >>>

b.mean()num1 -0.191767 num2 -0.332555 num3 -0.324583 num4 -0.427312

dtype:float64###写hdf5文件：>>>

h5

=pd.HDFStore('/data/stock/test1.h5','w')>>>

h5['data'] = b>>>

h5.close()>>>>>>

b num1 num2 num3num4 0 -2.361988 -1.275479 -1.403518 -0.663862 1

1.891591 -0.318385-0.206594 -1.023280 2 -2.027715 -0.813333

-0.936443 -0.915925 3-1.393950 0.258993 1.115918 -0.742329 4

-1.051044 -0.790692-1.465369 -0.014495 5 1.324834 0.320301

-1.235753 0.514551 60.912974 0.432428 1.772353 1.148793 7 0.934764

0.185927 0.301982-0.618616 8 0.044629 -0.992754 -0.863821

-1.530642###读hdf5文件。>>>

h5

=pd.HDFStore('/data/stock/test1.h5','r')>>>

c = h5['data']>>> c

num1 num2 num3num4 0 -2.361988 -1.275479 -1.403518 -0.663862 1

1.891591 -0.318385-0.206594 -1.023280 2 -2.027715 -0.813333

-0.936443 -0.915925 3-1.393950 0.258993 1.115918 -0.742329 4

-1.051044 -0.790692-1.465369 -0.014495 5 1.324834 0.320301

-1.235753 0.514551 60.912974 0.432428 1.772353 1.148793 7 0.934764

0.185927 0.301982-0.618616 8 0.044629 -0.992754 -0.863821

-1.530642>>>

h5.close()

4，TuShare读取股票数据存储

>>>import

tushare as ts>>> d

=ts.get_tick_data('600848',date='2015-01-09')>>>

type(d)

'pandas.core.frame.DataFrame'>>>>

len(d) 1212###保存数据>>>

h5

=pd.HDFStore('/data/stock/test2.h5','w')>>>

h5['data'] = d>>>

h5.close()>>>>>>###读取数据>>>

h5

=pd.HDFStore('/data/stock/test2.h5','r')>>>

e = h5['data']>>>

h5.close()>>>>>>

type(e)

'pandas.core.frame.DataFrame'>>>>

len(e) 1212

5，总结

使用文件直接存储到本地还是非常方便的。pandas直接封装的函数3行解决问题。 同时pandas封装的读出数据还是数组，可以直接进行操作。灰常方便。