我有一个约50GB的fits文件,其中包含多个HDU,它们都具有相同的格式：一个带有1E5对象和1E6时间戳的(1E5 x 1E6)数组. HDU描述了不同的物理属性,例如磁通,RA,DEC等.我只想从每个HDU中读取5个对象(即(5 x 1E6)阵列).

python 2.7,

糟糕的1.0.3,

linux x86_64

到目前为止,我尝试了很多发现的建议,但是没有任何效果.我最好的方法仍然是：

#the five objects I want to read out

obj_list = ['Star1','Star15','Star700','Star2000','Star5000']

dic = {}

with fits.open(fname, memmap=True, do_not_scale_image_data=True) as hdulist:

# There is a special HDU 'OBJECTS' which is an (1E5 x 1) array and contains the info which index in the fits file corresponds to which object.

# First, get the indices of the rows that describe the objects in the fits file (not necessarily in order!)

ind_objs = np.in1d(hdulist['OBJECTS'].data, obj_list, assume_unique=True).nonzero()[0] #indices of the candidates

# Second, read out the 5 object's time series

dic['FLUX'] = hdulist['FLUX'].data[ind_objs] # (5 x 1E6) array

dic['RA'] = hdulist['RA'].data[ind_objs] # (5 x 1E6) array

dic['DEC'] = hdulist['DEC'].data[ind_objs] # (5 x 1E6) array

此代码可以很好地且快速地适合约20 GB的文件,但对于较大的文件则用尽内存(较大的文件仅包含更多对象,而没有更多时间戳).我不明白为什么-astropy.io.fits本质上使用mmap,并且据我所知应该仅将(5x1E6)数组加载到内存中？因此,与文件大小无关,我要读出的内容始终具有相同的大小.

编辑-这是错误消息：

dic['RA'] = hdulist['RA'].data[ind_objs] # (5 x 1E6) array

File "/usr/local/python/lib/python2.7/site-packages/astropy-1.0.3-py2.7-linux-x86_64.egg/astropy/utils/decorators.py", line 341, in __get__

val = self._fget(obj)

File "/usr/local/python/lib/python2.7/site-packages/astropy-1.0.3-py2.7-linux-x86_64.egg/astropy/io/fits/hdu/image.py", line 239, in data

data = self._get_scaled_image_data(self._data_offset, self.shape)

File "/usr/local/python/lib/python2.7/site-packages/astropy-1.0.3-py2.7-linux-x86_64.egg/astropy/io/fits/hdu/image.py", line 585, in _get_scaled_image_data

raw_data = self._get_raw_data(shape, code, offset)

File "/usr/local/python/lib/python2.7/site-packages/astropy-1.0.3-py2.7-linux-x86_64.egg/astropy/io/fits/hdu/base.py", line 523, in _get_raw_data

return self._file.readarray(offset=offset, dtype=code, shape=shape)

File "/usr/local/python/lib/python2.7/site-packages/astropy-1.0.3-py2.7-linux-x86_64.egg/astropy/io/fits/file.py", line 248, in readarray

shape=shape).view(np.ndarray)

File "/usr/local/python/lib/python2.7/site-packages/numpy/core/memmap.py", line 254, in __new__

mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)

mmap.error: [Errno 12] Cannot allocate memory

编辑2：

谢谢,我现在包含了建议,它使我能够处理最大50GB的装订文件.新代码：

#the five objects I want to read out

obj_list = ['Star1','Star15','Star700','Star2000','Star5000']

dic = {}

with fits.open(fname, mode='denywrite', memmap=True, do_not_scale_image_data=True) as hdulist:

# There is a special HDU 'OBJECTS' which is an (1E5 x 1) array and contains the info which index in the fits file corresponds to which object.

# First, get the indices of the rows that describe the objects in the fits file (not necessarily in order!)

ind_objs = np.in1d(hdulist['OBJECTS'].data, obj_list, assume_unique=True).nonzero()[0] #indices of the candidates

# Second, read out the 5 object's time series

dic['FLUX'] = hdulist['FLUX'].data[ind_objs] # (5 x 1E6) array

del hdulist['FLUX'].data

dic['RA'] = hdulist['RA'].data[ind_objs] # (5 x 1E6) array

del hdulist['RA'].data

dic['DEC'] = hdulist['DEC'].data[ind_objs] # (5 x 1E6) array

del hdulist['DEC'].data

的

mode='denywrite'

没有引起任何变化.

memmap=True

确实不是默认值,需要手动设置.

del hdulist['FLUX'].data

等现在允许我读取50GB而不是20GB的文件

新问题：

大于50GB的任何内容仍然会导致相同的内存错误-但是,现在直接在第一行中.

dic['FLUX'] = hdulist['FLUX'].data[ind_objs] # (5 x 1E6) array