堆栈溢出，

我正在从事一个涉及一些大文件(10-50Gb)的基因组学项目，我想将这些文件读入Python 2.7进行处理。我不需要将整个文件读到内存中，而是简单地逐行读取每个文件，执行一个小任务，然后继续。

我发现类似的问题，并尝试实施一些解决方案：

当我在17Gb文件上运行以下代码时：

脚本1(itertools)：

#!/usr/bin/env python2importsysimportstringimportosimportitertoolsif__name__=="__main__":#Read in PosListposList=[]withopen("BigFile")asf:forlineiniter(f):posList.append(line.strip())sys.stdout.write(str(sys.getsizeof(posList)))

脚本2(文件输入)：

#!/usr/bin/env python2importsysimportstringimportosimportfileinputif__name__=="__main__":#Read in PosListposList=[]forlineinfileinput.input(['BigFile']):posList.append(line.strip())sys.stdout.write(str(sys.getsizeof(posList)))

SCRIPT3(用于行)：

#!/usr/bin/env python2importsysimportstringimportosif__name__=="__main__":#Read in PosListposList=[]withopen("BigFile")asf:forlineinf:posList.append(line.strip())sys.stdout.write(str(sys.getsizeof(posList)))

SCRIPT4(收益)：

#!/usr/bin/env python2importsysimportstringimportosdefreadInChunks(fileObj,chunkSize=30):whileTrue:data=fileObj.read(chunkSize)ifnotdata:breakyielddataif__name__=="__main__":#Read in PosListposList=[]f=open('BigFile')forchunkinreadInChunks(f):posList.append(chunk.strip())f.close()sys.stdout.write(str(sys.getsizeof(posList)))

在17Gb文件中，Python中最终列表的大小约为5％(来自sys.getsizeof())，但根据“顶部”，每个脚本使用的内存都超过43Gb。

我的问题是：为什么内存使用率比输入文件或最终列表高得多？如果最终列表只有5Gb，并且正在逐行读取17Gb文件输入，为什么每个脚本的内存使用量都达到约43Gb？有没有更好的方法可以读取大文件而不会发生内存泄漏(如果那就是它们的话)？

非常感谢。

编辑：

从'/ usr / bin / time -v python script3.py'输出：

Commandbeing timed:"python script3.py"Usertime(seconds):159.65Systemtime(seconds):21.74Percentof CPU this job got:99%Elapsed(wall clock)time(h:mm:ssorm:ss):3:01.96Averageshared text size(kbytes):0Averageunshared data size(kbytes):0Averagestack size(kbytes):0Averagetotal size(kbytes):0Maximumresident set size(kbytes):181246448Averageresident set size(kbytes):0Major(requiring I/O)page faults:0Minor(reclaiming a frame)page faults:10182731Voluntarycontext switches:315Involuntarycontext switches:16722Swaps:0Filesystem inputs:33831512Filesystem outputs:0Socketmessages sent:0Socketmessages received:0Signalsdelivered:0Pagesize(bytes):4096Exitstatus:0

从顶部输出：

15816user200727m609m2032R76.80.50:02.31python15816user2001541m1.4g2032R99.61.10:05.31python15816user2002362m2.2g2032R99.61.70:08.31python15816user2003194m3.0g2032R99.62.40:11.31python15816user2004014m3.8g2032R99.630:14.31python15816user2004795m4.6g2032R99.63.60:17.31python15816user2005653m5.3g2032R99.64.20:20.31python15816user2006457m6.1g2032R99.34.90:23.30python15816user2007260m6.9g2032R99.65.50:26.30python15816user2008085m7.7g2032R99.96.10:29.31python15816user2008809m8.5g2032R99.66.70:32.31python15816user2009645m9.3g2032R99.37.40:35.30python15816user20010.3g10g2032R99.680:38.30python15816user20011.1g10g2032R1008.60:41.31python15816user20011.8g11g2032R99.99.20:44.32python15816user20012.7g12g2032R99.39.90:47.31python15816user20013.4g13g2032R99.610.50:50.31python15816user20014.3g14g2032R99.911.10:53.32python15816user20015.0g14g2032R99.311.70:56.31python15816user20015.9g15g2032R99.912.40:59.32python15816user20016.6g16g2032R99.6131:02.32python15816user20017.3g17g2032R99.613.61:05.32python15816user20018.2g17g2032R99.914.21:08.33python15816user20018.9g18g2032R99.614.91:11.33python15816user20019.9g19g2032R10015.51:14.34python15816user20020.6g20g2032R99.316.11:17.33python15816user20021.3g21g2032R99.616.71:20.33python15816user20022.3g21g2032R99.917.41:23.34python15816user20023.0g22g2032R99.6181:26.34python15816user20023.7g23g2032R99.618.61:29.34python15816user20024.4g24g2032R99.619.21:32.34python15816user20025.4g25g2032R99.319.91:35.33python15816user20026.1g25g2032R99.920.51:38.34python15816user20026.8g26g2032R99.921.11:41.35python15816user20027.4g27g2032R99.621.71:44.35python15816user20028.5g28g2032R99.622.31:47.35python15816user20029.2g28g2032R99.922.91:50.36python15816user20029.9g29g2032R99.623.51:53.36python15816user20030.5g30g2032R99.624.11:56.36python15816user20031.6g31g2032R99.624.71:59.36python15816user20032.3g31g2032R10025.32:02.37python15816user20033.0g32g2032R99.625.92:05.37python15816user20033.7g33g2032R99.626.52:08.37python15816user20034.3g34g2032R99.627.12:11.37python15816user20035.5g34g2032R99.627.72:14.37python15816user20036.2g35g2032R99.628.42:17.37python15816user20036.9g36g2032R100292:20.38python15816user20037.5g37g2032R99.629.62:23.38python15816user20038.2g38g2032R99.630.22:26.38python15816user20038.9g38g2032R99.630.82:29.38python15816user20040.1g39g2032R10031.42:32.39python15816user20040.8g40g2032R99.6322:35.39python15816user20041.5g41g2032R99.632.62:38.39python15816user20042.2g41g2032R99.933.22:41.40python15816user20042.8g42g2032R99.633.82:44.40python15816user20043.4g43g2032R99.634.32:47.40python15816user20043.4g43g2032R10034.32:50.41python15816user20038.6g38g2032R10030.52:53.43python15816user20024.9g24g2032R99.719.62:56.43python15816user20012.0g11g2032R1009.42:59.44python

编辑2：

为了进一步澄清，这里是问题的扩展。我在这里所做的是读取FASTA文件中的位置列表(Contig1 / 1，Contig1 / 2等)。通过以下方式将其转换为充满N的字典：

keys=posList

values=['N']*len(posList)speciesDict=dict(zip(keys,values))

然后，我正在读取多个物种的堆积文件，再次逐行(将存在相同问题的地方)，并通过以下方式获得最终的碱基检出：

withopen(path+'/'+os.path.basename(path)+'.pileups',"r")asfilein:forlineiniter(filein):splitline=line.split()iflen(splitline)>4:node,pos,ref,num,bases,qual=line.split()loc=node+'/'+pos

cleanBases=getCleanList(ref,bases)finalBase=getFinalBase_Pruned(cleanBases,minread,thresh)speciesDict[loc]=finalBase

由于特定于物种的堆积文件的长度或顺序不同，因此，我正在创建列表以创建一种“公共花园”方式来存储单个物种数据。如果某个物种的给定站点没有可用的数据，则会调用“ N”。否则，将在词典中为该站点分配一个碱基。

最终结果是每个物种的文件，这些文件是有序的和完整的，我可以从中进行下游分析。

因为逐行读取正在消耗大量内存，所以即使最终数据结构比我预期的所需内存小得多，读取两个大文件也会使我的资源超载(增长列表的大小+单个内存)一次要添加数据的行)。

解决方案

sys.getsizeof(posList) is not giving you what I think you think it is: it's telling you the size of the list object containing the lines; this does not include the size of the lines themselves. Below are some outputs from reading a roughly 3.5Gb file into a list on my system:

In[2]:lines=[]In[3]:withopen('bigfile')asinf:...:forlineininf:...:lines.append(line)...:In[4]:len(lines)Out[4]:68318734In[5]:sys.getsizeof(lines)Out[5]:603811872In[6]:sum(len(l)forlinlines)Out[6]:3473926127In[7]:sum(sys.getsizeof(l)forlinlines)Out[7]:6001719285

That's a bit over six billion bytes, there; in top my interpreter was using about 7.5Gb at this point.

Strings have considerable overhead: 37 bytes each, it looks like:

In[2]:sys.getsizeof('0'*10)Out[2]:47In[3]:sys.getsizeof('0'*100)Out[3]:137In[4]:sys.getsizeof('0'*1000)Out[4]:1037

So if your lines are relatively short, a large part of the memory use will be overhead.