我在从文件中读取图表时遇到了类似的问题 . 处理包括计算不适合存储器的200 000x200 000浮点矩阵(一次一行) . 试使用 gc.collect() 释放计算之间的内存来修复问题的内存相关方面但是它导致性能问题：我不知道为什么但是即使使用的内存量保持不变，每次调用 gc.collect() 的时间也要花费更多时间比前一个 . 很快，垃圾收集占用了大部分的计算时间 .

为了解决内存和性能问题，我转而使用多线程技巧，我曾在某处读过(对不起，我再也找不到相关的帖子) . 在我在一个大的 for 循环中读取文件的每一行之前，处理它，并且每隔一段时间运行 gc.collect() 以释放内存空间 . 现在我调用一个函数来读取和处理新线程中的文件块 . 一旦线程结束，内存将自动释放，而不会出现奇怪的性能问题 .

实际上它的工作原理如下：

from dask import delayed # this module wraps the multithreading

def f(storage, index, chunk_size): # the processing function

# read the chunk of size chunk_size starting at index in the file

# process it using data in storage if needed

# append data needed for further computations to storage

return storage

partial_result = delayed([]) # put into the delayed() the constructor for your data structure

# I personally use "delayed(nx.Graph())" since I am creating a networkx Graph

chunk_size = 100 # ideally you want this as big as possible while still enabling the computations to fit in memory

for index in range(0, len(file), chunk_size):

# we indicates to dask that we will want to apply f to the parameters partial_result, index, chunk_size

partial_result = delayed(f)(partial_result, index, chunk_size)

# no computations are done yet !

# dask will spawn a thread to run f(partial_result, index, chunk_size) once we call partial_result.compute()

# passing the previous "partial_result" variable in the parameters assures a chunk will only be processed after the previous one is done

# it also allows you to use the results of the processing of the previous chunks in the file if needed

# this launches all the computations

result = partial_result.compute()

# one thread is spawned for each "delayed" one at a time to compute its result

# dask then closes the tread, which solves the memory freeing issue

# the strange performance issue with gc.collect() is also avoided