python3内存分析_调试和分析 - tracemalloc —- 跟踪内存分配 - 《Python 3.7 标准库》 - 书栈网 · BookStack...
tracemalloc —- 跟踪内存分配
3.4 新版功能.
The tracemalloc module is a debug tool to trace memory blocks allocated byPython. It provides the following information:
Traceback where an object was allocated
Statistics on allocated memory blocks per filename and per line number:total size, number and average size of allocated memory blocks
Compute the differences between two snapshots to detect memory leaks
To trace most memory blocks allocated by Python, the module should be startedas early as possible by setting the PYTHONTRACEMALLOC environmentvariable to 1, or by using -X tracemalloc command lineoption. The tracemalloc.start() function can be called at runtime tostart tracing Python memory allocations.
By default, a trace of an allocated memory block only stores the most recentframe (1 frame). To store 25 frames at startup: set thePYTHONTRACEMALLOC environment variable to 25, or use the-X tracemalloc=25 command line option.
例子
显示前10项
显示内存分配最多的10个文件:
importtracemalloc
tracemalloc.start()
# ... run your application ...
snapshot=tracemalloc.take_snapshot()
top_stats=snapshot.statistics('lineno')
print("[ Top 10 ]")
forstatintop_stats[:10]:
print(stat)
Python测试套件的输出示例:
[Top10]
:716:size=4855KiB,count=39328,average=126B
:284:size=521KiB,count=3199,average=167B
/usr/lib/python3.4/collections/__init__.py:368:size=244KiB,count=2315,average=108B
/usr/lib/python3.4/unittest/case.py:381:size=185KiB,count=779,average=243B
/usr/lib/python3.4/unittest/case.py:402:size=154KiB,count=378,average=416B
/usr/lib/python3.4/abc.py:133:size=88.7KiB,count=347,average=262B
:1446:size=70.4KiB,count=911,average=79B
:1454:size=52.0KiB,count=25,average=2131B
:5:size=49.7KiB,count=148,average=344B
/usr/lib/python3.4/sysconfig.py:411:size=48.0KiB,count=1,average=48.0KiB
We can see that Python loaded 4855 KiB data (bytecode and constants) frommodules and that the collections module allocated 244 KiB to buildnamedtuple types.
计算差异
获取两个快照并显示差异:
importtracemalloc
tracemalloc.start()
# ... start your application ...
snapshot1=tracemalloc.take_snapshot()
# ... call the function leaking memory ...
snapshot2=tracemalloc.take_snapshot()
top_stats=snapshot2.compare_to(snapshot1,'lineno')
print("[ Top 10 differences ]")
forstatintop_stats[:10]:
print(stat)
Example of output before/after running some tests of the Python test suite:
[Top10differences]
:716:size=8173KiB(+4428KiB),count=71332(+39369),average=117B
/usr/lib/python3.4/linecache.py:127:size=940KiB(+940KiB),count=8106(+8106),average=119B
/usr/lib/python3.4/unittest/case.py:571:size=298KiB(+298KiB),count=589(+589),average=519B
:284:size=1005KiB(+166KiB),count=7423(+1526),average=139B
/usr/lib/python3.4/mimetypes.py:217:size=112KiB(+112KiB),count=1334(+1334),average=86B
/usr/lib/python3.4/http/server.py:848:size=96.0KiB(+96.0KiB),count=1(+1),average=96.0KiB
/usr/lib/python3.4/inspect.py:1465:size=83.5KiB(+83.5KiB),count=109(+109),average=784B
/usr/lib/python3.4/unittest/mock.py:491:size=77.7KiB(+77.7KiB),count=143(+143),average=557B
/usr/lib/python3.4/urllib/parse.py:476:size=71.8KiB(+71.8KiB),count=969(+969),average=76B
/usr/lib/python3.4/contextlib.py:38:size=67.2KiB(+67.2KiB),count=126(+126),average=546B
We can see that Python has loaded 8173 KiB of module data (bytecode andconstants), and that this is 4428 KiB more than had been loaded before thetests, when the previous snapshot was taken. Similarly, the linecachemodule has cached 940 KiB of Python source code to format tracebacks, allof it since the previous snapshot.
If the system has little free memory, snapshots can be written on disk usingthe Snapshot.dump() method to analyze the snapshot offline. Then use theSnapshot.load() method reload the snapshot.
Get the traceback of a memory block
Code to display the traceback of the biggest memory block:
importtracemalloc
# Store 25 frames
tracemalloc.start(25)
# ... run your application ...
snapshot=tracemalloc.take_snapshot()
top_stats=snapshot.statistics('traceback')
# pick the biggest memory block
stat=top_stats[0]
print("%s memory blocks: %.1f KiB"%(stat.count,stat.size/1024))
forlineinstat.traceback.format():
print(line)
Example of output of the Python test suite (traceback limited to 25 frames):
903memory blocks:870.1KiB
File"",line716
File"",line1036
File"",line934
File"",line1068
File"",line619
File"",line1581
File"",line1614
File"/usr/lib/python3.4/doctest.py",line101
importpdb
File"",line284
File"",line938
File"",line1068
File"",line619
File"",line1581
File"",line1614
File"/usr/lib/python3.4/test/support/__init__.py",line1728
importdoctest
File"/usr/lib/python3.4/test/test_pickletools.py",line21
support.run_doctest(pickletools)
File"/usr/lib/python3.4/test/regrtest.py",line1276
test_runner()
File"/usr/lib/python3.4/test/regrtest.py",line976
display_failure=notverbose)
File"/usr/lib/python3.4/test/regrtest.py",line761
match_tests=ns.match_tests)
File"/usr/lib/python3.4/test/regrtest.py",line1563
main()
File"/usr/lib/python3.4/test/__main__.py",line3
regrtest.main_in_temp_cwd()
File"/usr/lib/python3.4/runpy.py",line73
exec(code,run_globals)
File"/usr/lib/python3.4/runpy.py",line160
"__main__",fname,loader,pkg_name)
We can see that the most memory was allocated in the importlib module toload data (bytecode and constants) from modules: 870.1 KiB. The traceback iswhere the importlib loaded data most recently: on the import pdbline of the doctest module. The traceback may change if a new module isloaded.
Pretty top
Code to display the 10 lines allocating the most memory with a pretty output,ignoring and files:
importlinecache
importos
importtracemalloc
defdisplay_top(snapshot,key_type='lineno',limit=10):
snapshot=snapshot.filter_traces((
tracemalloc.Filter(False,""),
tracemalloc.Filter(False,""),
))
top_stats=snapshot.statistics(key_type)
print("Top %s lines"%limit)
forindex,statinenumerate(top_stats[:limit],1):
frame=stat.traceback[0]
# replace "/path/to/module/file.py" with "module/file.py"
filename=os.sep.join(frame.filename.split(os.sep)[-2:])
print("#%s: %s:%s: %.1f KiB"
%(index,filename,frame.lineno,stat.size/1024))
line=linecache.getline(frame.filename,frame.lineno).strip()
ifline:
print(' %s'%line)
other=top_stats[limit:]
ifother:
size=sum(stat.sizeforstatinother)
print("%s other: %.1f KiB"%(len(other),size/1024))
total=sum(stat.sizeforstatintop_stats)
print("Total allocated size: %.1f KiB"%(total/1024))
tracemalloc.start()
# ... run your application ...
snapshot=tracemalloc.take_snapshot()
display_top(snapshot)
Python测试套件的输出示例:
Top10lines
#1: Lib/base64.py:414: 419.8 KiB
_b85chars2=[(a+b)forain_b85charsforbin_b85chars]
#2: Lib/base64.py:306: 419.8 KiB
_a85chars2=[(a+b)forain_a85charsforbin_a85chars]
#3: collections/__init__.py:368: 293.6 KiB
exec(class_definition,namespace)
#4: Lib/abc.py:133: 115.2 KiB
cls=super().__new__(mcls,name,bases,namespace)
#5: unittest/case.py:574: 103.1 KiB
testMethod()
#6: Lib/linecache.py:127: 95.4 KiB
lines=fp.readlines()
#7: urllib/parse.py:476: 71.8 KiB
forain_hexdigforbin_hexdig}
#8: :5: 62.0 KiB
#9: Lib/_weakrefset.py:37: 60.0 KiB
self.data=set()
#10: Lib/base64.py:142: 59.8 KiB
_b32tab2=[a+bforain_b32tabforbin_b32tab]
6220other:3602.8KiB
Totalallocated size:5303.1KiB
API
函数tracemalloc.clear_traces()
Clear traces of memory blocks allocated by Python.
See also stop().
tracemalloc.getobject_traceback(_obj)
Get the traceback where the Python object obj was allocated.Return a Traceback instance, or None if the tracemallocmodule is not tracing memory allocations or did not trace the allocation ofthe object.
tracemalloc.get_traceback_limit()
Get the maximum number of frames stored in the traceback of a trace.
The tracemalloc module must be tracing memory allocations toget the limit, otherwise an exception is raised.
The limit is set by the start() function.
tracemalloc.get_traced_memory()
Get the current size and peak size of memory blocks traced by thetracemalloc module as a tuple: (current: int, peak: int).
tracemalloc.get_tracemalloc_memory()
Get the memory usage in bytes of the tracemalloc module used to storetraces of memory blocks.Return an int.
tracemalloc.is_tracing()
True if the tracemalloc module is tracing Python memoryallocations, False otherwise.
See also start() and stop() functions.
tracemalloc.start(nframe: int=1)
Start tracing Python memory allocations: install hooks on Python memoryallocators. Collected tracebacks of traces will be limited to nframe_frames. By default, a trace of a memory block only stores the most recentframe: the limit is 1. _nframe must be greater or equal to 1.
Storing more than 1 frame is only useful to compute statistics groupedby 'traceback' or to compute cumulative statistics: see theSnapshot.compare_to() and Snapshot.statistics() methods.
Storing more frames increases the memory and CPU overhead of thetracemalloc module. Use the get_tracemalloc_memory() functionto measure how much memory is used by the tracemalloc module.
The PYTHONTRACEMALLOC environment variable(PYTHONTRACEMALLOC=NFRAME) and the -X tracemalloc=NFRAMEcommand line option can be used to start tracing at startup.
tracemalloc.stop()
Stop tracing Python memory allocations: uninstall hooks on Python memoryallocators. Also clears all previously collected traces of memory blocksallocated by Python.
Call take_snapshot() function to take a snapshot of traces beforeclearing them.
tracemalloc.take_snapshot()
Take a snapshot of traces of memory blocks allocated by Python. Return a newSnapshot instance.
The snapshot does not include memory blocks allocated before thetracemalloc module started to trace memory allocations.
Tracebacks of traces are limited to get_traceback_limit() frames. Usethe nframe parameter of the start() function to store more frames.
The tracemalloc module must be tracing memory allocations to take asnapshot, see the start() function.
域过滤器classtracemalloc.DomainFilter(inclusive: bool, domain: int)
Filter traces of memory blocks by their address space (domain).
3.6 新版功能.
inclusive
If inclusive is True (include), match memory blocks allocatedin the address space domain.
If inclusive is False (exclude), match memory blocks not allocatedin the address space domain.
domain
Address space of a memory block (int). Read-only property.
过滤器classtracemalloc.Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False, domain: int=None)
对内存块的跟踪进行筛选。
See the fnmatch.fnmatch() function for the syntax offilename_pattern. The '.pyc' file extension isreplaced with '.py'.
示例:
Filter(True, subprocess.file) only includes traces of thesubprocess module
Filter(False, tracemalloc.file) excludes traces of thetracemalloc module
Filter(False, "") excludes empty tracebacks
在 3.5 版更改: The '.pyo' file extension is no longer replaced with '.py'.
在 3.6 版更改: Added the domain attribute.
domain
Address space of a memory block (int or None).
tracemalloc uses the domain 0 to trace memory allocations made byPython. C extensions can use other domains to trace other resources.
inclusive
If inclusive is True (include), only match memory blocks allocatedin a file with a name matching filename_pattern at line numberlineno.
If inclusive is False (exclude), ignore memory blocks allocated ina file with a name matching filename_pattern at line numberlineno.
lineno
Line number (int) of the filter. If lineno is None, the filtermatches any line number.
filename_pattern
Filename pattern of the filter (str). Read-only property.
all_frames
If all_frames is True, all frames of the traceback are checked. Ifall_frames is False, only the most recent frame is checked.
This attribute has no effect if the traceback limit is 1. See theget_traceback_limit() function and Snapshot.traceback_limitattribute.
Frameclasstracemalloc.Frame
Frame of a traceback.
The Traceback class is a sequence of Frame instances.
filename
文件名(str).
lineno
行号 (int).
快照classtracemalloc.Snapshot
Snapshot of traces of memory blocks allocated by Python.
The take_snapshot() function creates a snapshot instance.
compareto(_old_snapshot: Snapshot, key_type: str, cumulative: bool=False)
Compute the differences with an old snapshot. Get statistics as a sortedlist of StatisticDiff instances grouped by key_type.
See the Snapshot.statistics() method for key_type and _cumulative_parameters.
The result is sorted from the biggest to the smallest by: absolute valueof StatisticDiff.size_diff, StatisticDiff.size, absolutevalue of StatisticDiff.count_diff, Statistic.count andthen by StatisticDiff.traceback.
dump(filename)
将快照写入文件
使用 load() 重载快照。
filtertraces(_filters)
Create a new Snapshot instance with a filtered tracessequence, filters is a list of DomainFilter andFilter instances. If filters is an empty list, return a newSnapshot instance with a copy of the traces.
All inclusive filters are applied at once, a trace is ignored if noinclusive filters match it. A trace is ignored if at least one exclusivefilter matches it.
在 3.6 版更改: DomainFilter instances are now also accepted in filters.
classmethodload(filename)
从文件载入快照。
statistics(key_type: str, cumulative: bool=False)
获取 Statistic 信息列表,按 key_type 分组排序:
key_type
描述
'filename'
filename
'lineno'
文件名和行号
'traceback'
回溯
If cumulative is True, cumulate size and count of memory blocks ofall frames of the traceback of a trace, not only the most recent frame.The cumulative mode can only be used with key_type equals to'filename' and 'lineno'.
The result is sorted from the biggest to the smallest by:Statistic.size, Statistic.count and then byStatistic.traceback.
traceback_limit
Maximum number of frames stored in the traceback of traces:result of the get_traceback_limit() when the snapshot was taken.
traces
Traces of all memory blocks allocated by Python: sequence ofTrace instances.
The sequence has an undefined order. Use the Snapshot.statistics()method to get a sorted list of statistics.
统计classtracemalloc.Statistic
统计内存分配
count
内存块数(整形)。
size
Total size of memory blocks in bytes (int).
traceback
Traceback where the memory block was allocated, Tracebackinstance.
StatisticDiffclasstracemalloc.StatisticDiff
Statistic difference on memory allocations between an old and a newSnapshot instance.
Snapshot.compare_to() returns a list of StatisticDiffinstances. See also the Statistic class.
count
Number of memory blocks in the new snapshot (int): 0 ifthe memory blocks have been released in the new snapshot.
count_diff
Difference of number of memory blocks between the old and the newsnapshots (int): 0 if the memory blocks have been allocated inthe new snapshot.
size
Total size of memory blocks in bytes in the new snapshot (int):0 if the memory blocks have been released in the new snapshot.
size_diff
Difference of total size of memory blocks in bytes between the old andthe new snapshots (int): 0 if the memory blocks have beenallocated in the new snapshot.
traceback
Traceback where the memory blocks were allocated, Tracebackinstance.
跟踪classtracemalloc.Trace
Trace of a memory block.
The Snapshot.traces attribute is a sequence of Traceinstances.
在 3.6 版更改: Added the domain attribute.
domain
Address space of a memory block (int). Read-only property.
tracemalloc uses the domain 0 to trace memory allocations made byPython. C extensions can use other domains to trace other resources.
size
Size of the memory block in bytes (int).
traceback
Traceback where the memory block was allocated, Tracebackinstance.
回溯classtracemalloc.Traceback
Sequence of Frame instances sorted from the oldest frame to themost recent frame.
A traceback contains at least 1 frame. If the tracemalloc modulefailed to get a frame, the filename "" at line number 0 isused.
When a snapshot is taken, tracebacks of traces are limited toget_traceback_limit() frames. See the take_snapshot() function.
The Trace.traceback attribute is an instance of Tracebackinstance.
在 3.7 版更改: Frames are now sorted from the oldest to the most recent, instead of most recent to oldest.
format(limit=None, most_recent_first=False)
Format the traceback as a list of lines with newlines. Use thelinecache module to retrieve lines from the source code.If limit is set, format the limit most recent frames if limit_is positive. Otherwise, format the abs(limit) oldest frames.If _most_recent_first is True, the order of the formatted framesis reversed, returning the most recent frame first instead of last.
Similar to the traceback.format_tb() function, except thatformat() does not include newlines.
示例:
print("Traceback (most recent call first):")
forlineintraceback:
print(line)
输出:
Traceback(most recent call first):
File"test.py",line9
obj=Object()
File"test.py",line12
tb=tracemalloc.get_object_traceback(f())
python3内存分析_调试和分析 - tracemalloc —- 跟踪内存分配 - 《Python 3.7 标准库》 - 书栈网 · BookStack...相关推荐
- mysql pmod项目_内置函数 - 数学函数 - 《Apache Doris 文档(201812)》 - 书栈网 · BookStack...
数学函数 abs(double a) 功能: 返回参数的绝对值 返回类型:double类型 使用说明:使用该函数需要确保函数的返回值是整数. acos(double a) 功能: 返回参数的反余弦值 ...
- dateutil 日期计算_日期时间 - 日期时间工具-DateUtil - 《Hutool 参考文档》 - 书栈网 · BookStack...
日期时间工具-DateUtil 由来 考虑到Java本身对日期时间的支持有限,并且Date和Calendar对象的并存导致各种方法使用混乱和复杂,故使用此工具类做了封装.这其中的封装主要是日期和字符串 ...
- linux 分析磁盘性能,03.分析性能瓶颈 - 3.4.磁盘瓶颈 - 《Linux性能调优指南》 - 书栈网 · BookStack...
磁盘瓶颈磁盘瓶颈性能调优选项 磁盘子系统通常是服务器性能的最重要方面,是瓶颈问题的高发部件.但是,磁盘问题表现的有时候并不是那么直接,比如说可能是内存不足.如果CPU周期浪费在等待I/O任务完成,应用 ...
- spark 相关性分析_基本统计 - correlations(相关性系数) - 《spark机器学习算法研究和源码分析》 - 书栈网 · BookStack...
相关性系数 计算两个数据集的相关性是统计中的常用操作.在MLlib中提供了计算多个数据集两两相关的方法.目前支持的相关性方法有皮尔森(Pearson)相关和斯皮尔曼(Spearman)相关. Stat ...
- mysql t 保存_检查 (调试) - 离线消息保存到 MySQL - 《EMQ X Enterprise v4.1 中文文档》 - 书栈网 · BookStack...
离线消息保存到 MySQL 搭建 MySQL 数据库,并设置用户名密码为 root/public,以 MacOS X 为例: $ brew install mysql $ brew services ...
- mysql集群跨地域同步部署_跨地域冗余 - 跨数据中心部署方案 - 《TiDB v2.1 用户文档》 - 书栈网 · BookStack...
跨数据中心部署方案 作为 NewSQL 数据库,TiDB 兼顾了传统关系型数据库的优秀特性以及 NoSQL 数据库可扩展性,以及跨数据中心(下文简称"中心")场景下的高可用.本文档 ...
- python3怎么使用pyrex_用户指南 - Cython 和 Pyrex 之间的区别 - 《Cython 3.0 中文文档》 - 书栈网 · BookStack...
Cython 和 Pyrex 之间的区别 警告 Cython 和 Pyrex 都是移动目标.已经到了这一点,两个项目之间所有差异的明确列表将很难列出和跟踪,但希望这个高级列表能够了解存在的差异.应该注 ...
- mediumtext和string转换_数据类型 - 字符串类型 - 《TiDB v3.0 用户文档》 - 书栈网 · BookStack...
字符串类型 TiDB 支持 MySQL 所有的字符串类型,包括 CHAR.VARCHAR.BINARY.VARBINARY.BLOB.TEXT.ENUM 以及 SET,完整信息参考这篇文档. 类型定义 ...
- wps在线预览接口_金山文档在线编辑 - 快速接入 - 《WPS开放平台技术文档》 - 书栈网 · BookStack...
快速接入 一.申请和上线流程如下: 1.申请Appid和SecretKey 需要前往https://open.wps.cn 注册服务商,并且申请开通金山文档在线编辑服务. 2.实现回调接口 根据本文档 ...
最新文章
- spell_picture3.1版本windows上手动拼图的软件的升级
- mac上的终端bash命令
- python 矩形补正方形
- c语言比较麻烦的编程题,C语言编程题,比较简单
- 等差数列划分 II - 子序列(动态规划)
- 将两大小完全相同的照片进行加权混合对比
- 诺奖文章里面的动图绘制教程来了!!
- 华为云·云享专家公开课:45分钟掌握Python项目部署与调度核心逻辑直播
- [转]pycharm的一些快捷键
- jasper 常用知识点总结
- 【廖雪峰官方网站/Java教程】多线程(1)
- 【开源】微信小程序、小游戏以及 Web 通用 Canvas 渲染引擎 - Cax
- margin background_div盒子的外部距离(margin)【202】。
- 网络 如何解决输入路由器管理地址192.168.1.1进不去
- 数据结构(王道计算机考研笔记)
- 命令行查看硬盘序列号
- vuejs登陆页面_20个最佳Vuejs登陆页面模板
- 连接防火墙/路由器的几种方式
- 日均5亿查询量,京东到家订单中心ES架构演进
- 皇甫兵兵php,段颎 汉 文旅网 - powered by phpwind.net