Tensorflow2对GPU内存的分配策略
一、问题源起
从以下的异常堆栈可以看到是BLAS程序集初始化失败，可以看到是执行MatMul的时候发生的异常，基本可以断定可能数据集太大导致memory不够用了。
2021-08-10 16:38:04.917501: E tensorflow/stream_executor/cuda/cuda_blas.cc:226] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2021-08-10 16:38:04.960048: E tensorflow/stream_executor/cuda/cuda_blas.cc:226] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2021-08-10 16:38:04.986898: E tensorflow/stream_executor/cuda/cuda_blas.cc:226] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2021-08-10 16:38:04.992366: E tensorflow/stream_executor/cuda/cuda_blas.cc:226] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2021-08-10 16:38:04.992389: W tensorflow/stream_executor/stream.cc:1455] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):File "/home/mango/PycharmProjects/DeepLearing/minist_conv.py", line 32, in <module>model.fit(train_images, train_labels, epochs=5, batch_size=64)File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/keras/engine/training.py", line 1183, in fittmp_logs = self.train_function(iterator)File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/eager/def_function.py", line 889, in __call__result = self._call(*args, **kwds)File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/eager/def_function.py", line 950, in _callreturn self._stateless_fn(*args, **kwds)File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/eager/function.py", line 3023, in __call__return graph_function._call_flat(File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/eager/function.py", line 1960, in _call_flatreturn self._build_call_outputs(self._inference_function.call(File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/eager/function.py", line 591, in calloutputs = execute.execute(File "/usr/local/lib/python3.9/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_executetensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError: Blas xGEMM launch failed : a.shape=[1,64,576], b.shape=[1,576,64], m=64, n=64, k=576[[node sequential/dense/MatMul (defined at home/mango/PycharmProjects/DeepLearing/minist_conv.py:32) ]] [Op:__inference_train_function_993]Function call stack:
train_function二、开发环境
mango@mango-ubuntu:~$ /usr/local/cuda/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Wed_Jul_14_19:41:19_PDT_2021
Cuda~~ compilation tools, release 11.4, V11.4.100==
Build cuda_11.4.r11.4/compiler.30188945_0mango@mango-ubuntu:~$ tail -n 10 /usr/include/cudnn_version.h
#ifndef CUDNN_VERSION_H_
#define CUDNN_VERSION_H_#define CUDNN_MAJOR 8
#define CUDNN_MINOR 2
#define CUDNN_PATCHLEVEL 2#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)#endif /* CUDNN_VERSION_H */mango@mango-ubuntu:~$ python3 --version
Python 3.9.5mango@mango-ubuntu:~$ nvidia-smi
Tue Aug 10 19:57:58 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 Off | N/A |
| N/A 54C P0 N/A / N/A | 329MiB / 2002MiB | 9% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------++-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1818 G /usr/lib/xorg/Xorg 186MiB |
| 0 N/A N/A 2002 G /usr/bin/gnome-shell 45MiB |
| 0 N/A N/A 3435 G ...AAAAAAAAA= --shared-files 75MiB |
| 0 N/A N/A 6016 G python3 13MiB |
+-----------------------------------------------------------------------------+mango@mango-ubuntu:~$ python3
Python 3.9.5 (default, May 11 2021, 08:20:37)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2021-08-10 18:33:05.917520: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
>>> tf.__version__
'2.5.0'
>>> 三、Tensorflow针对GPU内存的分配策略
By default, TensorFlow maps nearly all of the GPU memory of all GPUs (subject to CUDA_VISIBLE_DEVICES) visible to the process. This is done to more efficiently use the relatively precious GPU memory resources on the devices by reducing memory fragmentation.
默认情况下，为了通过减少内存碎片更有效地利用设备上相对宝贵的GPU内存资源，TensorFlow进程会使用所有可见的GPU。
In some cases it is desirable for the process to only allocate a subset of the available memory, or to only grow the memory usage as is needed by the process. TensorFlow provides two methods to control this.
在某些情况下，进程只分配可用内存的一个子集，或者只根据进程的需要增加内存使用量。TensorFlow提供了两种方法来控制这种情况。
The first option is to turn on memory growth by calling tf.config.experimental.set_memory_growth, which attempts to allocate only as much GPU memory as needed for the runtime allocations: it starts out allocating very little memory, and as the program gets run and more GPU memory is needed, the GPU memory region is extended for the TensorFlow process. Memory is not released since it can lead to memory fragmentation. To turn on memory growth for a specific GPU, use the following code prior to allocating any tensors or executing any ops.
第一种选择是通过调用tf.config.experimental.set_memory_growth来打开内存增长，它尝试只分配运行时所需的GPU内存:它开始分配很少的内存，当程序运行时需要更多的GPU内存时，GPU内存区域会进一步扩展增大。内存不会被释放，因为这会导致内存碎片。为了打开特定GPU的内存增长，在分配任何张量或执行任何操作之前，使用以下代码。
gpus = tf.config.list_physical_devices('GPU')
if gpus:try:# Currently, memory growth needs to be the same across GPUsfor gpu in gpus:tf.config.experimental.set_memory_growth(gpu, True)logical_gpus = tf.config.list_logical_devices('GPU')print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")except RuntimeError as e:# Memory growth must be set before GPUs have been initializedprint(e)
Another way to enable this option is to set the environmental variable TF_FORCE_GPU_ALLOW_GROWTH to true. This configuration is platform specific.
启用该选项的另一种方法是将环境变量TF_FORCE_GPU_ALLOW_GROWTH设置为true。此配置是特定于平台的。
The second method is to configure a virtual GPU device with tf.config.experimental.set_virtual_device_configuration and set a hard limit on the total memory to allocate on the GPU.
This is useful if you want to truly bound the amount of GPU memory available to the TensorFlow process. This is common practice for local development when the GPU is shared with other applications such as a workstation GUI.
第二种方法是使用tf.config.experimental.set_virtual_device_configuration配置虚拟GPU设备,并设置GPU上可分配的总内存的硬限制。
如果你想真正将GPU内存的数量绑定到TensorFlow进程中，这是非常有用的。当GPU与其他应用程序(如工作站GUI)共享时，这是本地开发的常见做法。
gpus = tf.config.list_physical_devices('GPU')
if gpus:# Restrict TensorFlow to only allocate 1GB of memory on the first GPUtry:tf.config.set_logical_device_configuration(gpus[0],[tf.config.LogicalDeviceConfiguration(memory_limit=1024)])logical_gpus = tf.config.list_logical_devices('GPU')print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")except RuntimeError as e:# Virtual devices must be set before GPUs have been initializedprint(e)
四、问题分析验证
通过上边对TensorFlow文档的分析，默认情况下会占用所有的GPU内存，但是TensorFlow提供了两种方式可以灵活的控制内存的分配策略；
我们可以直接设置GPU内存按需动态分配
import tensorflow as tf
physical_gpus = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_gpus[0], True)
通过以下命令可以看到执行过程中GPU内存的占用最高为697M
mango@mango-ubuntu:~$ while true; do nvidia-smi; sleep 0.2; done;
Tue Aug 10 20:30:58 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 Off | N/A |
| N/A 58C P0 N/A / N/A | 1026MiB / 2002MiB | 72% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------++-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1818 G /usr/lib/xorg/Xorg 186MiB |
| 0 N/A N/A 2002 G /usr/bin/gnome-shell 45MiB |
| 0 N/A N/A 3435 G ...AAAAAAAAA= --shared-files 73MiB |
| 0 N/A N/A 6016 G python3 13MiB |
| 0 N/A N/A 13829 C /usr/bin/python3.9 697MiB |
+-----------------------------------------------------------------------------+我们也可以限制最多使用1024M的GPU内存
import tensorflow as tf
physical_gpus = tf.config.list_physical_devices('GPU')
tf.config.set_logical_device_configuration(physical_gpus[0], [tf.config.LogicalDeviceConfiguration(memory_limit=1024)])
同样通过命令可以看到执行过程中GPU内存的占用最高为1455M
mango@mango-ubuntu:~$ while true; do nvidia-smi; sleep 0.2; done;
Tue Aug 10 20:31:24 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 Off | N/A |
| N/A 58C P0 N/A / N/A | 1784MiB / 2002MiB | 74% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------++-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1818 G /usr/lib/xorg/Xorg 186MiB |
| 0 N/A N/A 2002 G /usr/bin/gnome-shell 46MiB |
| 0 N/A N/A 3435 G ...AAAAAAAAA= --shared-files 72MiB |
| 0 N/A N/A 6016 G python3 13MiB |
| 0 N/A N/A 13570 C /usr/bin/python3.9 1455MiB |
+-----------------------------------------------------------------------------+五、GPU分配策略分析
通过四中的测试结果可得
- 默认的分配策略会占用所有的内存，并且执行中不会进行释放，如果训练数据量比较打很容易内存不够用；
- 限制最大使用内存，测试占用内存比设置的大，这个可能跟训练中间使用的模型和操作的复杂程度有关系，需要根据具体的业务场景设置合适的值；但是要注意不能设置大了，否则还是会报错，但是设置小了只是执行的慢一些罢了；
- 设置内存按需分配可能是一个相对比较中庸的方案，感觉可能是一个更好的方案，不知道TensorFlow为什么没有设置为默认值，留作一个问题，后续有新的认知的话再补充；
六、扩展
单GPU模拟多GPU环境
当我们的本地开发环境只有一个GPU，但却需要编写多GPU的程序在工作站上进行训练任务时，TensorFlow为我们提供了一个方便的功能，可以让我们在本地开发环境中建立多个模拟GPU，从而让多GPU的程序调试变得更加方便。以下代码在实体GPU GPU:0 的基础上建立了两个显存均为2GB的虚拟GPU。
gpus = tf.config.list_physical_devices('GPU')
if gpus:# Create 2 virtual GPUs with 1GB memory eachtry:tf.config.set_logical_device_configuration(gpus[0],[tf.config.LogicalDeviceConfiguration(memory_limit=1024),tf.config.LogicalDeviceConfiguration(memory_limit=1024)])logical_gpus = tf.config.list_logical_devices('GPU')print(len(gpus), "Physical GPU,", len(logical_gpus), "Logical GPUs")except RuntimeError as e:# Virtual devices must be set before GPUs have been initializedprint(e)
多GPU的数据并行
使用 tf.distribute.Strategy可以将模型拷贝到每个GPU上，然后将训练数据分批在不同的GPU上执行，达到数据并行。
tf.debugging.set_log_device_placement(True)
gpus = tf.config.list_logical_devices('GPU')
strategy = tf.distribute.MirroredStrategy(gpus)
with strategy.scope():inputs = tf.keras.layers.Input(shape=(1,))predictions = tf.keras.layers.Dense(1)(inputs)model = tf.keras.models.Model(inputs=inputs, outputs=predictions)model.compile(loss='mse',optimizer=tf.keras.optimizers.SGD(learning_rate=0.2))
Tensorflow2对GPU内存的分配策略相关推荐
- 深入理解JVM（4）——对象内存的分配策略
一.Java所承担的自动内存管理主要是针对对象内存的分配和回收. 二.在Java虚拟机的五块内存空间中,程序计数器.Java虚拟机栈.本地方法栈内存的分配和回收都具有确定性,一般在编译阶段就能确定需要 ...
- 内存对齐分配策略(含位域模式)
1:内存对齐定义: 现在使用的计算机中内存空间都是按照字节划分的,从理论上讲似乎对任何类型的变量的访问可以从任何地址开始,但是实际上计算机系统对于基本数据类型在内存 中的存放位置都有限制,要求 ...
- PyTorch可视化工具：GPU内存分配
本文来源 机器之心 编辑:杜伟 想要了解自己的 PyTorch 项目在哪些地方分配 GPU 内存以及为什么用完吗?不妨试试这个可视化工具. 近日,PyTorch 核心开发者和 FAIR 研究者 Z ...
- Memcache内存分配策略
转自:http://tank.blogs.tkiicpp.com/2010/12/14/memcache%e5%86%85%e5%ad%98%e5%88%86%e9%85%8d%e7%ad%96%e7 ...
- 《深入理解Java虚拟机》-----第3章 垃圾收集器与内存分配策略
Java与C++之间有一堵由内存动态分配和垃圾收集技术所围成的"高墙",墙外面的人想进去,墙里面的人却想出来. 3.1 概述 说起垃圾收集(Garbage Collection,G ...
- c++ vector查找_C++ vector内存分配策略浅析
(给CPP开发者加星标,提升C/C++技能) 来源:邱国禄https://blog.csdn.net/qiuguolu1108/article/details/107146184 vector是一个封 ...
- Java之JVM的内存分配策略
周末有小伙伴留言说上周面试时被问到内存分配策略的问题,但回答的不够理想,小伙伴说之前公号里看过这一块的文章的,当时看时很清楚,也知道各个策略是干嘛的,但面试时脑子里清楚,心里很明白,但嘴里就是说不清楚 ...
- AGG第七课 内存分配策略
说明 AGG采用new/delete函数操作堆内存,有时候并不是最佳的选择.另一方面,STL的内存分配策略太繁琐,因此没有采用.在agg_allocator.h文件中描述目前内存分配策略: templ ...
- 《深入理解java虚拟机》笔记2——GC算法与内存分配策略
说起垃圾收集(Garbage Collection, GC),想必大家都不陌生,它是JVM实现里非常重要的一环,JVM成熟的内存动态分配与回收技术使Java(当然还有其他运行在JVM上的语言,如Sca ...
- 深入理解JVM(2)——GC算法与内存分配策略
说起垃圾收集(Garbage Collection, GC),想必大家都不陌生,它是JVM实现里非常重要的一环,JVM成熟的内存动态分配与回收技术使Java(当然还有其他运行在JVM上的语言,如Sca ...
最新文章
- python网络爬虫_Python网络爬虫——爬取视频网站源视频！
- 钱学森在30年前对虚拟现实和元宇宙的思考
- SharePoint 2013 商城
- Mac终端建立替身 并置于桌面或Finder中
- VTK：几何对象之OrientedCylinder
- by mango怎么group_java – MongoDB中计算的group-by字段
- javascript 200列（3）
- 如何理解 scanf(%*s,str)?
- Python之父加入微软，一开口就知道是老“凡学家”了
- Vue2.0组织浏览器返回事件
- fw-cloud-framework项目配置、启动问题
- pku,杨建武:文本挖掘技术
- 干货 | 红队小技巧
- SpringBoot自动装配
- Zookeeper隐藏通道和羊群效应
- 一款捕鱼手游分析 —— 实现无限购买金币
- 某程序员哀叹：做码农真的很悲哀，有的35岁就没人要了
- 破解root密码，简单粗暴—干就完了！！！
- 张飞是一员粗中有细的猛将
- 智慧停车主题汇总 2023-02-20