InternalError: Blas GEMM launch failed : a.shape=(100, 784), b.shape=(784, 10), m=100, n=10...问题解决办法

在学习tensorflow的分布式学习时，在实验代码中遇到以下错误：

InternalError: Blas GEMM launch failed : a.shape=(100, 784), b.shape=(784, 10), m=100, n=10...

查看了几种解决方案：https://blog.csdn.net/Vinsuan1993/article/details/81142855 ，都不work，电脑重启后仍然出错。

最后通过做如下修改，错误解决，然后再将代码改回去，错误未复现。

IMAGE_PIXELS = 28
# x = tf.placeholder(dtype=tf.float32, shape=[None, IMAGE_PIXELS * IMAGE_PIXELS])
x = tf.placeholder(dtype=tf.float32, shape=[None, 784])## hid_w = tf.Variable(tf.truncated_normal([IMAGE_PIXELS * IMAGE_PIXELS, ##HIDDEN_UNITS],stddev=1.0 / IMAGE_PIXELS), name='hid_w')
hid_w = tf.Variable(tf.truncated_normal([784, HIDDEN_UNITS],stddev=1.0 / IMAGE_PIXELS), name='hid_w')

我知道这种办法有点SB，但是病急乱投医下管用了，也不知道为什么。这篇博客主要记录下自己当时碰到这个问题是怎么解决的，避免忘记，大家勿喷。

最后附实验源码：

"""
python single_device_single_gpu_mnist.py --job_name=ps --task_index=0
python single_device_single_gpu_mnist.py --job_name=worker --task_index=0
python single_device_single_gpu_mnist.py --job_name=worker --task_index=1
"""
# encoding:utf-8
import math
import tempfile
import time
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
import osflags = tf.app.flags
flags.DEFINE_string('job_name', None, 'job name: worker or ps')
flags.DEFINE_integer('task_index', None, 'Index of task within the job')
flags.DEFINE_string('gpu_available', "0,1", 'gpu_available')
flags.DEFINE_integer("issync", None, "是否采用分布式的同步模式，1表示同步模式，0表示异步模式")
FLAGS = flags.FLAGSos.environ["CUDA_DEVICES_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = FLAGS.gpu_available##选择服务器：
cluster = tf.train.ClusterSpec({'ps': ["101.6.66.3:22221"],'worker': ["101.6.66.3:22225"]})# 定义超参数
IMAGE_PIXELS = 28
DATA_DIR = "./mnist"
HIDDEN_UNITS=100
TRAIN_STEPS = 10000
BATCH_SIZE=100
LR=0.01## 获取数据集
mnist = input_data.read_data_sets(DATA_DIR, one_hot=True)
##判断终端输入的正误
if FLAGS.job_name not in ["worker","ps"]:raise ValueError("--------FLAGS.job_name input error,must be 'worker' or 'ps'")
if FLAGS.task_index not in range(0,2):raise ValueError("--------FLAGS.task_index input error, must be an int")
## 根据终端输入,选择服务器
server = tf.train.Server(cluster,job_name=FLAGS.job_name,task_index=FLAGS.task_index)if FLAGS.job_name == "ps":server.join()## 训练代码
is_chief = (FLAGS.task_index == 0)
target = server.target
worker_device = "/job:worker/task:{}".format(FLAGS.task_index)
ps_device = "/job:ps/task:0/device:CPU:0"with tf.device(tf.train.replica_device_setter(worker_device=worker_device,ps_device = ps_device,cluster=cluster)):x = tf.placeholder(dtype=tf.float32, shape=[None, IMAGE_PIXELS * IMAGE_PIXELS])y_ = tf.placeholder(dtype=tf.float32, shape=[None, 10])global_step = tf.Variable(0, name='global_step', trainable=False)  # 创建纪录全局训练步数变量## variableshid_w = tf.Variable(tf.truncated_normal([IMAGE_PIXELS * IMAGE_PIXELS, HIDDEN_UNITS],stddev=1.0 / IMAGE_PIXELS), name='hid_w')hid_b = tf.Variable(tf.zeros([100]), name='hid_b')sm_w = tf.Variable(tf.truncated_normal([HIDDEN_UNITS, 10],stddev=1.0 / math.sqrt(HIDDEN_UNITS)), name='sm_w')sm_b = tf.Variable(tf.zeros([10]), name='sm_b')## opshid_lin = tf.nn.xw_plus_b(x, hid_w, hid_b)hid = tf.nn.relu(hid_lin)y = tf.nn.softmax(tf.nn.xw_plus_b(hid, sm_w, sm_b))cross_entropy = -tf.reduce_sum(y_ * tf.log(tf.clip_by_value(y, 1e-10, 1.0)))opt = tf.train.AdamOptimizer(LR)train_step = opt.minimize(cross_entropy, global_step=global_step)##创建sess,执行训练config = tf.ConfigProto()config.gpu_options.allow_growth = Trueconfig.log_device_placement = Truewith tf.Session(target,config=config) as sess:sess.run(tf.global_variables_initializer())print("worker {} : session initialization complete.".format(FLAGS.task_index))batch_xs, batch_ys = mnist.train.next_batch(BATCH_SIZE)train_feed = {x: batch_xs, y_: batch_ys}_, step = sess.run([train_step, global_step], feed_dict=train_feed)print("successfully!")

InternalError: Blas GEMM launch failed : a.shape=(100, 784), b.shape=(784, 10), m=100, n=10...问题解决办法相关推荐

failed to run cuBLAS routine cublasSgemm_v2: CUBLAS_STATUS_EXECUTION_FAILED Blas GEMM launch failed
项目场景: 一个命名实体识别模型,BERT+BiGRU+CRF. 问题描述: 在以前的笔记本上面,显卡2060,python3.7,tensorflow1.13,cuda10.0,cudnn7.4一直 ...
TensorFlow: couldn’t open CUDA library cupti64_80.dll、InternalError: Blas SGEMM launch failed
1. couldn't open CUDA library cupti64_80.dll Win10 TensorFlow(gpu)安装详解在资源管理器中查询 cupti64_80.dll 的位置. ...
Blas SGEMM launch failed
Eclipse launch failed.Binary not found解决方案
配置完成后建立工程测试,发现建立Hello World c++ Project类型的项目后可以运行测试,直接建立空项目写个测试类无法运行,提示"launch failed.Binary no ...
成功解决Ubuntu下的make: gcc: Command not found Makefile:85: recipe for target 'obj/gemm.o' failed make: **
成功解决Ubuntu下的make: gcc: Command not found Makefile:85: recipe for target 'obj/gemm.o' failed make: ** ...
TortoiseSVN Launch Failed Error:系统找不到指定路径
右键操作TortoiseSVN 各种报 Launch Failed Error:系统找不到指定路径刚刚安装完成后出现的问题解决方案:重启电脑
提示 launch failed
2019独角兽企业重金招聘Python工程师标准>>> 错误提示:Could not launch"appName" process launch failed: ...
Eclipse C++的配置问题launch failed binary not found
首先下载eclipse c++ 我的是64bit版本安装好MinGW,并配置好环境变量,参考我的博客 http://www.cnblogs.com/fickleness/p/3273044.html ...
process launch failed: Security
运行系统:iOS9.1 运行工具:xCode7.1.1 问题描述:在真机iPhone6上运行xCode案例时,会提示"Could not launch "iOSCase" ...

InternalError: Blas GEMM launch failed : a.shape=(100, 784), b.shape=(784, 10), m=100, n=10...问题解决办法

InternalError: Blas GEMM launch failed : a.shape=(100, 784), b.shape=(784, 10), m=100, n=10...问题解决办法相关推荐

最新文章

热门文章