看你设置的epoch和服务器的性能，你这个应该是每一步大概2s，自己可以计算下大概耗时

| |

m13021933043

邮箱：m13021933043@163.com

|

Signature is customized by Netease Mail Master

在2020年07月09日 09:50，zhouyang-bigdata 写道：

一番折腾，后腾讯云换了个系统镜像，这应该是在gpu上运行了。请问这个训练一般耗时多久？

日志如下：

INFO:tensorflow: name = bert/encoder/layer_11/attention/output/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT

INFO:tensorflow: name = bert/encoder/layer_11/attention/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT

INFO:tensorflow: name = bert/encoder/layer_11/attention/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT

INFO:tensorflow: name = bert/encoder/layer_11/attention/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT

INFO:tensorflow: name = bert/encoder/layer_11/intermediate/dense/kernel:0, shape = (768, 3072), INIT_FROM_CKPT

INFO:tensorflow: name = bert/encoder/layer_11/intermediate/dense/bias:0, shape = (3072,), INIT_FROM_CKPT

INFO:tensorflow: name = bert/encoder/layer_11/output/dense/kernel:0, shape = (3072, 768), INIT_FROM_CKPT

INFO:tensorflow: name = bert/encoder/layer_11/output/dense/bias:0, shape = (768,), INIT_FROM_CKPT

INFO:tensorflow: name = bert/encoder/layer_11/output/LayerNorm/beta:0, shape = (768,), INIT_FROM_CKPT

INFO:tensorflow: name = bert/encoder/layer_11/output/LayerNorm/gamma:0, shape = (768,), INIT_FROM_CKPT

INFO:tensorflow: name = bert/pooler/dense/kernel:0, shape = (768, 768), INIT_FROM_CKPT

INFO:tensorflow: name = bert/pooler/dense/bias:0, shape = (768,), INIT_FROM_CKPT

INFO:tensorflow: name = bidirectional_rnn/fw/basic_lstm_cell/kernel:0, shape = (968, 800)

INFO:tensorflow: name = bidirectional_rnn/fw/basic_lstm_cell/bias:0, shape = (800,)

INFO:tensorflow: name = bidirectional_rnn/bw/basic_lstm_cell/kernel:0, shape = (968, 800)

INFO:tensorflow: name = bidirectional_rnn/bw/basic_lstm_cell/bias:0, shape = (800,)

INFO:tensorflow: name = u_omega:0, shape = (1168,)

INFO:tensorflow: name = output_weights:0, shape = (20, 1168)

INFO:tensorflow: name = output_bias:0, shape = (20,)

WARNING:tensorflow:From /usr/local/lib/python3.6/site-packages/tensorflow/python/training/learning_rate_decay_v2.py:321: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.

Instructions for updating:

Deprecated in favor of operator or tf.math.divide.

INFO:tensorflow:Done calling model_fn.

INFO:tensorflow:Create CheckpointSaverHook.

INFO:tensorflow:Graph was finalized.

2020-07-09 09:42:55.471208: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA

2020-07-09 09:42:55.649475: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2020-07-09 09:42:55.650252: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x8a58eb0 executing computations on platform CUDA. Devices:

2020-07-09 09:42:55.650293: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla V100-SXM2-32GB, Compute Capability 7.0

2020-07-09 09:42:55.663763: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2500000000 Hz

2020-07-09 09:42:55.664574: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x9a2a320 executing computations on platform Host. Devices:

2020-07-09 09:42:55.664608: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,

2020-07-09 09:42:55.665581: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:

name: Tesla V100-SXM2-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53

pciBusID: 0000:00:08.0

totalMemory: 31.72GiB freeMemory: 31.31GiB

2020-07-09 09:42:55.665603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0

2020-07-09 09:42:55.666469: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:

2020-07-09 09:42:55.666486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0

2020-07-09 09:42:55.666493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N

2020-07-09 09:42:55.666986: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30459 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-32GB, pci bus id: 0000:00:08.0, compute capability: 7.0)

INFO:tensorflow:Running local_init_op.

INFO:tensorflow:Done running local_init_op.

INFO:tensorflow:Saving checkpoints for 0 into ckpt/divorce/model.ckpt.

2020-07-09 09:43:22.977919: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally

INFO:tensorflow:global_step/sec: 2.03341

INFO:tensorflow:examples/sec: 65.0693

INFO:tensorflow:global_step/sec: 2.27393

INFO:tensorflow:examples/sec: 72.7656

INFO:tensorflow:global_step/sec: 2.27549

INFO:tensorflow:examples/sec: 72.8157

INFO:tensorflow:global_step/sec: 2.2709

INFO:tensorflow:examples/sec: 72.6686

INFO:tensorflow:global_step/sec: 2.27153

INFO:tensorflow:examples/sec: 72.6891

—

You are receiving this because you commented.

Reply to this email directly, view it on GitHub, or unsubscribe.