在训练时用softmax计算交叉熵，容易出现浮点下溢，导致log(0)的计算，这就造成了从此次以后的loss都是Nan，解决方法是限制网络输出范围：tf.log(tf.clip_by_value(pred, 1e-5, 1.0))。学习率过大也会造成Nan，一般出现这种情况的话每次学习率除以10地进行调试。

TensorFlow实现训练大致分两种方法。

最低效的是将data pipeline与训练的graph分割成两部分，然后在session中分次执行。代码示意如左，另一种是将data pipeline写进训练的graph中，让TensorFlow自动多线程处理，代码示意如右。

inputs, outputs = data_pipeline(...)X = tf.placeholder(...)
Y = tf.placeholder(...)pred = net(X)loss = loss_func(pred, Y)
train_op = optimizer.minimize(loss)trainX, trainY = sess.run([inputs, outputs])
sess.run(train_op, feed_dict={X:trainX, Y:trainY})

inputs, outputs = data_pipeline(...)pred = net(inputs)loss = loss_func(pred, outputs)
train_op = optimizer.minimize(loss)sess.run(train_op)

然而TensorFlow自动多线程的实现并不是很好，设置batch size 128，iter 1000次测试两种方法，分别耗时665.52s， 654.39s， 基本差别不大。GPU使用率曲线分别如下：

理论上来说，如果把训练数据全部读取到内存，那么只需要在内存与GPU直接通信就行了，但实际上训练集都会非常大，因此最耗时的是在硬盘读取上。所以要获得高效的训练，最好自己实现多线程。在这里我使用Python自带的Queue库和threading库，用4个producer产生数据，一个consumer训练网络，代码如下：

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os
import time
import Queue
import threading
import tensorflow as tf
from dataset.read_data import read_data
from nnets.vgg import vggos.environ['CUDA_VISIBLE_DEVICES'] = '1'class_num = 2def data_pipline(batch_size):data_batch, annotation = read_data(batch_size)iterator = data_batch.make_initializable_iterator()inputs, outputs = iterator.get_next()with tf.Session() as sess:sess.run(iterator.initializer)for _ in xrange(250):data = sess.run([inputs, outputs])message.put(data)message.put(None)def train():inputs = tf.placeholder(tf.float32, shape=[None, 128, 128, 3])outputs = tf.placeholder(tf.float32, shape=[None, class_num])tf.summary.image('inputs', inputs, 16)lr = tf.placeholder(tf.float32)keep_prob = tf.placeholder(tf.float32)pred = vgg(inputs, class_num, keep_prob)with tf.name_scope('cross_entropy'):cross_entropy = tf.reduce_mean(-tf.reduce_sum(outputs * tf.log(tf.clip_by_value(pred, 1e-5, 1.0)), reduction_indices=[1]))tf.summary.scalar('cross_entropy', cross_entropy)with tf.name_scope('accuracy'):correct = tf.equal(tf.argmax(pred, 1), tf.argmax(outputs, 1))accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))    tf.summary.scalar('accuracy', accuracy)with tf.name_scope('optimizer'):optimizer = tf.train.AdamOptimizer(lr).minimize(cross_entropy)merged = tf.summary.merge_all()saver = tf.train.Saver()with tf.Session() as sess:writer = tf.summary.FileWriter('./log/', sess.graph)sess.run(tf.global_variables_initializer())i, stop_count = 0, 0st = time.time()while True:i += 1if stop_count == producer_num:breakmsg = message.get()if msg is None:stop_count += 1continueimage, label = msglearning_rate = 1e-5 if i < 500 else 1e-6sess.run(optimizer, feed_dict={inputs:image, outputs:label, lr:learning_rate, keep_prob:0.5})# if i % 50 == 0:#     summary, acc, l = sess.run([merged, accuracy, cross_entropy], feed_dict={inputs:image, outputs:label ,keep_prob:1.0})#     print 'iter:{}, acc:{}, loss:{}'.format(i, acc, l)            #     writer.add_summary(summary, i)print 'run time: ', time.time() - stsaver.save(sess, './models/vgg.ckpt')  returnif __name__ == '__main__':BATCH_SIZE = 128producer_num = 4message = Queue.Queue(200)for i in xrange(producer_num):producer_name = 'p{}'.format(i)locals()[producer_name] = threading.Thread(target=data_pipline, args=(BATCH_SIZE,))locals()[producer_name].start()c = threading.Thread(target=train)1c.start()message.join()

耗时527.11s，下图为GPU使用率，可以看到基本上是100%。取消76-80行的注释会把中间结果写进tensorboard，但会多耗时一些，在执行这个步骤时GPU使用率也会降到0。

在这里只使用Baoli和Xingkai两种字体来做二分类，下图分别是训练时的accuracy和loss

5.评估

这里使用712字的次常用汉字来作为测试。测试是要注意的是，在训练时使用tf.image.per_image_standardization来将数据集进行标准化，若测试集用归一化，或者不做处理输出网络，那么所有的预测结果都会偏向于同一类。

如下是测试代码，在测试的时候偷了个懒，网络每次只接收一张图片，若要提升代码速度的话可以批量读取图片到内存，然后一起送进网络。

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os
import random
import numpy as np
from scipy.misc import imresize, imrotate
import matplotlib.pyplot as plt
import tensorflow as tf
from dataset.generator import read_text, draw_font
from nnets.vgg import vggdef generator_images(texts, fonts):for text in texts:for font in fonts:image, _ = draw_font(text, font, mode='test')image = np.asarray(image)yield image, textdef run():file_name = u'test.txt'# file_name = u'dataset/中国汉字大全.txt'texts = read_text(file_name)fonts_dir = os.path.join('dataset', 'fonts')fonts = [os.path.join(os.getcwd(), fonts_dir, path) for path in os.listdir(fonts_dir)] images_gen = generator_images(texts, fonts)inputs = tf.placeholder(tf.float32, shape = [None, None, 3])example = tf.cast(tf.image.resize_images(inputs, [128, 128]), tf.uint8) example = tf.image.per_image_standardization(example)example = tf.expand_dims(example, 0)outputs = vgg(example, 2, 1.0)sess = tf.Session()restorer = tf.train.Saver()restorer.restore(sess, 'models/vgg.ckpt')error = 0error_texts = []for index, info in enumerate(images_gen):image, text = infopred = sess.run(outputs, feed_dict={inputs:image})pred = np.squeeze(pred)label = np.squeeze(np.where(pred==np.max(pred)))if index % 2 != label:error_texts.append((text, pred.tolist()))error += 1print 'test num: {}, error num: {}, acc: {}'.format(index + 1, error, 1 - float(error) / index)

输出结果如下：

test num: 1424, error num: 6, acc:0.9957865168539326

因为类别为2，所以测试集大小为712×2=1424

接着将错误的类别可视化：

def show_errors(error_infos, fonts):length = len(error_infos)labels = len(fonts)for i in xrange(length):text, pred = error_infos[i]index = pred.index(max(pred))for j in xrange(labels):axis = plt.subplot(length, labels, i * labels + j + 1)axis.axis('off')font = fonts[j]image, _ = draw_font(text, font, mode='test')if index == j:plt.title(str(pred))plt.imshow(image)plt.show()

可以看到被误分类的字分别是皿、吆、蚣、豺、鹦、豁。具体的误分类情况为：

1. Baoli的皿以0.978的概率被判断为Xingkai
2. Xingkai的吆以0.783的概率被判断为Baoli
3. Baoli的蚣以0.801的概率被判断为Xingkai
4. Xingkai的豺以0.591的概率被判断为Baoli
5. Baoli的鹦以0.827的概率被判断为Xingkai
6. Baoli的豁以0.578的概率被判断为Xingkai

def generator_images(texts, fonts):mean, sigma = 0, 1random_rotate = 30random_scale = 0.2for text in texts:for font in fonts:image, _ = draw_font(text, font, mode='test')image = np.asarray(image)image = imresize(image, random.uniform(1-random_scale, 1+random_scale))image = imrotate(image, random.uniform(-random_rotate, random_rotate))image = image + np.random.normal(mean, sigma, size = image.shape)yield image, text