A common analogy in artificial intelligence(AI) circles is that training data is the new oil for machine learning models. Just like the precious commodity, training data is scarce and hard to get at scale. Supervised learning models reign supreme in today’s machine learning ecosystem. While these type of models are relatively easy to create compare to other alternatives, they have a strong dependency in training data that results prohibited for most organizations. This problem becomes bigger with the scale of the machine learning models. Recently, Uber engineers published a paper proposing a new method called Generative Teaching Networks(GTNs) that create learning algorithms that automatically generate training data.

人工智能(AI)界的一个常见比喻是，训练数据是机器学习模型的新动力。 就像珍贵的商品一样，培训数据稀缺且难以大规模获取。 监督学习模型在当今的机器学习生态系统中占据统治地位。 尽管与其他选择相比，这些类型的模型相对容易创建，但它们在训练数据上有很强的依赖性，这对于大多数组织来说都是禁止的。 随着机器学习模型的规模，这个问题变得更大。 最近， Uber工程师发表了一篇论文，提出了一种新方法，称为生成教学网络(GTN) ，该方法可以创建自动生成训练数据的学习算法。

The idea of generating training data using machine learning is not exactly novel. Techniques such as semi-supervised and omni-supervised learning rely on that principle to operate in data scarce environments. However the data-dependency challenges in machine learning models are growing faster than the potential solutions. Part of these challenges have their roots in some of the biggest misconceptions in modern machine learning.

使用机器学习生成训练数据的想法并不完全新颖。 半监督学习和全监督学习之类的技术都依靠该原理在数据稀缺的环境中运行。 但是，机器学习模型中的数据依赖性挑战比潜在的解决方案增长得更快。 这些挑战中的一部分源于现代机器学习中的一些最大误解。

关于培训数据的误解 (Misconceptions About Training Data)

The traditional approach to train a machine learning model tells us that models should be trained using large datasets and they should leverage the entire dataset during the process. Although well established, that idea seems counterintuitive as it assumes that all records in the training dataset have equal weight which is certainly rare. New approaches such as curriculum learning and active learning have focused on extracting a distribution from the training dataset based on the examples that generate the best version of the models. Some of these techniques have proven quite useful in the emergence of neural architecture search(NAS) techniques.

训练机器学习模型的传统方法告诉我们，应该使用大型数据集来训练模型，并且在此过程中应该利用整个数据集。 尽管已经很好地建立了，但是这个想法似乎违反直觉，因为它假设训练数据集中的所有记录具有相等的权重，这当然是很少见的。 课程学习和主动学习等新方法的重点是，基于生成最佳版本模型的示例，从训练数据集中提取分布。 这些技术中的某些已被证明在神经体系结构搜索(NAS)技术的出现中非常有用。

NAS are becoming one of the most popular trends in modern machine learning. Conceptually, NAS help to discover the best high performing neural network architectures for a given problems by performing evaluations across thousands of models. The evaluations performed by NAS methods require training data and they can result cost prohibited if they use complete training datasets in each iteration. Instead, NAS methods have become extremely proficient evaluating candidate architectures by training a predictor of how well a trained learner would perform, by extrapolating from previously trained architectures.

NAS正在成为现代机器学习中最受欢迎的趋势之一。 从概念上讲，NAS通过对数千个模型进行评估来帮助发现给定问题的最佳高性能神经网络体系结构。 NAS方法执行的评估需要训练数据，如果每次迭代都使用完整的训练数据集，则可能导致成本高昂。 取而代之的是，NAS方法已通过从先前受过训练的体系结构推断出的训练受训学习者表现的预测器，变得非常熟练地评估候选体系结构。

These two ideas: selecting the best examples from a training set and understanding how a neural network learns were the foundation of Uber’s creative method for training machine learning models.

这两个想法是：从训练集中选择最好的例子，并了解神经网络如何学习，这是Uber训练机器学习模型的创新方法的基础。

进入生成式教学网络 (Enter Generative Teaching Networks)

The core principle of Uber’s GTNs is based on a simple and yet radical idea: allowing machine learning to create the training data itself. GTNs leverage generative and meta-learning models while also driving inspiration from techniques such as generative adversarial neural networks(GANs).

Uber GTN的核心原理基于一个简单而又激进的想法：允许机器学习本身创建训练数据。 GTN在利用生成和元学习模型的同时，还从诸如生成对抗神经网络(GAN)之类的技术中获得启发。

The main idea in GTNs is to train a data-generating network such that a learner network trained on data it rapidly produces high accuracy in a target task. GTNs borrow ideas from GANs but there are also marked differences. In GTN models,the two networks cooperate instead of competing. The networks in a GTN model have their interests aligned towards having the improving the performance of the learner based on the produced training data. The generator network in a GTN model regularly produces completely new artificial data that a never-seen-before learner neural network trains on for a small number of learning steps. After that step, the learner network is evaluated on real data and its performance is optimized.

GTN中的主要思想是训练数据生成网络，以使训练有素的学习者网络快速在目标任务中产生高精度。 GTN向GAN借鉴了想法，但也存在明显差异。 在GTN模型中，两个网络合作而不是竞争。 GTN模型中的网络的兴趣在于根据生成的训练数据来提高学习者的表现。 GTN模型中的生成器网络会定期生成全新的人工数据，在很少的学习步骤中，这是从未见过的学习者神经网络所训练的。 在此步骤之后，将根据真实数据评估学习者网络并优化其性能。

Based on the original research paper, the architecture of GTNs can be explained in five simple steps:

基于原始研究论文，可以通过五个简单步骤来解释GTN的体系结构：

1) Noise is fed to the input generator which is used to create new synthetic data.

1)噪声被馈送到用于生成新的合成数据的输入生成器。

2) The learner is trained to perform well on the generated data.

2)对学习者进行训练，使其在生成的数据上表现出色。

3) The trained learner is then evaluated on the real training data in the outer-loop to compute the outer-loop meta-loss.

3)然后在外环中的实际训练数据上评估受训学习者，以计算外环元损失。

4) The gradients of the generator parameters are computed to the meta-loss to update the generator.

4)计算发电机参数的梯度到亚损失，以更新发电机。

5) Both a learned curriculum and weight normalization substantially improve GTN performance.

5)学习过的课程和体重标准化都可以大大改善GTN的表现。

行动中的GTN (GTNs in Action)

Uber evaluated GTNs across different neural network architectures. One of those scenarios was an image classification model trained using the famous MNIST dataset. After a few iterations, new learners trained using GTN were able to learn faster than the same models using real data. In this specific scenarios, the GTN-trained models achieved a remarkable 98.9 accuracy and did that in just 32 SGD steps (~0.5 seconds), seeing each of the 4,096 synthetic images in the curriculum once, which is less than 10 percent of the images in the MNIST training data set.

Uber在不同的神经网络架构中评估了GTN。 这些场景之一是使用著名的MNIST数据集训练的图像分类模型。 经过几次迭代，使用GTN训练的新学习者比使用实际数据的相同模型学习得更快。 在这种特定情况下，经过GTN训练的模型达到了惊人的98.9精度，并且仅用32个SGD步(约0.5秒)就做到了这一点，一次在课程中看到4,096张合成图像中的每张，均不到图像的10％在MNIST训练数据集中。

One of the surprising findings of using GTNs for image classification is that the synthetic dataset seem unrealistic to the human eye(see image below). Even more interesting is the fact that the recognizability of the images improves towards the end of the curriculum. Despite its alien appearance, the synthetic data proven to be effective when training neural networks. Intuitively, we would think that if neural network architectures were functionally more similar to human brains, GTNs’ synthetic data might more resemble real data. However, an alternate (speculative) hypothesis is that the human brain might also be able to rapidly learn an arbitrary skill by being shown unnatural, unrecognizable data. How crazy is that?

使用GTN进行图像分类的令人惊讶的发现之一是合成数据集对人眼来说似乎是不现实的(请参见下图)。 更有趣的是，图像的可识别性在课程结束时会有所改善。 尽管看起来很陌生，但是综合数据在训练神经网络时被证明是有效的。 直觉上，我们认为如果神经网络架构在功能上更类似于人的大脑，则GTN的合成数据可能更类似于真实数据。 但是，另一种(推测性的)假设是，人脑也可能通过显示不自然，无法识别的数据来快速学习任意技能。 那有多疯狂？

GTNs are a novel approach to improve the training of machine learning models using synthetic data. Theoretically, GTNs could have applications beyond traditional supervised learning in areas such as NAS methods. Certainly, applying GTNs in Uber’s massive machine learning infrastructure should yield amazing lessons that will help to improve this technique.

GTN是一种使用合成数据来改进机器学习模型训练的新颖方法。 从理论上讲，GTN可以在诸如NAS方法之类的领域中获得超越传统监督学习的应用。 当然，在Uber的大规模机器学习基础架构中应用GTN应该会产生令人惊讶的教训，这将有助于改进该技术。