Deep learning has a huge data problem.

深度学习有一个巨大的数据问题。

In fact, this was a major contributing factor to the ‘AI Winter’ in the late 90’s when research into neural networks died because no dataset large enough had been assembled to yield promising results.

实际上，这是90年代后期“人工智能冬季”的一个主要促成因素，当时对神经网络的研究因没有足够大的数据集产生令人满意的结果而死亡。

When deep neural networks are presented with an abundance of data, they perform well — shockingly well. The issue is that in practice, such large datasets standard architectures are built for, like ImageNet (which has over fourteen million samples), aren’t as feasible.

当向深度神经网络提供大量数据时，它们的性能会很好-令人震惊。 问题在于，在实践中，像ImageNet(拥有1400万个样本)这样的大型数据集标准体系结构是不可行的。

• The vast amount of data generated on social networking platforms, which rely on machine learning to survive, is unstructured and/or unlabeled. Forbes estimates the amount of unstructured data is growing by 65% per year, and the IDC estimates that 80% of worldwide data will be unstructured by 2025. This data must be used somehow.

  在社交网络平台上生成的大量数据是非结构化和/或未标记的，这些数据依赖于机器学习来生存。 福布斯(Forbes)估计，非结构化数据的数量每年以65％的速度增长，而IDC估计，到2025年，全球80％的数据将是非结构化的。 必须以某种方式使用此数据。

• Reinforcement learning is incredibly promising, but it requires too many simulations. It took 83 hours for a reinforcement learning model to reach the same performance at Atari games as a human could in 15 minutes. OpenAI created a model that could solve a Rubik’s cube, trained using 10,000 years of simulation. How will RL work with real-world training (e.g. self driving cars) or with limited resources?强化学习非常有前途，但是它需要太多的模拟。 强化学习模型花了83个小时才能在Atari游戏中达到与人类15分钟内相同的表现。 OpenAI创建了一个可以解决Rubik立方体的模型，并经过10,000年的模拟训练。 RL如何与现实世界中的训练(例如自动驾驶汽车)或有限的资源一起工作？

• Most common massive datasets like ImageNet or unstructured data like text require human labelling. As a) the amount of data increases, b) tasks become more difficult for humans to label, and c) the subjectivity of human labelling means there is no real ground truth, labels are becoming more expensive. How can models learn with few or even no labels?

  最常见的海量数据集(如ImageNet)或非结构化数据(如文本)需要人工标记。 随着( a )数据量的增加，( b )任务对人类的标记变得更加困难，以及( c )人工标记的主观性意味着没有真实的事实，标记变得越来越昂贵。 模型如何在几乎没有标签的情况下学习？

Obviously, there is a tremendous interest in making neural networks less reliant on vast amounts of expensive, structured, organized data and a little more like humans. How is it so that humans can learn from minimal amounts of unstructured data? For example, take some questions:

显然，人们非常关注使神经网络减少对大量昂贵，结构化，组织化数据的依赖，而对人类的依赖性更高。 人类如何才能从最少的非结构化数据中学习？ 例如，提出一些问题：

• Complete the pattern: 1, 2, 4, 2, 1, 2, …完成模式：1、2、4、2、1、2，...
• I throw a ball up. Will it eventually come down again?我把球扔了。 最终会再次下降吗？
• If you see a red light ahead, should you press the break or the pedal?如果您看到前方有红灯，应该踩一下刹车还是踩踏板？

Humans have no problem answering these questions. On the other hand, to train a deep learning model to perform the same task would require either careful creation of a massive labelled dataset or an intricate setup of a complex reinforcement learning environment.

人类可以毫无疑问地回答这些问题。 另一方面，要训练一个深度学习模型来执行相同的任务，则需要仔细创建大量的标记数据集或复杂的强化学习环境的复杂设置。

Take one example — how do humans learn to drive so quickly?

举个例子-人类如何学会如此快速地驾驶？

Of course, humans aren’t perfect, and this is the rationale behind self-driving cars in the first place. However, humans can safely drive a car after a few one-hour long lessons; deep learning models spend years in simulations alone learning the basics of driving. Why is this the case?

当然，人类并非十全十美，这首先就是无人驾驶汽车背后的原理。 但是，经过几个小时的课程，人类可以安全地驾驶汽车。 深度学习模型仅在模拟中就花费了数年时间，以学习驾驶基础知识。 为什么会这样呢？

Driving a car isn’t just about steering a wheel and pressing buttons — it requires an understand of the world in which the agent is operating in. Humans instinctively know red means stop and green means go, or even where to look for those instructions, since we’ve seen it done hundreds of times from the backseat. We know to generally drive at slow speeds because we know what will happen if we damage the car.

驾驶汽车不仅要操纵方向盘和按下按钮，还需要了解代理商所处的环境。人类本能地知道红色表示停止，绿色表示前进，甚至在哪里寻找这些指示，因为我们已经从后座上看到了数百次。 我们知道通常以低速行驶，因为我们知道如果损坏汽车会发生什么。

Humans know all this instinctually because we have had plenty of experience in this world — both the physical dynamics (acceleration) and social dynamics (don’t be a jerk on the road). As Yann LeCun puts it:

人类本能地了解所有这一切，因为我们在这个世界上拥有丰富的经验-身体动力(加速度)和社会动力(不要在路上混蛋)。 正如Yann LeCun所说：

[Self-supervised learning] is learning to represent the world before learning the task… to fill in the blanks.- Yann LeCun @ AAAI 2020

[自我监督学习]在学习任务之前正在学习代表世界……填补空白。-Yann LeCun @ AAAI 2020

It’s important to remember that this is how humans learn. If we don’t expect children to answer profound questions about a specific subject without learning the language and what words mean beforehand, we shouldn’t expect neural networks to.

重要的是要记住，这就是人类的学习方式。 如果我们不希望孩子们在不学习语言和单词是什么意思的情况下回答有关特定主题的深刻问题，那么我们就不应指望神经网络会这样做。

In fact, self-supervised learning appears in every state-of-the-art language model, like BERT or GPT-3. Models are pretrained on word-masking tasks, where the word replaced by a masked token in a sentence like…

实际上，自我监督学习出现在每个最新的语言模型中，例如BERT或GPT-3。 模型在单词掩盖任务中进行了预训练，其中单词在一个类似……的句子中被掩盖的标记代替

The United States <mask> one of the most powerful countries in the world.<mask>: is [grammar]

美国<mask>是世界上最强大的国家之一。<mask> ：是[语法]

It’s raining <mask> and dogs.<mask>: cats [cultural expression]

<mask>和狗在下雨。<mask> ：猫[文化表现形式]

Bob gave Alice a hug, so Alice felt <mask>. <mask>: happy [understanding actions]

鲍勃拥抱了爱丽丝，让爱丽丝感到<面具>。 <mask> ：高兴[理解动作]

Through this pretraining, the model becomes familiar with the world in which it is operating in. It can model basic grammar syntax, but also gains an understanding of cultural expressions and perhaps even the content of what it is saying (e.g. because of action, effect happened; to elaborate on topic, details; x and y have z relationship; because of past, future will happen; because of future, past happened).

通过这种预训练，该模型可以熟悉其运行所在的世界。它可以对基本语法进行建模，而且还可以理解文化表达方式，甚至可以理解其表达的内容(例如，由于行动，影响发生；详细说明主题，细节； x和y具有z关系；因为过去，将来会发生；因为将来，过去会发生)。

After the model becomes familiar with the dynamics and content of language, it can easily take on other more specific tasks within the realm, like text generation on a specific dataset or text classification. The pretrained model works with transfer learning: a model that already understands the world can be downloaded and fine-tuned.

在模型熟悉语言的动态和内容之后，它可以轻松地承担领域内其他更具体的任务，例如在特定数据集上生成文本或进行文本分类。 预先训练的模型可用于迁移学习：可以下载并微调已经了解世界的模型。

What is most important, however, is that this extremely beneficial masking task is exceptionally easy to set up. It can be created from a document, like all of Wikipedia, where sequences can be cut and words masked at random. This makes it fundamentally self-supervised:

然而，最重要的是，这一极其有益的掩蔽任务非常容易设置。 可以像整个Wikipedia一样从文档中创建该文档，在该文档中可以剪切序列并随机屏蔽单词。 这使得它从根本上可以自我监督：

The act of converting unlabeled data into a supervised task, either for the benefit of another model or as a task by itself.

将未标记的数据转换为受监管的任务的行为，该行为要么是出于另一个模型的利益，要么是作为一项任务本身。

Fundamentally, long strings of text are unstructured, but through clever manipulation we are able to use it without any labels. When self-supervised learning is used to aid another model, in this example masked word prediction to aid other tasks, it is usually through ‘latent variables’.

从根本上说，长字符串是无结构的，但是通过巧妙的操作，我们可以在没有任何标签的情况下使用它。 当使用自我监督学习来辅助另一个模型时，在本例中，通过掩盖单词预测来辅助其他任务，通常是通过“潜在变量”进行的。

Latent variables, which from Latin literally means ‘lie hidden’ variables, are used to refer to information indirectly learned that can be used to assist another task. For instance, in the case of driving a car, your ability to infer that red means stop and green means go is a latent variable.

潜在变量在拉丁语中的字面意思是“隐藏”变量，用于指间接学习的信息，可用于协助其他任务。 例如，在驾驶汽车的情况下，您推断红色表示停车而绿色表示行驶的能力是一个潜在变量。

Another example of self-supervised learning — this time in images — is with the prediction of rotation. For instance, if a dataset has a limited amount of images or a model that has a human sense of what is upright is desired, artificial labels can be created from an unlabeled set of images.

自我监督学习的另一个例子-这次是图像学习-是对旋转的预测。 例如，如果数据集的图像数量有限，或者模型具有人类直立的感觉，那么可以从一组未标记的图像中创建人造标签。

Through this task, the model learns underlying latent variables that would certainly assist a prediction task. In fact, indirectly the model must learn to differentiate between objects — for instance, not all images of animals hanging upside down are actually rotated 180 degrees (for instance, bats).

通过此任务，模型学习了潜在的潜在变量，这些变量肯定会有助于预测任务。 实际上，模型必须间接地学会区分对象-例如，并非所有倒挂的动物图像实际上都旋转了180度(例如，蝙蝠)。

Or, for instance, consider the task of compression. Machines will always be superior at finding methods to store information in efficient manners than humans are. Consider the task of compressing information, in which a model with a bottleneck in the middle (only one node) and must replicate the output: the y-label is the same as the x-label. The model simply must find a method to pass the information through the bottleneck.

或者，例如，考虑压缩任务。 在寻找有效地存储信息的方法上，机器将比人类总是更好。 考虑压缩信息的任务，在该任务中，模型的瓶颈在中间(仅一个节点)，并且必须复制输出： y -label与x -label相同。 该模型仅必须找到一种方法来通过瓶颈传递信息。

Then, after the model is trained, the encoder and decoder parts of the model (before and after the bottleneck node) can be taken apart and used to transmit information. If I were to send large image or text data, I could pass it through the encoder, obtain an intermediate value, and send the value to a friend, who would input it to a decoder.

然后，在训练模型之后，可以将模型的编码器和解码器部分(瓶颈节点之前和之后)拆开并用于传输信息。 如果要发送大图像或文本数据，则可以将其传递通过编码器，获取中间值，然后将该值发送给朋友，后者将其输入到解码器。

This is an example of self-supervised learning not necessarily for the learning of latent variables, but as a task itself.

这是自我监督学习的一个例子，不一定要学习潜变量，而是作为一项任务本身。

Another example of self-supervised learning as an independent problem are Generative Adversarial Networks, which artificially turn an unlabeled image dataset into a supervised task through a system of two models, a generator and a discriminator. These seek to generate fake samples and to identify if an image is from the dataset or generated, respectively.

自我监督学习作为独立问题的另一个示例是生成对抗网络，该网络通过生成器和鉴别器这两个模型的系统，将未标记的图像数据集人为地转换为监督任务。 这些试图生成假样本并分别识别图像是来自数据集还是已生成。

Yann LeCun brings up an analogy of machine intelligence in the form of a cake: self-supervised learning is the actual cake, supervised learning is the filling, and reinforcement learning is the cherry on top.

Yann LeCun以蛋糕的形式提出了机器智能的类比：自我监督学习是实际的蛋糕，监督学习是填充，而强化学习则是最重要的。

The actual cake that makes up the bulk of knowledge in the form of ‘underlying knowledge’ or latent variables is in the form of self-supervised learning. AI cannot get far without using self-supervised learning to establish and explore the world in which it runs tasks in. Supervised learning wraps the cake in a layer of icing, with applications in reinforcement learning.

以“基础知识”或潜在变量的形式构成大量知识的实际蛋糕是以自我监督学习的形式。 如果不使用自我监督学习来建立和探索其在其中运行任务的世界，人工智能就无法走得很远。监督学习将蛋糕锦上添花，并在强化学习中得到应用。

This is the key to understanding where AI is going. As perfectly hand-labelled and organized data is becoming more expensive, the label hungry field of deep learning must treat self-supervised learning at the forefront of its blueprint for a more efficient AI.

这是了解AI走向的关键。 随着完美的手工标记和组织化数据变得越来越昂贵，渴望深度学习的标签领域必须在其蓝图的最前沿对待自我监督学习，以实现更高效的AI。