On May 28th, 2020 OpenAI announced it’s a new monument to general AI: The GPT-3 system. It is an incredible piece of engineering: a deep neural network with 175 billion parameters. And it does play some incredible parlor tricks:

2020年5月28日，OpenAI宣布它是通用AI的新纪念碑： GPT-3系统。 这是一项令人难以置信的工程：具有1750亿个参数的深度神经网络。 它确实发挥了一些不可思议的客厅技巧：

1. It can translate from English to other languages and back它可以将英语翻译成其他语言并返回
2. Given a prompt, can generate some pretty readable short stories给出提示，可以生成一些易于阅读的短篇小说
3. Given a description of a user interface, it can generate the HTML web page layout给定用户界面的描述，它可以生成HTML网页布局
4. Give a description of a database search, it can generate the program to query a database给出数据库搜索的描述，它可以生成查询数据库的程序
5. Given some text, it can use common sense and knowledge of the real world to answer questions about the text给定一些文本，它可以使用常识和现实世界的知识来回答有关文本的问题
6. It can summarize long documents可以汇总长文件
7. It can emulate a real person in a chat (we are getting closer to passing the Turning Test)它可以在聊天中模拟真实的人(我们正在接近通过车削测试)
8. When asked to write a short paragraph, a real person can guess if it was written by GPT-3 or a real person 52% of the time (50% is a random guess)当要求撰写简短的段落时，真实的人可以猜出它是由GPT-3还是真实的人写的，有52％的时间(50％是随机猜测)
9. It can answer simple math questions它可以回答简单的数学问题

However, it is debatable how “smart” really is. It is very good at many language tasks that involve complex pattern matching, but it falls down on many common-sense tasks. In this blog, we will discuss if these new “generative” systems will be able to generate detailed lessons that can be customized to the need of a classroom of students or an individual student.

但是，究竟“智能”到底是什么，仍有待商bat。 它在涉及复杂模式匹配的许多语言任务中非常擅长，但在许多常识性任务中却有所下降。 在此博客中，我们将讨论这些新的“生成式”系统是否能够生成可根据学生教室或单个学生的需要进行定制的详细课程。

At the heart of GPT systems is the ability to “Generate” text. GPT stands for Generative Pre-trained Transformer. It means that the GPT-3 neural network was built using a Transformer model. The key difference between GPT-3 and its predecessor like GPT-2 is its mammoth size. It was trained on many English language data sources (Wikipedia is only about 6% of the input) that included almost 1 trillion words. The model size of 175B parameters is almost 100-times larger than the 1.5B parameters of GPT-2 which was released in February of 2019. So that is a factor of 100x in 16 months! But generating these models is not easy. It is estimated that just the cost of running the GPUs to crawl the web and train these models cost around $10 million dollars.

GPT系统的核心是“生成”文本的能力。 GPT表示对于G enerative P重新训练的牛逼ransformer。 这意味着GPT-3神经网络是使用Transformer模型构建的。 GPT-3及其前身(如GPT-2 )之间的主要区别在于其庞大的尺寸。 它接受了许多英语数据源的培训(维基百科仅占输入的6％)，其中包含近1万亿个单词。 175B参数的模型大小几乎比2019年2月发布的GPT-2的1.5B参数大100倍。因此，这是16个月内的100倍！ 但是生成这些模型并不容易。 据估计，仅运行GPU来爬行网络和训练这些模型的成本就大约需要1000万美元。

I should also mention that the Transformer models are not unique to OpenAI and GPT models. They are also used by Google in the autocomplete features of Gmail and in automatic grammar checking programs like Grammarly, which I am using to help write this article. Transformer models like BERT and their kin have revolutionized natural language processing (NLP) technologies in the past two years.

我还应该提到，Transformer模型并不是OpenAI和GPT模型所独有的。 Google还在Gmail的自动完成功能和语法自动检查程序(如Grammarly)中使用了它们，我将用来帮助撰写本文。 在过去两年中，像BERT这样的变压器模型及其近亲彻底改变了自然语言处理(NLP)技术。

So how could we use Transformers to generate lesson plans for our classrooms? First, we need to understand how Transformers and GPT systems work. You can see from the list above that a single GPT-3 model does lots of different things. The way we vary its behavior is by giving it some short new prompts in addition to the underlying “base” pre-trained model. Once the prompts are given the network is “primed” to return the right types of answers to questions. It is almost like telling people the “rules” of a new game you want them to play. These systems are called “few shot leaners,” because the programmer only needs to add a very few short examples and the GPT systems then learn how to generalize the intent of the prompts.

那么我们如何使用变形金刚为我们的教室生成课程计划呢？ 首先，我们需要了解Transformers和GPT系统的工作方式。 您可以从上面的列表中看到，单个GPT-3模型执行许多不同的操作。 除了基本的“基础”预训练模型外，我们还通过一些简短的新提示来改变其行为。 给出提示后，网络将被“灌注”以返回正确的问题答案类型。 这几乎就像告诉人们您希望他们玩的新游戏的“规则”。 这些系统被称为“精打细算的人”，因为程序员只需要添加一些简短的示例，然后GPT系统就可以学习如何概括提示的意图。

So to generate new lesson plans we would have to provide a “training set” of sample descriptions of lesson plans and what we would expect the results to be. This dataset of input-output pairs would be our “few-shot learning” examples that we layer above the 175B parameter GPT-3 model. Once this is done we would then send it a short English language description of a lesson plan we desired and GPT-3 would return the lesson plan. Sounds simple, right? We could have AI in every classroom by the end of the year.

因此，要生成新的课程计划，我们将必须提供课程计划样本描述的“训练集”以及我们期望的结果。 输入输出对的数据集将是我们的“几次学习”示例，我们将它们放在175B参数GPT-3模型之上。 完成此操作后，我们将向其发送所需的课程计划的简短英语说明，GPT-3将返回该课程计划。 听起来很简单，对吧？ 到今年年底，我们可以在每个教室使用AI。

Well, hold on minute folks! Generating detailed lesson plans is not quite that simple! Let’s take a look at some of the challenges using GPT-3 and some of the complexities that will come up when we attempt to build this system. We will also describe the strategies we can take to overcome these limitations.

好吧，等一下吧！ 生成详细的课程计划并不是那么简单！ 让我们看一下使用GPT-3所面临的一些挑战以及在尝试构建该系统时将会出现的一些复杂性。 我们还将描述克服这些限制可以采取的策略。

Since the start of the COVID pandemic, I have been helping organizations move their lesson plans online. Let's see how we could build a training set for lesson plan generation using these lesson plans. We have standardized on using GitHub Pages, Markdown, mkdocs, and Google Material widgets to build these pages. You can see examples of our curriculum for Scratch, Python, Arduino, and Web.

自从COVID大流行开始以来，我一直在帮助组织将其课程计划在线转移。 让我们看看如何使用这些课程计划为课程计划生成构建训练集。 我们已标准化使用GitHub Pages，Markdown，mkdocs和Google Material小部件来构建这些页面。 您可以看到有关Scratch ， Python ， Arduino和Web的课程示例。

The first thing to understand is that we don’t really want to generate the raw-lower-level HTML code for these sites. That would be difficult to manage. What we want to do is to generate easy-to-understand and easy-to-maintain Markdown. Here is an example of what that input might look like for learning how to use the SVG Circle element:

首先要了解的是，我们实际上并不希望为这些网站生成原始的较低级HTML代码。 那将很难管理。 我们要做的是生成易于理解和易于维护的Markdown 。 这是一个示例，该示例显示了如何学习如何使用SVG Circle元素的输入：

# Drawing a Circle with SVGIn this lession, we will generate a circle using the SVG language.  We will show you how to position the circle, change the size of the circle and change the fill and border color of the circle.## PrerequisitesBefore you begin, you will need to know how to edit markup and add new attributes to elements.  To test the drawing you will need to render the code in a web browser.## Cicrle Attributescx = x or horizontal position of the center of the circle from the leftcy = y or vertical position of the center of the circle from the topr = radius## Sample Code...## Rendering...## Experiments to try1. What would happen if you change the fill from blue to red?2. How would you change the color of the border of the circle## Resourceshttps://developer.mozilla.org/en-US/docs/Web/SVG/Element/circle

Note you can see the actual source of the SVG Circle Markdown here.

请注意，您可以在此处查看SVG Circle Markdown的实际来源。

In the SVG circle example above the block of text after the first title line is the description “input” preamble. It is the job of the Transformer model to generate the rest of the Markdown file.

在第一个标题行之后的文本块上方的SVG圆圈示例中，描述“输入”前导。 生成其余的Markdown文件是Transformer模型的工作。

How would it do this? It would need to be trained on many example tutorials of how other courses taught Web, HTML, and SVG labs. It would build a neural network with each of the words in these tutorials and what other words followed any word within the context of each tutorial document. What is important to understand is that GPT-3 is kind of already doing some of this today. Here is an example of GPT-3 generating an HTML web page from a written description of that page.

它会怎么做？ 需要在许多示例教程中对它进行培训，这些示例教程说明了其他课程如何教授Web，HTML和SVG实验室。 它将使用这些教程中的每个单词以及每个教程文档的上下文中的每个单词之后的单词构建一个神经网络。 重要的是要了解，GPT-3目前已经在做一些这样的事情。 这是GPT-3从该页面的书面描述生成HTML网页的示例。

One of the challenges here is understanding the implied context of the lesson plan generation. If you are working with junior-high students they may need some additional background information and a slower more gentle introduction to the concepts. College students on the other hand usually will have prior experiences that will accurate their learning. Their lesson plans can make assumptions of their prior knowledge or just provide some links for any necessary background concepts they must master.

这里的挑战之一是了解课程计划生成的隐含上下文。 如果您与初中学生一起工作，他们可能需要一些其他背景信息以及对这些概念的较慢的介绍。 另一方面，大学生通常将具有可以使他们的学习准确的先前经验。 他们的课程计划可以假设他们的先验知识，或者只是提供一些必须掌握的必要背景概念的链接。

So although this is a good example, I have not been able to test this yet since GPT-3 keys are being only given to a few people. By October 2020 we think that there will be commercial versions of the GPT-3 API available that we can start testing. Note that the pricing is based on “tokens generated”. This is roughly equivalent to the number of words in a lesson plan.

因此，尽管这是一个很好的示例，但是由于GPT-3密钥仅提供给少数人，因此我还无法进行测试。 我们认为，到2020年10月，我们将可以开始测试GPT-3 API的商业版本。 请注意，定价基于“生成的代币”。 这大致等于课程计划中的单词数。

I hope I have convinced some of you that using Transform models to generate a lesson plan is not that far off. What we will need are good training sets and some volunteers that want to try this out in their classrooms and mentoring sessions.

我希望我已经说服了一些人，即使用Transform模型生成课程计划不是很遥远。 我们需要的是好的培训教材，还有一些想在教室和辅导课上尝试一下的志愿者。

If you would like to help out building these tools for teachers, mentors, and students, please connect with on LinkedIn.

如果您想帮助教师，导师和学生构建这些工具，请在LinkedIn上与保持联系。

I look forward to hearing from you! — Dan

我期待着您的回音！ 丹