介绍 (Introduction)

What if I tell you that you can enable your own device to write for you in your own style in under 100 lines of code?

如果我告诉您，您可以使自己的设备在100行以下的代码中以您自己的风格为您编写，该怎么办？

The idea to make your device write on your behalf is remarkably inspiring. This practice is referred to as Text Generation or Natural Language Generation, which is a subfield of Natural Language Processing (NLP).

使设备代表您写的想法非常鼓舞人心。 这种做法称为“ 文本生成”或“ 自然语言生成” ，它是自然语言处理(NLP)的子字段。

The fundamentals of text generation can be easily broken down into a simple supervised machine learning problem, wherein, there exists certain features (called x) with their corresponding labels (called y), and using these we can create our own prediction function which will then generate our predicted labels (called ŷ or yhat). We then map these predicted labels to the actual labels to determine the cost and optimise it using an optimisation algorithm such as Gradient Descent, RMSprop or even the Adam optimiser.

文本生成的基础可以很容易地分解为一个简单的有监督的机器学习问题，其中，存在某些特征(称为x)及其相应的标签(称为y)，使用这些功能，我们可以创建自己的预测函数，然后该函数生成我们的预测标签(称为ŷ或yhat)。 然后，我们将这些预测的标签映射到实际标签，以确定成本并使用诸如梯度下降，RMSprop甚至是Adam优化器之类的优化算法对其进行优化。

And transitively we have reduced the task of text generation to a simple prediction problem. Hence, through this, we can have a corpus of text within which we may have several sentences. We extract each sentence (say it has n words) and within each of these sentences, we mark the starting n-1 words as the features (called x) and the nth words as the label for it (called y).

在传递上，我们将文本生成的任务简化为一个简单的预测问题。 因此，通过这种方式，我们可以拥有一个文本语料库，其中可以包含多个句子。 我们提取每个句子(假设它有n个单词)，并且在每个句子中，我们将前n-1个单词标记为特征(称为x)，将第n个单词标记为其特征(称为y)。

Now, let’s say we have the sentence, “Deep learning has automated our world,” here, the phrase “Deep learning has automated our” can be the feature (called x) and the last word “world” can be the label (called y). So in the future, whenever the device encounters the text “Deep learning has automated our” it will know to predict “world” as the next word.

现在，假设我们有一个句子“深度学习使我们的世界自动化”，在这里，短语“深度学习使我们的世界自动化”可以作为特征(称为x)，最后一个词“世界”可以作为标签(称为y)。 因此，将来，只要设备遇到文本“深度学习使我们自动化”，它就会知道预测“世界”作为下一个单词。

Resultantly, if we train a network with a fair amount of data, we get a fairly sophisticated model that can predict the next words and voilà, we have taught our device to write.

作为结果，如果我们培养的网络与数据相当数量的，我们得到的是可以预测接下来的话就万事大吉了一个相当复杂的模型，我们已经教我们的设备编写。

使用TensorFlow生成文本 (Text generation using TensorFlow)

We will now learn how to apply the practical method for generating new texts. For this, we will use TensorFlow 2.0 which is an open source machine learning library.

现在，我们将学习如何将实际方法应用于生成新文本。 为此，我们将使用TensorFlow 2.0这是一个开放源代码的机器学习库。

The steps that need to be followed for this are:

为此，需要遵循的步骤是：

• Data pre-processing:

  数据预处理：

Let’s take two sample sentences:

我们来看两个示例句子：

AI is the new electricity

人工智能是新的力量

AI is my power

人工智能是我的力量

(Note that the sample taken here is infinitesimally small compared to the actual data that machine learning practitioners use to train the model. Usually, an entire corpus of a writers publication is used or an entire book is used as the dataset, however here, the author has only taken a small fraction of it for easy understanding.)

( 请注意，与机器学习从业人员用来训练模型的实际数据相比，此处获取的样本无限小。通常，使用作家出版物的整个语料库或整个书籍作为数据集，但是在这里，作者仅将其中的一小部分用于易于理解。)

For simplicity and to reduce the large number of words in our collection, we can convert each word in our sentence to lowercase as this will not alter the meaning of the sentence (New, new and NEW all mean the same). We can do this by using .lower() in Python 3.

为简单起见并减少集合中大量单词，我们可以将句子中的每个单词都转换为小写，因为这不会改变句子的含义(New，new和NEW的含义相同)。 我们可以通过在Python 3中使用.lower()来实现。

So we have the lowercase version of our sentences as:

因此，我们将句子的小写形式为：

ai is the new electricity

艾是新的电

ai is my power

艾是我的力量

We now have a sample to train our model on. The next thing to do is to generate a unique token for each of the words. Repeating words in the corpus are assigned the same token. This can be easily done using the Tokenizer in TensorFlow.

现在，我们有一个样本来训练我们的模型。 接下来要做的是为每个单词生成一个唯一的令牌。 语料库中的重复单词被分配相同的标记。 使用TensorFlow中的Tokenizer可以轻松完成此操作。

For the first sentence:

对于第一句话：

ai -> 1

ai-> 1

is -> 2

是-> 2

the -> 3

-> 3

new -> 4

新增-> 4

electricity -> 5

电力-> 5

For the second sentence:

对于第二句话：

ai -> 1

ai-> 1

is -> 2

是-> 2

my -> 6

我的-> 6

power -> 7

功率-> 7

Now, we may represent our sentences as a list of numbers:

现在，我们可以将句子表示为数字列表：

[1, 2, 3, 4, 5] -> For the first sentence

[1,2,3,4,5] ->对于第一句话

[1, 2, 6, 7] -> For the second sentence

[1、2、6、7] -> 第二句话

We now generate n-gram sequences, wherein, the first two words of such sentences can be one sequence, and, the first three words can be the next sequence and so on. Therefore, giving our list the following possible sequences:

现在，我们生成n元语法序列，其中，此类句子的前两个单词可以是一个序列，而前三个单词可以是下一个序列，依此类推。 因此，给我们的列表以下可能的顺序：

For the first sentence [1, 2, 3, 4, 5]

对于第一句话[1、2、3、4、5]

[1, 2]

[1，2]

[1, 2, 3]

[1,2,3]

[1, 2, 3, 4]

[1、2、3、4]

[1, 2, 3, 4, 5]

[1、2、3、4、5]

For the second sentence [1, 2, 6, 7]

第二句[1、2、6、7]

[1, 2]

[1，2]

[1, 2, 6]

[1、2、6]

[1, 2, 6, 7]

[1、2、6、7]

We now append each of these generated sequences to another list (forming a 2-D list).

现在，我们将这些生成的序列中的每个序列附加到另一个列表(形成2-D列表)。

One key thing to note here is that the length of each of these generated sequences are different, so to maintain order, we pre-pad with zeros, with each sequence length being equal to the longest sequence available. Since in the present case, the maximum length of the sequence is 5, we pre-pad accordingly. Padding can be done using the pad_sequences() method from Keras.

这里要注意的一件事是每个生成的序列的长度都不同，因此为了保持顺序，我们用零预填充，每个序列的长度等于可用的最长序列。 由于在当前情况下，序列的最大长度为5，因此我们预先填充。 可以使用Keras的pad_sequences()方法进行填充。

After padding each sequence will look like:

填充后，每个序列如下所示：

For the first sentence [1, 2, 3, 4, 5]

对于第一句话[1、2、3、4、5]

[0, 0, 0, 1, 2]

[0，0，0，1，2]

[0, 0, 1, 2, 3]

[0，0，1，2，3]

[0, 1, 2, 3, 4]

[0，1，2，3，4]

[1, 2, 3, 4, 5]

[1、2、3、4、5]

For the second sentence [1, 2, 6, 7]

第二句[1、2、6、7]

[0, 0, 0, 1, 2]

[0，0，0，1，2]

[0, 0, 1, 2, 6]

[0，0，1，2，6]

[0, 1, 2, 6, 7]

[0，1，2，6，7]

We now know that for each of these sequences the first 4 words are the features (called x) and the last word is the label (called y). We then extract our features and their corresponding labels by doing simple NumPy array slicing.

现在我们知道，对于这些序列中的每一个，前四个单词是特征(称为x)，最后一个单词是标签(称为y)。 然后，我们通过简单的NumPy数组切片来提取特征及其对应的标签。

One thing left to do is to convert our labels into one-hot encoded vectors for easy optimisation.

剩下要做的一件事就是将我们的标签转换为一键编码的矢量，以便于优化。

Say the feature is x = [0, 1, 2, 3, 4]

假设特征为x = [0，1，2，3，4]

The corresponding label is Label = [5]

相应的标签为Label = [5]

So the one-hot encoded vector of the Label is: [0, 0, 0, 0, 1] i.e. there is 1 present in the 5th position of the vector and 0 in the remaining position.

因此，Label的一键编码矢量为：[0，0，0，0，1]，即在矢量的第5位有1个，其余位置有0个。

Hence, y = [0, 0, 0, 0, 1]

因此， y = [0，0，0，0，1]

We are now ready with our training data which can be fed to the model.

现在，我们可以将训练数据输入模型中。

• Constructing the model: Single Layer LSTM Model

  构造模型：单层LSTM模型

We define a sequential model wherein each layer has exactly one input tensor and one output tensor.

我们定义一个顺序模型，其中每一层都具有一个输入张量和一个输出张量。

1. The first layer is the Embedding Layer which would be the first layer in the network. This layer takes three arguments namely, the input dimension (the total number of unique words in our corpus); the output dimension (or the embedding dimension in which vectors will be mapped); and the input length (the length of each sequence that is fed).

   第一层是嵌入层 ，它将是网络中的第一层。 该层采用三个参数，即输入维(语料库中唯一词的总数)； 输出尺寸(或矢量将映射到的嵌入尺寸)； 和输入长度(输入的每个序列的长度)。

2. For the second layer, we add a Bidirectional LSTM layer of 20 units which is efficient for taking care of the vanishing gradient problem in long sentences and dependencies.

   对于第二层，我们添加了一个20单位的双向LSTM层，该层可以有效地解决长句和依赖项中的消失梯度问题。

3. And consequently, in the third layer, we add a Dense Layer with p units. Where p is the total number of unique words in our corpus and we apply softmax activation to each of these units.

   因此，在第三层中，我们添加了具有p个单位的密集层 。 其中p是语料库中唯一词的总数，我们将softmax激活应用于这些单元中的每个单元。

• Compiling the model:

  编译模型：

Since the prediction of next words is a multi-class classification problem, we choose our loss function as Categorical Cross-entropy and the optimiser as Adam. By choosing this optimiser, we allow TensorFlow to adjust the learning rate on its own, which removes one hyper-parameter for us to fine tune. We then train the model for let’s say 500 epochs.

由于下一个单词的预测是一个多类分类问题，因此我们选择损失函数作为分类交叉熵，并选择优化器作为亚当。 通过选择此优化器，我们允许TensorFlow自行调整学习率，从而删除了一个超参数供我们进行微调。 然后，我们训练模型约500个纪元。

After fitting the training data, our model showed training accuracy close to 95% and all that is left for us to do is use this model to make predictions.

拟合训练数据后，我们的模型显示出训练准确率接近95％，剩下要做的就是使用此模型进行预测。

• Predicting the next words:

  预测下一个单词：

Now, in order to make predictions, we have to give our model some perception for it to know in what sense we want it to generate the texts. For this, we give an initial phrase or sentence to it, which the model then parses and makes computational predictions based upon our corpus of text that it was trained on. Accordingly, the model will predict the token for the next word and then convert that token back to the original word and finally, display the entire predicted string.

现在，为了做出预测，我们必须给我们的模型一些认识，让它知道我们希望它在什么意义上生成文本。 为此，我们给它一个初始的短语或句子，然后模型根据我们接受过训练的文本语料库解析并做出计算预测。 因此，模型将预测下一个单词的令牌，然后将该令牌转换回原始单词，最后显示整个预测的字符串。

Now to understand the whole process via an example, let’s train the model with the following sentence:

现在，通过一个示例来了解整个过程，让我们用以下语句训练模型：

“The scope of deep learning has been increasing at an exponential rate, the reason deep learning has bloomed is hidden in the fact that there exists a vast number of applications in todays world that we take for granted, from using Hey Siri on our iPhone (trigger word detection) to using automatic replies on our Gmail/LinkedIn (sentiment analysis); deep learning has automated our world without us even realising. The world needs deep learning to sustain as it has become necessary.”

“深度学习的范围正在以指数级的速度增长，深度学习之所以蓬勃发展，是因为当今世界上存在着大量应用程序，这些应用程序被我们视为理所当然，这是因为我们在iPhone上使用了Hey Siri(触发单词检测)以对我们的Gmail / LinkedIn使用自动回复(情感分析)； 深度学习使我们的世界自动化了，而我们甚至没有意识到。 世界需要深度学习以维持其必要性。”

Here, the seed to generate the new text is:

在这里，生成新文本的种子是：

Input = scope of artificial intelligence

输入=人工智能的范围

Whereupon, the model predicted:

因此，该模型预测：

Output = scope of artificial intelligence learning has automated our world

产出=人工智能学习的范围已使我们的世界自动化

Now it may as well happen that the predicted sentence may not be syntactically valid but the overall gist of the sentence can be easily understood, thanks to our model.

现在，由于我们的模型，预测的句子可能在语法上无效，但是很容易理解句子的整体要点，这很可能会发生。

总结一下 (To sum up)

One may think that the predictions made by the model looks somewhat trivial, but it has to be kept in mind that prior to Text Generation our device was incapable of forming novel sentences, in-fact it was unaware of even the basic alphabets needed to construct words.

可能有人认为该模型所做的预测看起来有些琐碎，但必须记住，在“文本生成”之前，我们的设备无法形成新颖的句子，实际上，它甚至不知道构建所需的基本字母话。

From making our device understand the sentiment in the text to predicting new words for a given sequence, NLP has come a long way and we can go even further with it.

从使我们的设备了解文本中的情感到为给定序列预测新单词，NLP取得了长足的进步，我们可以做得更好。

Here is the link to the author’s Github repository which can be referred for the unabridged code.

这是指向作者的Github存储库的链接，该链接可用于未删节的代码。