deeplearning.ai——字符级语言模型-恐龙岛

数据集包含了所有恐龙的名字，构建一个字符级语言模型来创建新的恐龙名称，算法能够学习不同的名称模式，并随机生成新的名称。

完成这项作业能够学到：

如何存储文本数据以便使用RNN进行处理
如何合成数据，通过在每个时间步采样预测值并将其传递给下一个RNN单元
如何构建一个字符级文本生成循环神经网络
为什么剪裁梯度很重要

1 - Problem Statement

1.1 - Dataset and Preprocessing

运行下面单元来读取恐龙名字的数据集，创建一个唯一字符的列表，计算数据集和词表大小。

data = open('dinos.txt', 'r').read()
data= data.lower()
chars = list(set(data))
data_size, vocab_size = len(data), len(chars)
print('There are %d total characters and %d unique characters in your data.' % (data_size, vocab_size))

There are 19909 total characters and 27 unique characters in your data.

字符包括a-z，加上\n，它在这节作业中的作用类似于<EOS>，代表恐龙名字的结束而不是一行的结束。创建一个python字典来映射每个索引和相应的字符，这会帮助找到softmax层的概率分布输出中的字符对应的索引。

char_to_ix = { ch:i for i,ch in enumerate(sorted(chars)) }
ix_to_char = { i:ch for i,ch in enumerate(sorted(chars)) }
print(ix_to_char)

{0: '\n', 1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z'}

1.2 - Overview of the model

模型有以下结构：

初始化参数
运行优化的循环：前向传播计算损失，反向传播计算损失的梯度，裁剪梯度避免梯度爆炸，使用梯度下降更新策略更新参数
返回学习到的参数

在每个时间步，RNN尝试预测给定之前的字符后下一个字符是什么， $X=(x^{<1>},x^{<2>},...,x^{<T_{x}>})$ 是训练集， $Y=(y^{<1>},y^{<2>},...,y^{<T_{x}>})$ 是测试集，其中， $y^{<t>}=x^{<t+1>}$

2 - Building blocks of the model

构建两个模块：

梯度裁剪：避免梯度爆炸
采样：用来生成字符

2.1 - Clipping the gradients in the optimization loop

实现一个在优化循环内部调用的clip函数，整个循环结构包括前向传播、损失计算、反向传播、参数更新。参数更新之前，需要时要执行梯度裁剪，来确保梯度不会爆炸。

该clip函数接收一个梯度的字典，返回一个裁剪版本的梯度，如果需要的话。裁剪梯度有不同的方法，使用一个简单的element-wise裁剪过程，梯度向量的每一个元素都被裁剪为介于某个范围[-N,N]的值，一般来说，需要提供一个最大值（比如10）。在本例中，如果梯度向量的任何分量大于10，则设为10，如果梯度向量的任何分量小于-10，则设为-10。

练习：实现下列函数来返回裁剪后的梯度。

### GRADED FUNCTION: clipdef clip(gradients, maxValue):'''Clips the gradients' values between minimum and maximum.Arguments:gradients -- a dictionary containing the gradients "dWaa", "dWax", "dWya", "db", "dby"maxValue -- everything above this number is set to this number, and everything less than -maxValue is set to -maxValueReturns: gradients -- a dictionary with the clipped gradients.'''dWaa, dWax, dWya, db, dby = gradients['dWaa'], gradients['dWax'], gradients['dWya'], gradients['db'], gradients['dby']### START CODE HERE #### clip to mitigate exploding gradients, loop over [dWax, dWaa, dWya, db, dby]. (≈2 lines)for gradient in [dWax, dWaa, dWya, db, dby]:np.clip(gradient, -maxValue, maxValue, out=gradient)### END CODE HERE ###gradients = {"dWaa": dWaa, "dWax": dWax, "dWya": dWya, "db": db, "dby": dby}return gradients

2.2 - Sampling

生成新文本的过程：

练习：实现采样函数，需要以下四步：

设置 $x^{<1>}=\vec {0}$ ， $a^{<0>}=\vec{0}$
运行一步前向传播得到 $a^{<1>}$ 和 $\hat{y}^{<1>}$ ，公式为：，其中 $\hat{y}^{<t+1>}$ 是一个softmax概率向量， $\hat{y}_{i}^{<t+1>}$ 代表下标为i的字符是下一个字符的概率值。
采样：根据概率分布选取下一个字符的索引，如果 $\hat{y}_{i}^{<t+1>}=0.16$ ，则选取索引“i”的概率为16%，可以使用np.random.choice：
最后一步是覆盖x，该变量当前存储 $x^{<t>}$ 和 $x^{<t+1>}$ 的值，通过创建一个与选择的预测字符对应的one-hot向量来表示 $x^{<t+1>}$ ，前向传播 $x^{<t+1>}$ ，不断重复直到得到一个\n字符，表示已到达恐龙名字的结尾。

# GRADED FUNCTION: sampledef sample(parameters, char_to_ix, seed):"""Sample a sequence of characters according to a sequence of probability distributions output of the RNNArguments:parameters -- python dictionary containing the parameters Waa, Wax, Wya, by, and b. char_to_ix -- python dictionary mapping each character to an index.seed -- used for grading purposes. Do not worry about it.Returns:indices -- a list of length n containing the indexes of the sampled characters."""# Retrieve parameters and relevant shapes from "parameters" dictionaryWaa, Wax, Wya, by, b = parameters['Waa'], parameters['Wax'], parameters['Wya'], parameters['by'], parameters['b']# Waa:(n_a, n_a)# Wax:(n_a, vocab_size)# Wya:(vocab_size, n_a)# by:(vocab_size, 1)# b:(n_a, 1)vocab_size = by.shape[0]n_a = Waa.shape[1]### START CODE HERE #### Step 1: Create the one-hot vector x for the first character (initializing the sequence generation). (≈1 line)x = np.zeros((vocab_size, 1))# Step 1': Initialize a_prev as zeros (≈1 line)a_prev = np.zeros((n_a, 1))# Create an empty list of indices, this is the list which will contain the list of indexes of the characters to generate (≈1 line)indices = []# Idx is a flag to detect a newline character, we initialize it to -1idx = -1 # Loop over time-steps t. At each time-step, sample a character from a probability distribution and append # its index to "indexes". We'll stop if we reach 50 characters (which should be very unlikely with a well # trained model), which helps debugging and prevents entering an infinite loop. counter = 0newline_character = char_to_ix['\n']while (idx != newline_character and counter != 50):# Step 2: Forward propagate x using the equations (1), (2) and (3)a = np.tanh(np.dot(Wax, x) + np.dot(Waa, a_prev) + b)z = np.dot(Wya, a) + byy = softmax(z)# for grading purposesnp.random.seed(counter+seed) # Step 3: Sample the index of a character within the vocabulary from the probability distribution yidx = np.random.choice(list(range(vocab_size)), p=y.ravel())# Append the index to "indices"indices.append(idx)# Step 4: Overwrite the input character as the one corresponding to the sampled index.x = np.zeros((vocab_size, 1))x[idx] = 1# Update "a_prev" to be "a"a_prev = a# for grading purposesseed += 1counter +=1### END CODE HERE ###if (counter == 50):indices.append(char_to_ix['\n'])return indices

3 - Building the language model

3.1 - Gradient descent

实现一个函数来执行一步随机梯度下降（使用裁剪梯度），每次都要遍历整个训练样本，所以优化算法是随机梯度下降，RNN中通常的优化循环有以下步骤：

前向传播计算损失
通过时间的反向传播计算损失相对于参数的梯度
裁剪梯度，如果必要
使用梯度下降更新参数

练习：实现优化过程（随机梯度下降的一步）

提供以下函数：

# GRADED FUNCTION: optimizedef optimize(X, Y, a_prev, parameters, learning_rate = 0.01):"""Execute one step of the optimization to train the model.Arguments:X -- list of integers, where each integer is a number that maps to a character in the vocabulary.Y -- list of integers, exactly the same as X but shifted one index to the left.a_prev -- previous hidden state.parameters -- python dictionary containing:Wax -- Weight matrix multiplying the input, numpy array of shape (n_a, n_x)Waa -- Weight matrix multiplying the hidden state, numpy array of shape (n_a, n_a)Wya -- Weight matrix relating the hidden-state to the output, numpy array of shape (n_y, n_a)b --  Bias, numpy array of shape (n_a, 1)by -- Bias relating the hidden-state to the output, numpy array of shape (n_y, 1)learning_rate -- learning rate for the model.Returns:loss -- value of the loss function (cross-entropy)gradients -- python dictionary containing:dWax -- Gradients of input-to-hidden weights, of shape (n_a, n_x)dWaa -- Gradients of hidden-to-hidden weights, of shape (n_a, n_a)dWya -- Gradients of hidden-to-output weights, of shape (n_y, n_a)db -- Gradients of bias vector, of shape (n_a, 1)dby -- Gradients of output bias vector, of shape (n_y, 1)a[len(X)-1] -- the last hidden state, of shape (n_a, 1)"""### START CODE HERE #### Forward propagate through time (≈1 line)loss, cache = rnn_forward(X, Y, a_prev, parameters)# Backpropagate through time (≈1 line)gradients, a = rnn_backward(X, Y, parameters, cache)# Clip your gradients between -5 (min) and 5 (max) (≈1 line)gradients = clip(gradients, 5)# Update parameters (≈1 line)parameters = update_parameters(parameters, gradients, learning_rate)### END CODE HERE ###return loss, gradients, a[len(X)-1]

3.2 - Training the model

给定了恐龙名字的数据集，使用数据集的每一行（一个名字）作为一个训练样本，每100步随机梯度下降，抽样10个随机选择的名字，看看算法如何运作的，记得打乱数据集，随机梯度下降就可以随机访问样本。

练习：根据说明实现model()，当examples[index]包含了一个恐龙名字，为了构建样本(X,Y)，可以：

# GRADED FUNCTION: modeldef model(data, ix_to_char, char_to_ix, num_iterations = 35000, n_a = 50, dino_names = 7, vocab_size = 27):"""Trains the model and generates dinosaur names. Arguments:data -- text corpusix_to_char -- dictionary that maps the index to a characterchar_to_ix -- dictionary that maps a character to an indexnum_iterations -- number of iterations to train the model forn_a -- number of hidden neurons in the softmax layerdino_names -- number of dinosaur names you want to sample at each iteration. vocab_size -- number of unique characters found in the text, size of the vocabularyReturns:parameters -- learned parameters"""# Retrieve n_x and n_y from vocab_sizen_x, n_y = vocab_size, vocab_size# Initialize parametersparameters = initialize_parameters(n_a, n_x, n_y)# Initialize loss (this is required because we want to smooth our loss, don't worry about it)loss = get_initial_loss(vocab_size, dino_names)# Build list of all dinosaur names (training examples).with open("dinos.txt") as f:examples = f.readlines()examples = [x.lower().strip() for x in examples]# Shuffle list of all dinosaur namesshuffle(examples)# Initialize the hidden state of your LSTMa_prev = np.zeros((n_a, 1))# Optimization loopfor j in range(num_iterations):### START CODE HERE #### Use the hint above to define one training example (X,Y) (≈ 2 lines)index = j % len(examples)X = [None] + [char_to_ix[ch] for ch in examples[index]]Y = X[1:] + [char_to_ix['\n']]# Perform one optimization step: Forward-prop -> Backward-prop -> Clip -> Update parameters# Choose a learning rate of 0.01curr_loss, gradients, a_prev = optimize(X, Y, a_prev, parameters)### END CODE HERE #### Use a latency trick to keep the loss smooth. It happens here to accelerate the training.loss = smooth(loss, curr_loss)# Every 2000 Iteration, generate "n" characters thanks to sample() to check if the model is learning properlyif j % 2000 == 0:print('Iteration: %d, Loss: %f' % (j, loss) + '\n')# The number of dinosaur names to printseed = 0for name in range(dino_names):# Sample indexes and print themsampled_indexes = sample(parameters, char_to_ix, seed)print_sample(sampled_indexes, ix_to_char)seed += 1  # To get the same result for grading purposed, increment the seed by one. print('\n')return parameters

生成恐龙名字的效果：