Assignment #2

在这节作业中，神经网络的输入会是行向量，因为这对于TensorFlow来说是标准操作（有些内置TensorFlow函数假定输入是行向量），这意味着隐藏层的权重矩阵会右乘输入而不是左乘。

1.Tensorflow Softmax

实现一个线性分类器，损失函数定义为： $J(W)=CE(y,softmax(xW))$

其中x是行向量特征，W是模型的权重矩阵，我们将使用TensorFlow的自动微分功能来将该模型与所提供的数据相匹配。

(a)使用TensorFlow实现softmax函数： $softmax(x)_{i}=\frac{e^{x_{i}}}{\sum_{j}^{ }e^{x_{j}}}$

不要使用tf.nn.softmax或相关的内置函数。

def softmax(x):"""Compute the softmax function in tensorflow.You might find the tensorflow functions tf.exp, tf.reduce_max,tf.reduce_sum, tf.expand_dims useful. (Many solutions are possible, so you maynot need to use all of these functions). Recall also that many commontensorflow operations are sugared (e.g. x * y does a tensor multiplicationif x and y are both tensors). Make sure to implement the numerical stabilityfixes as in the previous homework!Args:x:   tf.Tensor with shape (n_samples, n_features). Note feature vectors arerepresented by row-vectors. (For simplicity, no need to handle 1-dinput as in the previous homework)Returns:out: tf.Tensor with shape (n_sample, n_features). You need to construct thistensor in this problem."""# YOUR CODE HEREx -= tf.reduce_max(x, axis=1, keep_dims=True)out = tf.exp(x) / tf.reduce_sum(tf.exp(x), axis=1, keep_dims=True)# END YOUR CODEreturn out

Softmax test 1 passed!
Softmax test 2 passed!
Basic (non-exhaustive) softmax tests pass

(b)使用TensorFlow实现交叉熵损失： $CE(y,\widehat{y})=-\sum_{i=1}^{N_{c}}y_{i}log(\widehat{y_{i}})$

其中 $y\in R^{N_{c}}$ 是one-hot标签向量， $N_{c}$ 是类别数量，这个损失是对一个minibatch的所有样本（行）的求和。

不要使用TensorFlow的内置交叉熵函数。

def cross_entropy_loss(y, yhat):"""Compute the cross entropy loss in tensorflow.The loss should be summed over the current minibatch.y is a one-hot tensor of shape (n_samples, n_classes) and yhat is a tensorof shape (n_samples, n_classes). y should be of dtype tf.int32, and yhat shouldbe of dtype tf.float32.The functions tf.to_float, tf.reduce_sum, and tf.log might prove useful. (Manysolutions are possible, so you may not need to use all of these functions).Note: You are NOT allowed to use the tensorflow built-in cross-entropyfunctions.Args:y:    tf.Tensor with shape (n_samples, n_classes). One-hot encoded.yhat: tf.Tensorwith shape (n_sample, n_classes). Each row encodes aprobability distribution and should sum to 1.Returns:out:  tf.Tensor with shape (1,) (Scalar output). You need to construct thistensor in the problem."""# YOUR CODE HEREout = -tf.reduce_sum(tf.to_float(y) * tf.log(yhat))# END YOUR CODEreturn out

Cross-entropy test 1 passed!
Basic (non-exhaustive) cross-entropy tests pass

(c)简单解释placeholder变量和feed字典的目的，实现add_placeholders和create_feed_dict函数。

提示：配置变量存储在Config类中

class Config(object):"""Holds model hyperparams and data information.The config class is used to store various hyperparameters and datasetinformation parameters. Model objects are passed a Config() object atinstantiation."""n_samples = 1024n_features = 100n_classes = 5batch_size = 64n_epochs = 50lr = 1e-4

TensorFlow使用占位符变量表示计算图中插入数据的位置。这些占位符被模型构建的其余部分用作输入，并将在训练时提供数据。

class SoftmaxModel(Model):"""Implements a Softmax classifier with cross-entropy loss."""def add_placeholders(self):"""Generates placeholder variables to represent the input tensors.These placeholders are used as inputs by the rest of the model buildingand will be fed data during training.Adds following nodes to the computational graphinput_placeholder: Input placeholder tensor of shape(batch_size, n_features), type tf.float32labels_placeholder: Labels placeholder tensor of shape(batch_size, n_classes), type tf.int32Add these placeholders to self as the instance variablesself.input_placeholderself.labels_placeholder"""# YOUR CODE HEREself.input_placeholder = tf.placeholder(shape=(Config.batch_size, Config.n_features), dtype=tf.float32)self.labels_placeholder = tf.placeholder(shape=(Config.batch_size, Config.n_classes), dtype=tf.int32)# END YOUR CODEdef create_feed_dict(self, inputs_batch, labels_batch=None):"""Creates the feed_dict for training the given step.A feed_dict takes the form of:feed_dict = {<placeholder>: <tensor of values to be passed for placeholder>,....}If label_batch is None, then no labels are added to feed_dict.Hint: The keys for the feed_dict should be the placeholdertensors created in add_placeholders.Args:inputs_batch: A batch of input data.labels_batch: A batch of label data.Returns:feed_dict: The feed dictionary mapping from placeholders to values."""# YOUR CODE HEREfeed_dict = {}feed_dict[self.input_placeholder] = inputs_batchif labels_batch is not None:feed_dict[self.labels_placeholder] = labels_batch# END YOUR CODEreturn feed_dict

(d)在add_prediction_op函数中实现softmax分类器的转换，在add_loss_op函数中加上交叉熵损失。

    def add_prediction_op(self):"""Adds the core transformation for this model which transforms a batch of inputdata into a batch of predictions. In this case, the transformation is a linear layer plus asoftmax transformation:y = softmax(Wx + b)Hint: Make sure to create tf.Variable as needed.Hint: For this simple use-case, it's sufficient to initialize both weights Wand biases b with zeros.Args:input_data: A tensor of shape (batch_size, n_features).Returns:pred: A tensor of shape (batch_size, n_classes)"""# YOUR CODE HEREW = tf.Variable(tf.zeros(shape=(Config.n_features, Config.n_classes)))b = tf.Variable(tf.zeros(shape=(Config.batch_size, Config.n_classes)))pred = softmax(tf.matmul(self.input_placeholder, W) + b)# END YOUR CODEreturn preddef add_loss_op(self, pred):"""Adds cross_entropy_loss ops to the computational graph.Hint: Use the cross_entropy_loss function we defined. This should be a veryshort function.Args:pred: A tensor of shape (batch_size, n_classes)Returns:loss: A 0-d tensor (scalar)"""# YOUR CODE HEREloss = cross_entropy_loss(self.labels_placeholder, pred)# END YOUR CODEreturn loss

(e)实现add_training_op函数，解释TensorFlow的自动微分如何消除我们明确定义梯度的需要。

TensorFlow的自动微分意味着我们只需要定义模型的前向传播过程，反向传播过程会自动完成，允许我们不需要明确地定义梯度就可以优化模型。

    def add_training_op(self, loss):"""Sets up the training Ops.Creates an optimizer and applies the gradients to all trainable variables.The Op returned by this function is what must be passed to the`sess.run()` call to cause the model to train. Seehttps://www.tensorflow.org/versions/r0.7/api_docs/python/train.html#Optimizerfor more information.Hint: Use tf.train.GradientDescentOptimizer to get an optimizer object.Calling optimizer.minimize() will return a train_op object.Args:loss: Loss tensor, from cross_entropy_loss.Returns:train_op: The Op for training."""# YOUR CODE HEREtrain_op = tf.train.GradientDescentOptimizer(Config.lr).minimize(loss)# END YOUR CODEreturn train_op

测试函数：

def t_softmax_model():"""Train softmax model for a number of steps."""config = Config()# Generate random data to train the model onnp.random.seed(1234)inputs = np.random.rand(config.n_samples, config.n_features)labels = np.zeros((config.n_samples, config.n_classes), dtype=np.int32)labels[:, 0] = 1# Tell TensorFlow that the model will be built into the default Graph.# (not required but good practice)with tf.Graph().as_default():# Build the model and add the variable initializer Opmodel = SoftmaxModel(config)init = tf.global_variables_initializer()# If you are using an old version of TensorFlow, you may have to use# this initializer instead.# init = tf.initialize_all_variables()# Create a session for running Ops in the Graphwith tf.Session() as sess:# Run the Op to initialize the variables.sess.run(init)# Fit the modellosses = model.fit(sess, inputs, labels)# If Ops are implemented correctly, the average loss should fall close to zero# rapidly.assert losses[-1] < .5print ("Basic (non-exhaustive) classifier tests pass")

结果如下：

Epoch 0: loss = 60.23 (0.029 sec)
Epoch 1: loss = 21.22 (0.014 sec)
Epoch 2: loss = 11.44 (0.014 sec)
Epoch 3: loss = 7.65 (0.014 sec)
Epoch 4: loss = 5.70 (0.014 sec)
Epoch 5: loss = 4.52 (0.013 sec)
Epoch 6: loss = 3.74 (0.013 sec)
Epoch 7: loss = 3.19 (0.014 sec)
Epoch 8: loss = 2.78 (0.014 sec)
Epoch 9: loss = 2.46 (0.014 sec)
Epoch 10: loss = 2.20 (0.014 sec)
Epoch 11: loss = 1.99 (0.013 sec)
Epoch 12: loss = 1.82 (0.015 sec)
Epoch 13: loss = 1.68 (0.016 sec)
Epoch 14: loss = 1.55 (0.015 sec)
Epoch 15: loss = 1.45 (0.017 sec)
Epoch 16: loss = 1.36 (0.021 sec)
Epoch 17: loss = 1.27 (0.077 sec)
Epoch 18: loss = 1.20 (0.086 sec)
Epoch 19: loss = 1.14 (0.063 sec)
Epoch 20: loss = 1.08 (0.075 sec)
Epoch 21: loss = 1.02 (0.043 sec)
Epoch 22: loss = 0.98 (0.045 sec)
Epoch 23: loss = 0.93 (0.036 sec)
Epoch 24: loss = 0.89 (0.053 sec)
Epoch 25: loss = 0.86 (0.049 sec)
Epoch 26: loss = 0.82 (0.035 sec)
Epoch 27: loss = 0.79 (0.019 sec)
Epoch 28: loss = 0.76 (0.014 sec)
Epoch 29: loss = 0.74 (0.014 sec)
Epoch 30: loss = 0.71 (0.014 sec)
Epoch 31: loss = 0.69 (0.014 sec)
Epoch 32: loss = 0.67 (0.014 sec)
Epoch 33: loss = 0.65 (0.014 sec)
Epoch 34: loss = 0.63 (0.014 sec)
Epoch 35: loss = 0.61 (0.019 sec)
Epoch 36: loss = 0.59 (0.030 sec)
Epoch 37: loss = 0.58 (0.052 sec)
Epoch 38: loss = 0.56 (0.060 sec)
Epoch 39: loss = 0.55 (0.032 sec)
Epoch 40: loss = 0.53 (0.017 sec)
Epoch 41: loss = 0.52 (0.016 sec)
Epoch 42: loss = 0.51 (0.046 sec)
Epoch 43: loss = 0.49 (0.037 sec)
Epoch 44: loss = 0.48 (0.031 sec)
Epoch 45: loss = 0.47 (0.022 sec)
Epoch 46: loss = 0.46 (0.015 sec)
Epoch 47: loss = 0.45 (0.014 sec)
Epoch 48: loss = 0.44 (0.014 sec)
Epoch 49: loss = 0.43 (0.014 sec)
Basic (non-exhaustive) classifier tests pass

CS224N刷题——Assignment2.1_TensorflowSoftmax相关推荐

牛年前的一小结——打响本命年的第一枪，继续刷题！
经过一段时间的小尝试,摸索出了一点点头儿吧. 总结一下子. 关于面试的java,像我这个经验层次(1-2year普通厂)的都不会太难.最多超不出力扣中等难度. 多练习链表.树.指针类的比较基础的题目: ...
牛客网里刷题：JS获取输入的数组
有的时候我们刷题会遇到下面这种输入格式,那么用js怎么把它变成数组呢? [1,2,3,4,5] 难道用readline()之后在踢掉首尾的字符吗?这样也太麻烦了! 我发现了一个好用的方法: let l ...
【Leetcode】刷题之路2（python）
哈希映射类题目(简单题小试牛刀啦bhn) 242.有效的字母异位词 349.两个数组的交集 1002.查找常用字符 202.快乐数 383.赎金信 242. 有效的字母异位词用python的Coun ...
【Leetcode】刷题之路1（python）
leetcode 刷题之路1(python) 看到有大佬总结了一些相关题目,想着先刷一类. 1.两数之和 15.三数之和 16.最接近的三数之和 11.盛最多的水 18.四数之和 454.四数相加II ...
力扣(LeetCode)刷题，简单+中等题(第35期)
力扣(LeetCode)定期刷题,每期10道题,业务繁重的同志可以看看我分享的思路,不是最高效解决方案,只求互相提升. 第1题:解码异或后的排列试题要求如下: 回答(C语言): /*** Note: ...
力扣(LeetCode)刷题，简单+中等题(第34期)
目录第1题:整数转罗马数字第2题:电话号码的字母组合第3题:二叉树的所有路径第4题:砖墙第5题:下一个排列第6题:括号生成第7题:删除并获得点数第8题:全排列第9题:颜色分类第10 ...
力扣(LeetCode)刷题，简单+中等题(第33期)
目录第1题:Z 字形变换第2题:删除字符串中的所有相邻重复项第3题:基本计算器 II 第4题:螺旋矩阵第5题:螺旋矩阵 II 第6题:盛最多水的容器第7题:删除有序数组中的重复项 II 第8 ...
力扣(LeetCode)刷题，简单+中等题(第32期)
目录第1题:数组的度第2题:托普利茨矩阵第3题:爱生气的书店老板第4题:翻转图像第5题:有效的数独第6题:无重复字符的最长子串第7题:区域和检索 - 数组不可变第8题:二维区域和检索 ...
力扣(LeetCode)刷题，简单+中等题(第31期)
目录第1题:同构字符串第2题:最后一块石头的重量第3题:最小路径和第4题:键盘行第5题:存在重复元素 II 第6题:两数相加第7题:三个数的最大乘积第8题:等价多米诺骨牌对的数量第9题 ...
力扣(LeetCode)刷题，简单+中等题(第30期)
目录第1题:单词规律第2题:找不同第3题:在排序数组中查找元素的第一个和最后一个位置第4题:使用最小花费爬楼梯第5题:寻找峰值第6题:字符串中的第一个唯一字符第7题:两个数组的交集 II ...

CS224N刷题——Assignment2.1_TensorflowSoftmax

Assignment #2

1.Tensorflow Softmax

CS224N刷题——Assignment2.1_TensorflowSoftmax相关推荐

最新文章

热门文章