在本文中，我们将看看：

• 在高级机器学习(ML)应用程序中使用自定义损失函数
• 定义自定义损失函数并集成到基本Tensorflow神经网络模型
• 一个简单的知识蒸馏学习的例子

介绍

机器学习中预定义的损失函数，可为您尝试优化的问题提供合适的损失值。我们常见的有用于分类的交叉熵损失和用于回归问题的均方误差(MSE)或均方根误差(RMSE)。流行的机器学习(ML)包包括前端(如Keras)和后端(如Tensorflow)，其中包括一组用于大多数分类和回归任务的基本损失函数。但是，极个别情况下您可能需要解决某个问题的自定义损失函数，这些函数仅受有效张量运算的约束。

在Keras中，您可以在技术上创建自己的损失函数，但是损失函数的形式仅限于some_loss(y_true，y_pred)。如果您尝试以some_loss_1(y_true，y_pred，** kwargs)的形式向损失添加其他参数，Keras将抛出运行时异常。有很多方法可以解决这个问题，但总的来说我们需要一种 可扩展的方法来编写一个接受我们传递给它的任何有效参数的损失函数，并以标准和预期的方式在我们的张量上运行。我们将看到如何直接使用Tensorflow从头开始编写神经网络并构建自定义损失函数来训练它。

Tensorflow

Tensorflow(TF)是一种符号和数值计算引擎，它允许我们将张量一起串联到计算图中并对它们进行反向传播。Keras是在Tensorflow之上运行的API或前端，它可以方便地打包使用Tensorflow构建的标准架构(例如各种预定义的神经网络层)，并抽象出TF的许多低级机制。然而，在使这些架构的过程中，粒度级别的控制和执行非常具体的事情的能力就丧失了。

为了简单起见，张量是多维数组，其形状类似元组(feature_dim, n_features)

例子是能够定义接受任意数量参数的自定义损失函数，并且能够使用网络内部的任意张量和网络外部的输入张量计算损失。严格地说，TF中的损失函数甚至不需要是python函数，而只需是TF张量对象上操作的有效组合。前一点很重要，因为自定义损失来自于计算任意张量上的损失，而不仅仅是严格意义上的监督目标张量和网络输出张量(y_true, y_pred)的形式。

在我们得到自定义损失之前，让我们简要回顾一下基本的2层dense网络(MLP)，看看它是如何在TF中定义和训练的。虽然有预定义的TF层，但我们从头开始定义权重和偏差。Python代码如下：

# A simple Tensorflow 2 layer dense network exampleimport tensorflow as tfimport numpy as npfrom sklearn import datasetsfrom sklearn.preprocessing import MinMaxScalerfrom sklearn.decomposition import PCAfrom sklearn.preprocessing import LabelBinarizerimport matplotlib.pyplot as pltfrom mpl_toolkits.mplot3d import Axes3D# load the sklearn breast cancer datasetbc = datasets.load_breast_cancer()X = bc.data[:, :]Y = bc.target# min max scale and binarize the target labelsscaler = MinMaxScaler()X = scaler.fit_transform(X,Y)label = LabelBinarizer()Y = label.fit_transform(Y)# train fractionfrac = 0.9# shuffle datasetidx = np.random.randint(X.shape[0], size=len(X))X = X[idx]Y = Y[idx]train_stop = int(len(X) * frac)X_ = X[:train_stop]Y_ = Y[:train_stop]X_t = X[train_stop:]Y_t = Y[train_stop:]# plot the first 3 PCA dimensions of the sampled datafig = plt.figure(1, figsize=(8, 6))ax = Axes3D(fig, elev=-150, azim=110)X_reduced = PCA(n_components=3).fit_transform(X_)ax.scatter(X_reduced[:, 0], X_reduced[:, 1], X_reduced[:, 2], c=Y_.ravel(), cmap=plt.cm.Set1, edgecolor='k', s=40)ax.set_title("First three PCA directions")ax.set_xlabel("1st eigenvector")ax.w_xaxis.set_ticklabels([])ax.set_ylabel("2nd eigenvector")ax.w_yaxis.set_ticklabels([])ax.set_zlabel("3rd eigenvector")ax.w_zaxis.set_ticklabels([])plt.show()# create the TF neural net# some hyperparamstraining_epochs = 200n_neurons_in_h1 = 10n_neurons_in_h2 = 10learning_rate = 0.1n_features = len(X[0])labels_dim = 1############################################## basic 2 layer dense net (MLP) example adapted from# https://becominghuman.ai/creating-your-own-neural-network-using-tensorflow-fa8ca7cc4d0e# these placeholders serve as our input tensorsx = tf.placeholder(tf.float32, [None, n_features], name='input')y = tf.placeholder(tf.float32, [None, labels_dim], name='labels')# TF Variables are our neural net parameter tensors, we initialize them to random (gaussian) values in# Layer1. Variables are allowed to be persistent across training epochs and updatable bt TF operationsW1 = tf.Variable(tf.truncated_normal([n_features, n_neurons_in_h1], mean=0, stddev=1 / np.sqrt(n_features)), name='weights1')b1 = tf.Variable(tf.truncated_normal([n_neurons_in_h1], mean=0, stddev=1 / np.sqrt(n_features)), name='biases1')# note the output tensor of the 1st layer is the activation applied to a# linear transform of the layer 1 parameter tensors# the matmul operation calculates the dot product between the tensorsy1 = tf.sigmoid((tf.matmul(x, W1) + b1), name='activationLayer1')# network parameters(weights and biases) are set and initialized (Layer2)W2 = tf.Variable(tf.random_normal([n_neurons_in_h1, n_neurons_in_h2], mean=0, stddev=1), name='weights2')b2 = tf.Variable(tf.random_normal([n_neurons_in_h2], mean=0, stddev=1), name='biases2')# activation function(sigmoid)y2 = tf.sigmoid((tf.matmul(y1, W2) + b2), name='activationLayer2')# output layer weights and biasesWo = tf.Variable(tf.random_normal([n_neurons_in_h2, labels_dim], mean=0, stddev=1 ), name='weightsOut')bo = tf.Variable(tf.random_normal([labels_dim], mean=0, stddev=1), name='biasesOut')# the sigmoid (binary softmax) activation is absorbed into TF's sigmoid_cross_entropy_with_logits losslogits = (tf.matmul(y2, Wo) + bo)loss = tf.nn.sigmoid_cross_entropy_with_logits(labels = y, logits = logits)# tap a separate output that applies softmax activation to the output layer# for training accuracy readouta = tf.nn.sigmoid(logits, name='activationOutputLayer')# optimizer used to compute gradient of loss and apply the parameter updates.# the train_step object returned is ran by a TF Session to train the nettrain_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)# prediction accuracy# compare predicted value from network with the expected value/targetcorrect_prediction = tf.equal(tf.round(a), y)# accuracy determinationaccuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32), name="Accuracy")############################################## ***NOTE global_variables_initializer() must be called before creating a tf.Session()!***init_op = tf.global_variables_initializer()# create a session for training and feedforward (prediction). Sessions are TF's way to run# feed data to placeholders and variables, obtain outputs and update neural net parameterswith tf.Session() as sess: # ***initialization of all variables... NOTE this must be done before running any further sessions!*** sess.run(init_op) # training loop over the number of epochs batch_size = 50 batches = int(len(X_) / batch_size) for epoch in range(training_epochs): losses = 0 accs = 0 for j in range(batches): idx = np.random.randint(X_.shape[0], size=batch_size) X_b = X_[idx] Y_b = Y_[idx] # train the network, note the dictionary of inputs and labels sess.run(train_step, feed_dict={x: X_b, y: Y_b}) # feedforwad the same data and labels, but grab the accuracy and loss as outputs acc, l, soft_max_a = sess.run([accuracy, loss, a], feed_dict={x: X_b, y: Y_b}) losses = losses + np.sum(l) accs = accs + np.sum(acc) print("Epoch %.8d " % epoch, "avg train loss over