有关深层学习的FAU讲义 (FAU LECTURE NOTES ON DEEP LEARNING)

These are the lecture notes for FAU’s YouTube Lecture “Deep Learning”. This is a full transcript of the lecture video & matching slides. We hope, you enjoy this as much as the videos. Of course, this transcript was created with deep learning techniques largely automatically and only minor manual modifications were performed. Try it yourself! If you spot mistakes, please let us know!

这些是FAU YouTube讲座“ 深度学习 ”的 讲义 。 这是演讲视频和匹配幻灯片的完整记录。 我们希望您喜欢这些视频。 当然，此成绩单是使用深度学习技术自动创建的，并且仅进行了较小的手动修改。 自己尝试！ 如果发现错误，请告诉我们！

导航 (Navigation)

Previous Lecture / Watch this Video / Top Level / Next Lecture

上一个讲座 / 观看此视频 / 顶级 / 下一个讲座

Welcome back to deep learning! So today, we want to talk about the further advanced methods of image segmentation. Let’s look at our slides. You can see here this is part two of this lecture video series on image segmentation and object detection.

欢迎回到深度学习！ 因此，今天，我们要讨论图像分割的其他高级方法。 让我们看一下幻灯片。 您可以在这里看到这是本讲座视频系列中有关图像分割和目标检测的第二部分。

Now, the key idea that we need to know about is how to integrate the context knowledge. Just using this encoder-decoder structure that we talked about in the last video will not be enough to get a good segmentation. The key concept is that you somehow have to tell your method where what happened in order to get a good segmentation mask. You need to balance local and global information. Of course, this is very important because the local information is crucial to give good pixel accuracy and the global context is important in order to figure out the classes correctly. CNNs typically struggle with this balance. So, we now need some good ideas on how to incorporate this context information.

现在，我们需要了解的关键思想是如何整合上下文知识。 仅使用我们在上一个视频中讨论过的这种编解码器结构还不足以实现良好的分割效果。 关键概念是您必须以某种方式告诉您的方法发生了什么，才能获得良好的分割蒙版。 您需要平衡本地和全局信息。 当然，这非常重要，因为本地信息对于提供良好的像素精度至关重要，而全局上下文对于正确地确定类别至关重要。 CNN通常会为此寻求平衡。 因此，我们现在需要一些有关如何合并此上下文信息的好主意。

Now Long et al. showed one of the first approaches to do so. They essentially using an upsampling that is consisting of learnable transposed convolutions. The key idea was that you want to add links combining the final prediction with the previous lower layers in the finer strides. Additionally, he had 1x1 convolutions after the pooling layer, and then the predictions were added up to make local predictions with a global structure. So the network topology is a directed acyclic graph with skip connections from lower to higher layers. Therefore, you can then refine a coarse segmentation.

现在龙等。 展示了这样做的第一种方法。 他们本质上使用的是由可学习的转置卷积组成的升采样。 关键思想是您要添加将最终预测与以前的较低层结合在一起的链接，以实现更精细的步幅。 此外，他在池化层之后具有1x1卷积，然后将这些预测相加以形成具有全局结构的局部预测。 因此，网络拓扑是有向无环图，具有从较低层到较高层的跳过连接。 因此，您可以随后细化粗略的细分。

So, let’s look at this idea in some more detail. You can see now if you have the ground truth here on the bottom right, this has a very high resolution. If you would simply use your CNN and upsample, you would get a very coarse resolution as shown on the left-hand side. So, what is Long et al. proposing? Well, they propose then to use the information from the previous downsampling step which still had higher resolution and use it within the decoder branch using a sum to produce a more highly resolved image. Of course, you can then do this again in the decoder branch. You can see that this way we can upsample the segmentation and reuse the information from the encoder branch in order to produce better highly resolved results. Now, you can introduce those skip connections and they produce much better segmentations than if you were just using the decoder and upsample that information.

因此，让我们更详细地看一下这个想法。 现在，您可以看到是否在右下角具有基本事实，这具有很高的分辨率。 如果仅使用CNN和上采样，您将获得非常粗糙的分辨率，如左侧所示。 那么，Long等人是什么。 提出？ 好吧，他们建议然后使用来自先前的下采样步骤的信息，该信息仍然具有更高的分辨率，并在解码器分支中使用总和来使用它来生成分辨率更高的图像。 当然，您可以在解码器分支中再次执行此操作。 您可以看到，通过这种方式，我们可以对细分进行上采样并重新使用来自编码器分支的信息，以便产生更好的高度解析结果。 现在，您可以引入那些跳过连接，与仅使用解码器并对该信息进行上采样相比，它们可以产生更好的分段效果。

You see integrating context knowledge is key. In SegNet, a different approach was taken here. You also have this encoder-decoder structure that is convolutional. Here, the key idea was that in the upsampling step, you reuse the information from the max-pooling in the downsampling steps such that you get better-resolved decoding. This is already a pretty good idea to integrate the context knowledge.

您会看到整合上下文知识是关键。 在SegNet中，此处采用了另一种方法。 您还具有卷积的编码器-解码器结构。 在这里，关键思想是，在上采样步骤中，您将在下采样步骤中重用最大池中的信息，以便获得更好解析的解码。 集成上下文知识已经是一个很好的主意。

An even better idea is then demonstrated in U-net. Here, the network consists of the encoder branch which is then a contracting path to capture the context. The decoder branch does symmetric expansion for the localization. So, the encoder follows the typical structure of a CNN. The decoder now consists of the upsampling step and a concatenation of the previous feature maps of the respective layers of the corresponding encoder step. So, then the training strategy relies also on data augmentation. There were non-rigid deformations, rotations, and translations that were used to give the U-net an additional kick of performance.

然后在U-net中展示了一个更好的主意。 在此，网络由编码器分支组成，该分支随后是捕获上下文的收缩路径。 解码器分支进行对称扩展以进行本地化。 因此，编码器遵循CNN的典型结构。 解码器现在包括上采样步骤和相应编码器步骤各个层的先前特征图的串联。 因此，训练策略也依赖于数据扩充。 存在非刚性的变形，旋转和平移，这些变形，旋转和平移使U-net有了更多的性能提升。

You can say that U-net is essentially the state-of-the-art method for image segmentation. This is also the reason why it has this name. It stems from its shape. You can see that you get this U structure because you have a high resolution on the fine levels. Then you downsample to a lower resolution. The decoder branch upsamples everything again. The key information is here the skip connections that are connecting the two respective levels of the decoder and the encoder. This way you can get very, very good image segmentation. It’s quite straightforward to train and this paper has been cited thousands of times (August 11th, 2020: 16471 citations). Every day you can check the citation count and it already increased. Olaf Ronneberger was able to publish a very important paper here and it’s dominating the entire scene of image segmentation.

可以说，U-net本质上是用于图像分割的最新方法。 这也是它具有此名称的原因。 它源自其形状。 您会看到您获得了此U结构，因为您在精细级别上具有高分辨率。 然后，您将采样降低到较低的分辨率。 解码器分支再次对所有内容进行升采样。 此处的关键信息是跳过连接，它们连接了解码器和编码器的两个相应级别。 这样，您可以获得非常非常好的图像分割。 培训非常简单，本文被引用了数千次 (2020年8月11日：16471次引用)。 每天您都可以检查引文计数，并且它已经增加了。 Olaf Ronneberger能够在这里发表非常重要的论文，它主导着整个图像分割领域。

You can see that there are many additional approaches. They can be implemented with the U-net. So they can use dilated convolutions and many more. There have been many of these very small changes that have been suggested and they may be useful for particular tasks but for general image segmentation, the U-net has been shown to still outperform such approaches. Still, there are things using like dilated convolutions, there are network stacks that can be very beneficial, and there’s also multi-scale networks that then even further go into this idea of using the image at different scales. You can also do things like deferring the context modeling to another network. Then, you can also incorporate recurrent neural networks. Also very nice is the idea to refine the resulting segmentation maps using a conditional random field.

您可以看到还有许多其他方法。 它们可以通过U-net实施。 因此，他们可以使用膨胀卷积等等。 已经提出了许多这些非常小的更改，它们可能对特定任务很有用，但对于一般的图像分割， 已显示出U-net仍然优于此类方法 。 仍然有一些东西使用诸如膨胀卷积之类的东西，有一些网络堆栈可能会非常有益，而且还有多尺度网络，它们甚至进一步陷入了以不同尺度使用图像的想法。 您还可以执行将上下文建模推迟到另一个网络的操作。 然后，您还可以合并递归神经网络。 使用条件随机字段细化生成的分割图的想法也非常好。

We have some of these additional approaches here, such that you can see what we are talking about. The dilated convolutions, here, is the idea that you want to use those atrous convolutions that we already talked about. So, the idea is that you use dilated convolutions to support exponentially expanding the receptive field without losing the resolution. Then, you introduce the dilation rate L that controls the upsampling factor. You then stack this on top such that you make the receptive field grow exponentially while the number of parameters for the filters grows linear. So, in specific applications where you have a broad range of magnifications happening, this can be very useful. So, it really depends on your application.

我们这里有一些其他方法，您可以了解我们在说什么。 这里的膨胀卷积就是您想要使用我们已经讨论过的那些无用卷积的想法。 因此，我们的想法是使用膨胀卷积来支持以指数方式扩展接收场而不损失分辨率。 然后，介绍控制上采样因子的膨胀率L。 然后，将其堆叠在顶部，以使接收场呈指数增长，而过滤器的参数数量呈线性增长。 因此，在发生各种放大倍数的特定应用中，这可能非常有用。 因此，这实际上取决于您的应用程序。

Examples for this are DeepLab, ENet, and the multi-scale context aggregation module in [28]. The main issue, of course, is there’s no efficient implementation available. So, the benefit is somewhat unclear.

例子包括DeepLab，ENet和[28]中的多尺度上下文聚合模块。 当然，主要问题是没有有效的实现方法。 因此，收益尚不清楚。

Another approach that I would like to show you here is these so-called stacked hourglass network. So, here, the idea is that you use something very similar to a U-net, but you would put in an additional trainable part in the skip connection. So, that’s essentially the main idea.

我想在这里向您展示的另一种方法是这些所谓的堆叠沙漏网络。 因此，这里的想法是您使用与U-net非常相似的东西，但是您将在跳过连接中加入一个额外的可训练部分。 因此，这基本上是主要思想。

Then, you can use this hourglass module and stack it behind each other. So, you have essentially multiple refinement steps after each other and you always return to the original resolution. You can plug in a second network essentially as a kind of artifact correction network. Now, what’s really nice about this kind of hourglass network approach is that you return to the original resolution.

然后，您可以使用此沙漏模块并将其堆叠在一起。 因此，您彼此之间实际上有多个优化步骤，并且您始终会返回原始分辨率。 您可以实质上将第二种网络插入为一种伪像校正网络。 现在，这种沙漏网络方法的真正好处是您恢复了原始分辨率。

Let’s say you’re predicting several classes at the same time. Then, you end up with several segmentation masks for the different classes. This idea can then be picked up in something that is called a convolutional pose machine. In the convolutional pose machine, you then use the area where your hourglasses connect, where you have one U-net essentially stacked on top of another U-net. At this layer, you can then also use the resulting segmentation maps per class in order to inform them of each other. So, you can use the context information of other things that have been detected in the image in order to steer this refinement. In convolutional pose machines, you do that for pose detection of joints of a body model. Of course, if you have the left knee joint and the right knee joint and other joints of the body the information about the other joints helps in decoding the correct position.

假设您要同时预测几个课程。 然后，您将获得针对不同类的多个细分掩码。 然后可以在称为卷积姿势机的东西中找到这个想法。 然后在卷积姿势机中，使用沙漏连接的区域，在该区域中，一个U网络实际上堆叠在另一个U网络的顶部。 然后，在此层，您还可以使用每个类生成的分割图，以相互告知它们。 因此，您可以使用已在图像中检测到的其他事物的上下文信息来引导此优化。 在卷积姿势机中，您可以执行此操作以检测人体模型的关节。 当然，如果您的身体有左膝关节和右膝关节以及其他关节，则有关其他关节的信息将有助于解码正确的位置。

This idea has also been used by my colleague Bastian Bier for the detection of anatomic landmarks in the analysis of x-ray projections. I’m showing a small video here. You’ve already seen that in the introduction and now you finally have all the context that you need to understand the method. So, here you have an approach behind it that is very similar to convolutional pose machines that then start informing the landmarks about each other’s orientation and position in order to get improved detection results.

我的同事巴斯蒂安·比尔( Bastian Bier)也已将此想法用于在X射线投影分析中检测解剖标志 。 我在这里显示一个小视频。 您已经在引言中看到了这一点，现在终于有了理解该方法所需的所有上下文。 因此，这里有一个与卷积姿势机非常相似的方法，该方法然后会开始向界标通知彼此的方向和位置，以便获得更好的检测结果。

So what else? I already hinted at the conditional random fields. Here, the idea is that you refine the output using a conditional random field. So, the pixel is modeled as a node in a random field. Theses pair-based terms between the pixels are very interesting because they can capture long-range dependencies and fine, local information.

还有什么？ 我已经暗示条件随机字段。 这里的想法是，您可以使用条件随机字段来优化输出。 因此，将像素建模为随机字段中的节点。 这些像素之间的基于对的术语非常有趣，因为它们可以捕获远程依赖关系和精细的本地信息。

So if you see the output here, this is from DeepLab. Here, you see how the iterative refinement of the conditional random field then can help to improve the segmentation. So, you can then also combine this with artous convolutions as in [4]. You could even model the conditional random field with recurrent neural networks as shown in reference [29]. This then also allows end-to-end training of the entire conditional random field.

因此，如果您在此处看到输出，则来自DeepLab。 在这里，您将看到条件随机字段的迭代细化如何可以帮助改善分割。 因此，您还可以将其与虚假卷积相结合，如[4]中所示。 您甚至可以使用递归神经网络为条件随机场建模，如参考文献[29]所示。 然后，这还允许对整个条件随机字段进行端到端训练。

There are also a couple of advanced topics that I still want to hint at. Of course, you can also work with the losses. So far, we’ve only seen the segmentation loss itself but of course, you can also mix and match from previous ideas that we already saw in this class. For example, you can use a GAN in order to augment your loss. The idea here is then that you can essentially create a segmentor. You can then use the output of the segmentor as an input to a GAN type of discriminator. The discriminator now gets the task to say whether this is an automatic segmentation or a manual one. Then, this can be used as a kind of additional adversarial loss inspired by the ideas of the generative adversarial networks. You find that very often in literature as the so-called adversarial loss.

我还想暗示几个高级主题。 当然，您也可以弥补损失。 到目前为止，我们仅看到细分损失本身，但是当然，您也可以将在本课程中已经看到的以前的想法中混搭起来。 例如，您可以使用GAN来增加损失。 然后，这里的想法是，您基本上可以创建一个细分器。 然后，您可以将分段器的输出用作GAN类型的鉴别器的输入。 现在，判别器将获得任务来说这是自动分段还是手动分段。 然后，这可以用作一种受生成对抗网络思想启发的附加对抗损失。 您在文学中经常发现这种所谓的对抗损失。

So, how is this then implemented? Well, the idea is that if you have a data set given of N training images and the corresponding label maps, you can then build the following loss function: This is essentially the multi-class cross-entropy loss and then you put on top the adversarial loss that works on your segmentation masks. So here, you can then essentially train your segmentation both with the ground truth label and on fooling the discriminator. This is essentially nothing else than a multi-task learning approach with an adversarial task.

那么，如何实现呢？ 好吧，这个想法是，如果您有一个包含N个训练图像和相应标签图的数据集，则可以构建以下损失函数：这本质上是多类交叉熵损失，然后将对您的细分蒙版有效的对抗性损失。 因此，在这里，您实际上可以使用基本事实标签和欺骗鉴别器来训练您的细分。 从本质上讲，这只是带有对抗任务的多任务学习方法。

Okay. So, this already brings us to the end of our short video today. You see that we’ve seen now the key ideas on how to build good segmentation networks. In particular, U-net is one of the key ideas that you should know about. Now that we have discussed the segmentation networks, we can talk in the next lecture about object detection and how to actually implement that very quickly. So, this is the other side of image interpretation. We will also be able to figure out where different instances in the image actually are. So I hope, you liked this small video and I’m looking forward to seeing you in the next one. Thank you very much and bye-bye.

好的。 因此，这已经使我们结束了今天的简短视频。 您会看到，我们已经看到了有关如何构建良好的细分网络的关键思想。 特别是，U-net是您应该了解的关键思想之一。 既然我们已经讨论了分割网络，那么我们可以在下一讲中讨论对象检测以及如何非常快地实现它。 因此，这是图像解释的另一面。 我们还将能够找出图像中不同实例的实际位置。 因此，我希望您喜欢这个小视频，希望在下一个视频中见到您。 非常感谢，再见。

If you liked this post, you can find more essays here, more educational material on Machine Learning here, or have a look at our Deep LearningLecture. I would also appreciate a follow on YouTube, Twitter, Facebook, or LinkedIn in case you want to be informed about more essays, videos, and research in the future. This article is released under the Creative Commons 4.0 Attribution License and can be reprinted and modified if referenced. If you are interested in generating transcripts from video lectures try AutoBlog.

如果你喜欢这篇文章，你可以找到这里更多的文章 ，更多的教育材料，机器学习在这里 ，或看看我们的深入 学习 讲座 。 如果您希望将来了解更多文章，视频和研究信息，也欢迎关注YouTube ， Twitter ， Facebook或LinkedIn 。 本文是根据知识共享4.0署名许可发布的 ，如果引用，可以重新打印和修改。 如果您对从视频讲座中生成成绩单感兴趣，请尝试使用AutoBlog 。