This post covers Markov Random Fields and their application to specific problems in image processing. We don’t go deep or formal, just intuitive.

这篇文章介绍了马尔可夫随机场及其在图像处理中特定问题的应用。 我们不深入，也不正式，仅凭直觉。

A Markov Random Field is a graph whose nodes model random variables, and whose edges model desired local influences among pairs of them. Local influences propagate globally, leveraging the connectivity of the graph.

马尔可夫随机场是一个图，其节点建模随机变量，并且其边缘建模它们对之间的所需局部影响。 利用图的连通性，局部影响会在全球范围内传播。

Here is an example. Consider an image on a rectangular grid. It’s composed of pixels. Each pixel has a value, denoting its color. Internally a pixel may be represented as a binary variable for black-and-white images, a continuous variable for grey-scale images, and use mixtures of multiple variables (e.g., RGB) for color images. These differences will not matter to us here. As far as we are concerned, a pixel has a value that represents its color.

这是一个例子。 考虑在矩形网格上的图像。 它由像素组成。 每个像素都有一个值，表示其颜色。 在内部，像素可以表示为用于黑白图像的二进制变量，用于灰度图像的连续变量，并可以将多个变量(例如RGB)的混合用于彩色图像。 这些差异对我们而言并不重要。 就我们而言，像素具有代表其颜色的值。

A natural MRF that models this has a node for each of the pixels. An edge connects two nodes that are adjacent (on the grid). Below is an example MRF on a 3x3 grid. The vertical and horizontal lines are its edges.

对此建模的自然MRF对于每个像素都有一个节点。 一条边连接两个相邻的节点(在网格上)。 以下是3x3网格上的示例MRF。 垂直线和水平线是其边缘。

o — o — o|   |   |o — o — o|   |   |o — o — o

Okay, the structure of this MRF matches the grid. That sounds like a good thing. Is there a deeper reason for choosing this structure? Yes.

好的，此MRF的结构与网格匹配。 听起来不错。 选择这种结构是否有更深层次的原因？ 是。

Real-world images tend to be smooth. That is, adjacent pixels tend to have similar colors. Sure, they might also be abrupt transitions such as edges or object boundaries. However, there is generally a lot of smoothness between these transitions.

现实世界中的图像趋于平滑。 即，相邻像素倾向于具有相似的颜色。 当然，它们也可能是突然的过渡，例如边缘或对象边界。 但是，这些过渡之间通常存在很多平滑度。

It is the MRF’s edges that somehow favor that adjacent pixels have similar values. How? First, it will help to introduce some new concepts. Then we will come back to answer this question.

MRF的边缘以某种方式支持相邻像素具有相似的值。 怎么样？ 首先，它将有助于引入一些新概念。 然后，我们将回来回答这个问题。

Cliques And Energy Functions

派系和能量函数

This material is going to be a bit obtuse. You might wonder, why do I need to know all this? The dots will start connecting soon, so bear with me a while.

这种材料会有点钝。 您可能想知道，为什么我需要了解所有这一切？ 点将很快开始连接，请耐心等待一段时间。

A clique in a graph is a set of nodes in which every pair is connected by an edge. A clique is maximal if it is not part of a larger one.

图中的集团是一组节点，其中每对节点通过一条边连接。 如果集团不是较大集团的一部分，则它是最大的。

Maximal cliques play a crucial role in how an MRF operates. Intuitively each such clique specifies a set of nodes, all of whose values directly influence each other.

最高集团在MRF的运作中起着至关重要的作用。 直观地，每个这样的集团都指定了一组节点，所有节点的值都直接相互影响。

In our 2D image example, the maximal cliques are the graph’s edges. No triangles are possible since an edge can connect only adjacent pixels.

在我们的2D图像示例中，最大集团是图形的边缘。 三角形是不可能的，因为边缘只能连接相邻的像素。

Okay, so we’ve characterized which sets of nodes directly influence each other. How do we control the nature of this influence? In our 2D image example, we want adjacent pixels to have similar values.

好的，我们已经确定了哪些节点集会直接相互影响。 我们如何控制这种影响的性质 ？ 在我们的2D图像示例中，我们希望相邻像素具有相似的值。

We control the specifics of the interaction among the nodes’ values in a clique by a so-called energy function of our choice. This function (one per maximal clique) inputs values for all the clique nodes and outputs its energy, a scalar. The energy function we choose should assign low energy to a good combination of values of these nodes. That is, low energy is good.

我们通过选择的所谓能量函数来控制群中节点值之间的交互作用的细节。 此函数(每个最大集团一个)输入所有集团节点的值并输出其能量(标量)。 我们选择的能量函数应将低能量分配给这些节点的值的良好组合。 即，低能量是好的。

The global energy for a particular assignment of values to its nodes is the sum of all the maximal cliques’ energies in the graph.

对其节点进行特定值分配的全局能量是图中所有最大派系能量的总和。

Smoothness-favoring Energy Function. In our image MRF, what energy function would deliver smoothness? Consider |xi-xj|. Here i and j denote adjacent pixels, with xi and xj denoting their values. This energy function prefers that xi and xj have similar values. As a result, global energy will be minimized by an image that is maximally smooth.

光滑度有利的能量函数 。 在我们的图像MRF中，什么能量函数可以提供平滑度？ 考虑| xi - xj |。 在此， i和j表示相邻像素，而xi和xj表示其值。 该能量函数优选xi和xj具有相似的值。 结果，将通过最大程度平滑的图像将全局能量最小化。

Note that, under-the-hood, xi, and xj, might have multiple dimensions, e.g., RGB for color. We don’t care so long as we can meaningfully calculate a distance between the two colors, which is what |xi-xj| is.

注意，引擎盖xi和xj可能具有多个尺寸，例如，RGB用于颜色。 只要我们可以有意义地计算两种颜色之间的距离，我们就不在乎了。 xi - xj | 是。

Well, that’s nice, but what good is a maximally smooth image? All pixels would have the same color.

好吧，那很好，但是最大平滑的图像有什么好处呢？ 所有像素将具有相同的颜色。

Below we see some specific useful image processing problems in which smoothness, hence this MRF, plays an important role.

下面我们看到一些特定的有用图像处理问题，其中平滑度(因此称为MRF)起着重要作用。

Image Denoising

图像去噪

Consider this example.

考虑这个例子。

Look at the white dots (stars). Let’s say someone would like to think of them as noise (even though they are not). With a click of a button, the person can ask this image to be denoised. The denoiser’s job is to identify these white dots as the most likely culprits and blacken them.

看白点(星号)。 假设有人希望将它们视为噪音(即使它们不是噪音)。 单击按钮，此人可以要求将此图像去噪。 去噪器的工作是将这些白点识别为最可能的元凶并将其变黑。

Why would we expect the MRF to be able to detect these white dots as noise? For the sake of discussion, assume that each white dot is a single pixel. A dot’s pixel is white, and all its neighbors are black. Moreover, black and white are very different-looking colors. The MRF doesn’t like this disharmony. That is, global energy would decrease significantly if we made the spot’s color black. This reasoning applies to all the white dots.

为什么我们期望MRF能够将这些白点检测为噪声？ 为了讨论起见，假设每个白点是单个像素。 点的像素为白色，所有相邻像素均为黑色。 而且，黑白是非常不同的颜色。 MRF不喜欢这种不和谐。 也就是说，如果将斑点的颜色设为黑色，则全球能源将大大减少。 此推理适用于所有白点。

As to any pixel in the bright red sphere, it has at least some neighbors with a similar color. So drastically changing the color of such a pixel will not reduce the energy.

对于亮红色球体中的任何像素，它至少具有一些具有相似颜色的邻居。 因此，大幅改变此类像素的颜色不会降低能量。

A Richer MRF for Image Denoising

更丰富的MRF图像去噪

This MRF has two sheets of nodes. The nodes in both sheets represent the pixels, as laid out on the grid. The edges of the first sheet are determined by the grid, as before. There are no edges among nodes in the second sheet. An edge connects the two nodes in the two sheets that represent the same pixel.

该MRF有两层节点。 两张纸上的节点均表示像素，如网格所示。 如前所述，第一张纸的边缘由网格确定。 第二张表中的节点之间没有边缘。 一条边连接了代表相同像素的两个图纸中的两个节点。

Below is an example MRF on a 1X3 grid.

以下是1X3网格上的示例MRF。

o — o — o      sheet one|   |   |o   o   o      sheet two

We’ll denote the nodes in sheet one as Y and those in sheet two as X. We’ll set X’s values from the image to be denoised. These values stay frozen. The values in Y represent the denoised image. These values change as the denoising process evolves.

我们将工作表1中的节点表示为Y，将工作表2中的节点表示为X。我们将从要去噪的图像中设置X的值。 这些值保持冻结。 Y中的值表示去噪图像。 这些值随着去噪过程的发展而变化。

We’ll use the same energy function as before. What is the effect of this on the global energy? It tries to find a denoised image that is both close to the input image and smooth(er) than it.

我们将使用与以前相同的能量函数。 这对全球能源有什么影响？ 它试图找到去噪图像，该图像既靠近输入图像，又比输入图像平滑(平滑)。

Image Restoration

影像还原

Here we are given an image that is both noisy (i.e. some of its pixels have incorrect values) and incomplete (i.e., some of its pixels don’t have a value). The two-sheet MRF is a good fit for solving this problem as well. As before, we’ll set X’s values from the input image. Except that pixels in the input image that are missing values are left free.

在这里，我们给出的图像既有噪点 (即某些像素的像素值不正确)又有残缺 (即某些像素的像素值没有值)。 两页式MRF也非常适合解决此问题。 和以前一样，我们将从输入图像中设置X的值。 输入图像中缺少值的像素除外。

Image Segmentation

图像分割

Here we’d like to segment the image into maximally homogenous regions. Homogeneity means that some property remains similar in the entire area. The property might be color. Or texture. Or a combination. Maximally means that different regions, especially adjacent ones, are dissimilar. Sound like clustering? Yes, in a 2D setting on image properties: color, etc.

在这里，我们想将图像分割为最大同质区域。 同质性意味着某些属性在整个区域中保持相似。 该属性可能是颜色。 或纹理。 或组合。 最大限度地意味着不同的区域(尤其是相邻区域)是不同的。 听起来像集群吗？ 是的，在图像属性的2D设置中：颜色等。

A particular critical case of segmentation is to split the image into foreground and background. In our sphere example, the sphere would be the foreground, and the black sky (including the stars) the background.

分割的一个特殊关键情况是将图像分为前景和背景。 在我们的球体示例中，球体将是前景，而黑色的天空(包括星星)将是背景。

In general, even such binary segmentation is not easy to do. Unlike the simple background in our example (black sky), others may have varying textures or colors. Plus, the image may contain multiple objects. In fact, when this is the case, how is the foreground even defined?

通常，即使这样的二进制分段也不容易做到。 与我们的示例中的简单背景(黑色的天空)不同，其他背景可能具有不同的纹理或颜色。 此外，图像可能包含多个对象。 实际上，在这种情况下，前景如何定义？

An effective way to both simplify and disambiguate the problem is called interactive segmentation. The user draws a bounding box around the desired object to be segmented. The algorithm takes this box as a rough initial segmentation and tries to improve upon it.

简化和消除歧义的有效方法称为交互式细分 。 用户围绕要分割的所需对象绘制一个边界框。 该算法将此框作为粗略的初始分割，并尝试对其进行改进。

Take the sky example modified so that there are two distinct spheres in it. Think of them like the sun and the moon. The user might put the bounding box around the moon. So both the sun and the sky would be in the background.

以修改后的天空示例为例，其中有两个不同的球体。 把它们想像成太阳和月亮。 用户可以将包围盒放在月球周围。 因此，太阳和天空都将在背景中。

Architecture and Energy Function

建筑与能源功能

We’ll use the two-sheet MRF structure here as well. As before, sheet X will contain the image. Sheet Y will capture its segmentation. Sheet Y’s nodes are binary-valued, with zero denoting background and one foreground. The edges in Y will form the same grid structure as before. The edges connecting X to Y will also be the same.

我们还将在这里使用两页MRF结构。 和以前一样，工作表X将包含图像。 表格Y将捕获其细分。 工作表Y的节点是二进制值， 零表示背景， 一个表示前景。 Y中的边缘将形成与以前相同的网格结构。 连接X和Y的边也将相同。

We’ll use a different energy function tailored to this problem. Why? It seems reasonable to impose substantial homogeneity on the foreground and weaker (if any) on the background. How can we make this happen? By choosing an energy function that propagates a 1-value with more ‘force’ than a 0-value to neighboring nodes.

我们将使用针对此问题的不同能量函数。 为什么？ 在前景上强加同质性，在背景上强弱(如果有)，似乎是合理的。 我们如何做到这一点？ 通过选择一个能量函数，该函数将具有比0值更大的“力”的1值传播到相邻节点。

We’ll illustrate this algorithm and energy function on this, especially simple example. (Albeit, not good-looking.)

我们将在这个特别简单的示例上说明该算法和能量函数。 (尽管不好看。)

______ __|___ ____| ____ ______

This example is a 1D binary image. The horizontal line segments are runs of black pixels of varying lengths. These are punctuated by runs of white pixels. The vertical bars demarcate the bounding box.

本示例是一维二进制图像。 水平线段是长度可变的黑色像素的行。 这些通过白色像素点来打断。 垂直条划分边界框。

Now to the illustration. The initial assignment of Y given the bounding box is

现在到插图。 给定边界框，Y的初始赋值为

Y           ________X ______ __|___ ____| ____ ______

Note that the white pixel in the interior of the box in X is already black in Y. It is as if we were using the prior belief that the interior of the bounding box specified by the user should all be deemed as being in the foreground unless there is substantive evidence to the contrary.

请注意，X中的框内部的白色像素在Y中已经是黑色的。这似乎是我们在先使用一种信念，即用户指定的边界框的内部应全部视为前景，除非有大量相反的证据。

From this initial Y, the algorithm will extend the bounding box two pixels to the left. Why? Because each of the components of the energy function will like this extension. Why? Because the smoothness component likes to propagate a 1-value to its neighbors. And it so happens that the actual pixels in the image at those points are also 1-valued (black). So the algorithm will stop at the Y depicted below.

从此初始Y开始，算法会将边界框向左扩展两个像素。 为什么？ 因为能量函数的每个组件都会喜欢这个扩展。 为什么？ 因为平滑度组件喜欢将1值传播到其邻居。 碰巧的是，这些点上图像中的实际像素也为1值(黑色)。 因此，该算法将在以下所示的Y处停止。

Y        ___________X ______ __|___ ____| ____ ______

Higher-Order Models

高阶模型

Can we enrich our model without sacrificing its simplicity and elegance? Here we touch on how? For more extensive treatment, see [1].

我们可以在不牺牲其简洁性和优雅性的前提下丰富我们的模型吗？ 在这里我们谈如何？ 有关更广泛的治疗方法，请参见[1]。

One approach is to extend the neighborhood of a pixel, for example, by adding diagonal connections, or connecting two pixels if they are within a specific distance d. For the latter, we might use Euclidean distance or Manhattan distance.

一种方法是例如通过添加对角线连接或如果两个像素在特定距离d之内则连接两个像素来扩展像素的邻域。 对于后者，我们可以使用欧几里得距离或曼哈顿距离。

If we are going to extend the neighborhood of a pixel, we might introduce weights on the edges decaying with the distance between the pixels they connect.

如果要扩展像素的邻域，可以在边缘上引入权重，该权重随它们所连接的像素之间的距离而衰减。

A second approach is to have nodes represent not individual pixels but square patches of pixels. The value of a patch would be whatever we want it to be. For instance, the average intensity over the intensities of the pixels in the patch. An edge will connect the nodes representing two patches if the patches are sufficiently close. A weight could also be attached to an edge based on the distance between the patches.

第二种方法是让节点代表的像素不是单个像素，而是正方形像素。 补丁的价值将是我们想要的价值。 例如，贴片中像素强度之上的平均强度。 如果补丁足够接近，则边缘将连接代表两个补丁的节点。 也可以基于补片之间的距离将重物附接至边缘。

Further Reading

进一步阅读

1. Models for grids

   网格模型

2. Gibbs Fields & Markov Random Fields 1 Gibbs Fields

   吉布斯场和马尔可夫随机场1吉布斯场

3. Markov Random Fields and Gibbs Sampling for Image Denoising

   马尔可夫随机场和Gibbs采样进行图像降噪