This project stemmed from my predilection of the visual arts — as a computing student, I’ve always envied art students in their studios with splodges of colors and pristine white canvases. However, when I’m faced with a blank canvas and a (few) bottle(s) of paint, I don’t know where to start. Without the hours of training Art students have, I simply can’t go very far unless I’m following some Bob Ross tutorial. That’s why I’ve decided to explore and see if I can code something that will enable me to create art “automatically”.

这个项目源于我对视觉艺术的偏爱-作为一名计算机专业的学生，​​我总是在他们的工作室里用色彩斑p的原始画布和白色的原始画布羡慕艺术学生。 但是，当我面对一块空白的画布和几瓶油漆时，我不知道从哪里开始。 如果没有美术专业学生的训练，除非我正在听一些Bob Ross的教程，否则我走不了多远。 这就是为什么我决定探索并查看是否可以编写使我能够“自动”创作艺术作品的原因。

I did this project a looong time ago for my final year project when I wanted to create some software program that can allow its users to sketch something stickman-like on a pre-selected background, select the category of what was drawn, and the programme will intelligently figure out which image to poisson-blend in. After that, the user can also select a neural transfer style to be applied on the image.

我在不久前的最后一个项目中做了这个项目，当时我想创建一些软件程序，该程序可以让用户在预选的背景上绘制类似于stickman的东西，选择绘制的类别，然后选择该程序会智能地找出要进行泊松混合的图像。此后，用户还可以选择要应用于图像的神经传递样式。

So in this article I will be giving a brief walkthrough of this project. Because there are many parts to it, I won’t be going in-depth with the technical implementations. However, if you’d like me to do so, leave a comment and let me know! :)

因此，在本文中，我将简要介绍该项目。 因为涉及到很多部分，所以我不会深入探讨技术实现。 但是，如果您希望我这样做，请发表评论并告诉我！ :)

第1部分-背景图片的选择 (Part 1 — Selection of the background image)

Because I wanted the user to have a range of images to choose from, I’ve coded a python script that acted as an image search engine. If an image is selected as the user browses through a range of available background images, similar looking background images would be retrieved. The similarity is calculated based on colour histograms in the HSV colour space. A more detailed explanation can be found here.

因为我希望用户可以选择一系列图像，所以我编写了一个Python脚本，用作图像搜索引擎。 如果在用户浏览一系列可用的背景图像时选择了图像，则将检索相似外观的背景图像。 基于HSV颜色空间中的颜色直方图计算相似度。 在这里可以找到更详细的解释。

Here’s the result. If a user selects this background image:

这是结果。 如果用户选择此背景图片：

Similar looking images would be retrieved:

看起来相似的图像将被检索：

第2部分-开始绘制！ (Part 2 — Start drawing!)

This is the fun part! The program seeks to match whatever you’re drawing with the set of images in the database. So if you draw an image with the wings spread out versus if you draw an oval (for example) the retrieved results would be different. Additionally, because the programme is not as robust, the user would have to specify the category (for example bird or boat, or car). For this particular project, I have six categories. The drawing function is also coded in Python with openCV. Once the outline is drawn, the list of coordinates would be saved. 100 sample points (the number I used for my testing, this can vary — the more points the more accurate but the programme would take longer) was taken from this outline of the drawing.

这是有趣的部分！ 该程序试图将您要绘制的内容与数据库中的图像集进行匹配。 因此，如果您绘制的图像张开了翅膀，而绘制了椭圆形(例如)，则检索到的结果将有所不同。 另外，由于程序不那么健壮，因此用户必须指定类别(例如，鸟，船或汽车)。 对于这个特定的项目，我有六个类别。 绘图功能也使用openCV在Python中编码。 绘制轮廓后，将保存坐标列表。 从该图形的轮廓中获取了100个采样点(我用于测试的数字可能会有所不同-采样点越多越准确，但是程序将花费更长的时间)。

第3部分-将样本点与预处理的图像描述符进行比较 (Part 3 — Comparing the sample points with pre-processed image descriptors)

Initially, I spent a very long time trying to figure out how to use image segmentation methods to get a clean cut of the target image objects so I could crop them out of their original images, figure out which one(s) matched my sketch the most, and them poisson-blend into my result image. However, this was really tricky to do and OpenCV’s image saliency techniques only worked well with certain types of images. For example,

最初，我花了很长时间试图弄清楚如何使用图像分割方法对目标图像对象进行清晰的裁剪，以便可以将其从原始图像中裁剪出来，从而找出与我的素描相匹配的图像。大多数情况下，它们都将泊松融合到我的结果图像中。 但是，这样做确实很棘手，而且OpenCV图像显着性技术仅适用于某些类型的图像。 例如，

However, if the background gets a little complicated or if the subject is not as clear, this happens:

但是，如果背景变得有点复杂或主题不太清楚，则会发生这种情况：

Hence, I decided to use the COCODataset instead.

因此，我决定改用COCODataset。

With these masks, I can use the outline of my sketch to compare against the shape context histogram descriptors of these masks. How it works can be broken down into four steps:

有了这些蒙版，我可以使用草图的轮廓与这些蒙版的形状上下文直方图描述符进行比较。 它的工作方式可以分为四个步骤：

Step 1: Finding the collection of points on shape contour

步骤1：在形状轮廓上找到点的集合

Step 2: Computing the shape context

步骤2：计算形状上下文

Step 3: Computing the cost matrix

步骤3：计算成本矩阵

Step 4: Finding the matching that minimizes total cost

步骤4：寻找可将总费用降至最低的匹配项

BY THE WAY, shape context is another fun topic to discuss that’ll take a full article by itself, so I shall not be discussing it here — but do let me know if that’ll be of interest!

顺便说一句，形状上下文是另一个有趣的话题，需要单独阅读全文，因此在这里我将不再讨论它-但请告诉我是否有兴趣！

Here’s some results:

结果如下：

So it works OKAY. The problem with the Hungarian matching algorithm is that it is extremely time costly — its O(n³)! This is also the part where I’m thinking of implementing some ML/DL techniques to speed up the process.

这样就可以了。 匈牙利匹配算法的问题在于，它非常耗时-O(n³)！ 这也是我正在考虑实现某些ML / DL技术以加速该过程的部分。

第4部分-将最接近的匹配项混合到所选背景图像中 (Part 4 — Blending the closest match into the selected background image)

The last part of this project (after using the target mask to “crop” the image out) is to use poisson blending to blend the image into our initially selected background image.

该项目的最后一部分(使用目标蒙版“裁剪”出图像之后)是使用泊松混合将图像混合到我们最初选择的背景图像中。

Poisson blending is one of the gradient domain image processing methods. Pixels in the resultant image are computed by solving a Poisson equation.

泊松混合是梯度域图像处理方法之一。 通过求解泊松方程来计算结果图像中的像素。

For each of the final pixels:

对于每个最终像素：

if mask(x,y) is 0:    final(x,y) = target(x,y) else:    final(x,y) = 0    for each neighbor (nx,ny) in (x-1,y), (x+1,y), (x,y-1), (x,y+1):    final(x,y) += source(x,y) - source(nx,ny)      if mask(nx,ny) is 0:        final(x,y) += target(nx,ny)      else:        final(x,y) += final(nx,ny)

All masked pixels with a non-zero value affects each other’s final value. The matrix equation Ax=b is solved to compute everything simultaneously. The size of vectors x and b are both number of pixels in the target image — vector x is what we are solving for and contains all pixels in the final image while vector b is the guiding gradient plus the sum of all non-masked neighbour pixels in the target image. These non-masked neighbour pixels are the values of pixels at the mask boundary and the guiding gradient defines the second derivative of the final mask area. Matrix A is a sparse matrix and computes the gradient of the final image’s masked pixels.The equation for x is solved to get the final image. If the guiding gradient is zero, we are just solving a Laplace equation and the values at the mask boundary are blended smoothly across. With the gradient of the source image as the guiding gradient, the mask area takes on the appearance of the source image but is smoothly blended at the boundary.

具有非零值的所有蒙版像素都会影响彼此的最终值。 求解矩阵方程Ax = b可以同时计算所有内容。 向量x和b的大小都是目标图像中的像素数-向量x是我们要求解的图像，并且包含最终图像中的所有像素，而向量b是引导梯度加所有未遮罩的相邻像素之和在目标图像中。 这些未遮罩的相邻像素是遮罩边界处的像素值，并且引导渐变定义了最终遮罩区域的二阶导数。 矩阵A是一个稀疏矩阵，用于计算最终图像的蒙版像素的梯度。求解x的方程以获得最终图像。 如果引导坡度为零，则我们正在求解拉普拉斯方程，并且将遮罩边界处的值平滑地混合在一起。 以源图像的梯度作为引导梯度，遮罩区域具有源图像的外观，但在边界处平滑融合。

OpenCV’s seamless clone function is an implementation of the algorithm mentioned above.

OpenCV无缝克隆功能是上述算法的实现。

Some results:

一些结果：

第5部分-使用预先训练的模型在图像上应用神经样式传递 (Part 5 — Applying neural style transfer on the image with pre-trained models)

I did a separate article on this here! Do check it out if interested. But in a nutshell, what it does is it takes the image above and apply some cool art style to it. There is of course a range to choose from and the range largely depends on which pre-trained neural models you use.

我在这里做了另一篇文章！ 如果有兴趣，请检查一下。 简而言之，它的作用是获取上面的图像并为其应用一些酷炫的艺术风格。 当然，有一个范围可供选择，范围很大程度上取决于您使用的预训练神经模型。

Here’s some examples:

这里有一些例子：

This project is actually far from done. Mainly because the matching algorithm takes too long and the pre-processing phase requires some manual intervention still. However, most of this (save for the last part) is just mathematical image manipulation techniques! If ML/DL techniques are used, the processing phase could potentially be improved. Nonetheless, I think it is still a fun project to work on and it made me learn a lot more about image processing techniques (and math lol).

这个项目实际上还远远没有完成。 主要是因为匹配算法花费的时间太长，并且预处理阶段仍然需要一些人工干预。 但是，大多数(最后一部分除外)只是数学图像处理技术！ 如果使用ML / DL技术，则可能会改善处理阶段。 尽管如此，我认为这仍然是一个有趣的项目，它使我学到了更多有关图像处理技术(和数学笑)的知识。

Anyhow, let me know if I should do a more in-depth walkthrough of each part!

无论如何，请告诉我是否应该对每个部分进行更深入的演练！

Happy OpenCV-ing! ୧(﹒︠ᴗ﹒︡)୨

OpenCV-ing祝您愉快！ ୧(﹒︠ᴗ﹒︡)୨