摄影测量计算机视觉领域

In this blog series, we will go over every aspect of the creation of our demo “Book of the Dead”. Today, we focus on photogrammetry assets, trees and VFX. This is the fourth blog in the series, take a look back at the last two blogs that go through creating characters and concept art from “Book of the Dead”.

在这个博客系列中，我们将介绍演示“ 死者之书 ”的创建过程的各个方面 。 今天，我们专注于摄影测量资产，树木和视觉特效。 这是系列中的第四个博客，回顾一下过去两个博客，这些博客通过 “死者之书” 创作 人物 和 概念艺术 。

Hello, my name is Zdravko Pavlov and I am a CG and VFX artist with background in VFX, video compositing, editing, graphic design. I’ve been working with Unity’s Demo team since 2014 and contributed various particles, rigid body dynamics and cloth simulations on the demos  “Viking village”, “The Blacksmith” and “Adam”.
The “Book Of The Dead” demo was a little bit different. A completely new territory for me, since my role for this project would be to create various environment assets using a photogrammetry approach. Outdoor photography is my hobby, so I was more than happy to handle such a task. Creating trees? I mean, how hard can it be, right? In the following blog post I’ll try to describe everything that I learned during the pre-production and development phase of the project.

您好，我叫Zdravko Pavlov，我是CG和VFX艺术家，具有VFX，视频合成，编辑，图形设计的背景。 自2014年以来，我一直在Unity的演示团队工作，并在“维京村”，“铁匠”和“亚当”的演示中贡献了各种粒子，刚体动力学和布料模拟。
“死者之书”演示有点不同。 对我来说，这是一个全新的领域，因为我在该项目中的职责是使用摄影测量方法创建各种环境资产。 户外摄影是我的爱好，因此我很乐意完成这项任务。 创造树木？ 我的意思是，这有多困难，对吗？ 在下面的博客文章中，我将尝试描述在项目的预生产和开发阶段学到的一切。

摄影测量工作流程 (The Photogrammetry workflow)

Fortunately, at this point, the Internet is full of valuable info regarding that process, so that’s where my learning began. What most of the articles would tell you is that what you need is any DSLR, camera with 50mm prime lens. I didn’t have any of those at my disposal at the time, so I decided to make my initial tests with my 24MP mirrorless Sony a7II with a 16mm-35mm zoom lens instead. And let me tell you right away it works just fine! The wider lens gives you more distortion, but you can always fix that in Lightroom for example, but in fact, it is better if you don’t! The photogrammetry software handles it gracefully. Prime lenses are more rigid and in theory, should give you a sharper image. They are really great if you scan in a controlled studio environment and I highly recommend it in such scenarios. Out in the field, however, being able to properly frame the desired object with a quality build zoom lens will give you an advantage.

幸运的是，此时，Internet上充满了有关该过程的宝贵信息，因此这就是我学习的起点。 大多数文章会告诉您，您需要的是任何配备50mm定焦镜头的数码单反相机。 当时我没有任何可用的设备，因此我决定使用配备16mm-35mm变焦镜头的24MP无反光镜Sony a7II进行初始测试。 而且，我马上告诉您，它的效果很好！ 较宽的镜头会给您带来更大的失真，但是您始终可以在Lightroom中进行修正，但实际上，如果不这样做，效果会更好！ 摄影测量软件可以很好地处理它。 定焦镜头更坚固，从理论上讲，应该可以为您提供更清晰的图像。 如果您在受控的工作室环境中进行扫描，它们确实很棒，我强烈建议在这种情况下进行扫描。 但是，在野外，能够使用高质量的内置变焦镜头正确地构筑所需的物体将为您带来优势。

I tried out most of the more popular photogrammetry software out there and some of them worked quite well. I chose RealityCapture because of its significantly better performance and ability to process a high amount of photos without running out of RAM. The amount of details it manages to reconstruct from the photos is amazing! I managed to get models, sometimes up to 185 million triangles and successfully export the geometry in PLY format.

我试用了大多数最受欢迎的摄影测量软件，其中一些运行良好。 我之所以选择RealityCapture，是因为它的性能明显更好，并且能够在不耗尽RAM的情况下处理大量照片。 它设法从照片中重建的细节数量真是惊人！ 我设法获得了模型，有时多达1.85亿个三角形，并成功以PLY格式导出了几何图形。

That, of course, is more than enough and also a little bit extreme. Most of my reconstructions ended up roughly about 50 to 90 million triangles. At first, I was using GF980TI, but later upgraded to GF1080 which gave me a slight performance boost.
At some point, I also upgraded my camera to a 42MP Sony aRII with a Planar T* FE 50mm f/1.4 ZA Lens. However, doubling the resolution and using the superior super sharp prime lens didn’t give me the “WOW” results I was expecting. For one thing, the longer (and narrower) prime lens means that you have to step a few steps back in order to have the image overlap that you need for a successful reconstruction. That’s not always possible when you are in the middle of the forest, with all the other trees, shrubs and everything. It also means that you have to manage, store and process twice as many gigabytes of image data. But that doesn’t necessarily lead you to higher definition scans. Having more images is what gets you there and having it in 24MP is more manageable. That may sound obvious, but it didn’t occur to me until I actually tried it first hand.

当然，这绰绰有余，而且还有些极端。 我的大部分重建工作最终大约形成了50到9000万个三角形。 最初，我使用的是GF980TI，但后来升级到GF1080，这使我的性能有了一点提升。
在某些时候，我还将相机升级为42MP Sony aRII，并配备了Planar T * FE 50mm f / 1.4 ZA镜头。 但是，将分辨率提高一倍并使用优质的超锐利定焦镜头并没有达到我期望的“哇”效果。 一方面，更长(更窄)的定焦镜头意味着您必须向后退几步才能使图像重叠，这是成功重建所需的图像。 当您与其他所有树木，灌木和所有其他事物一起位于森林中间时，这并非总是可能的。 这也意味着您必须管理，存储和处理两倍于千兆字节的图像数据。 但这并不一定会导致您进行高清扫描。 拥有更多图像可以带您到那里，而24MP中的图像则更易于管理。 这听起来似乎很明显，但是直到我真正进行第一手尝试之后，我才意识到这一点。

As I mentioned I used a PLY format to export the insanely dense geometry. I prefer that over FBX even though the PLY exporter of Reality Capture didn’t have scale and axis orientation controls so unlike the FBXs, the PLYs were out of scale and rotated. I chose to deal with that because I was getting some errors when baking textures using the FBX. Also, the binary FBX export was implemented later.
Not a lot of software can handle that amount of polygons, so I just stored the file and used RC’s decimation features to make a low poly version of the same model. Usually around 1M triangles. And that one can be opened in ZBrush, MeshLab or any other modeling software, where it can be retopologized and unwrapped. Depending on the model, I used different techniques for retopology. Often ZRemesher and sometimes by hand.

正如我提到的，我使用PLY格式导出疯狂密集的几何图形。 尽管Reality Capture的PLY导出器没有缩放比例和轴方向控件，但我还是更喜欢FBX，因此与FBX不同，PLY超出比例并旋转。 之所以选择处理这个问题，是因为使用FBX烘焙纹理时遇到一些错误。 同样，二进制FBX导出在稍后实现。
并不是很多软件都能处理那么多的多边形，所以我只是存储了文件，并使用了RC的抽取功能来制作同一模型的低多边形版本。 通常约1M个三角形。 可以在ZBrush，MeshLab或任何其他建模软件中打开该文件，并在其中重新拓扑和展开。 根据模型，我对拓扑使用了不同的技术。 通常是ZRemesher，有时是手工操作。

Then I used xNormal to bake textures. xNormal doesn’t seem to be bothered by the hundreds of millions of triangles and handles it with ease. I baked the diffuse texture using the vertex color info. The vertex density in the highpoly was more than enough to produce a clean and sharp texture without any interpolation between vertices. I never used RC’s integrated unwrapping and texturing features.
That being said, if for some reason your dense cloud is not dense enough, or there are some areas missing (like in the image below), projecting a texture from your photos can bring additional detail to those areas.

然后，我使用xNormal烘焙纹理。 xNormal似乎不受数亿个三角形的困扰，并且可以轻松地处理它。 我使用顶点颜色信息烘焙了漫反射纹理。 高多边形中的顶点密度足以产生干净而清晰的纹理，而顶点之间没有任何插值。 我从未使用过RC的集成式展开和纹理化功能。
就是说，如果由于某种原因您的密集云不够密集，或者缺少某些区域(如下图所示)，则从照片投射纹理可以为这些区域带来更多细节。

What most of the photogrammetry tutorials would teach you is that it is best if you avoid direct, harsh lighting and shadows when scanning an object. If it is a small rock, that you are about to capture, you can bring it in the shade or even in the studio and use softboxes and turntables. You can’t really do that with trees though, so I was watching the forecast and hoping for cloudy weather. However, even in overcast conditions, there were some shadows and ambient occlusion. This is solved with Unity’s DeLighting tool. All it takes is a normal map, a bent normal map and baked AO. It keeps the diffuse values intact while removing the shadows.

大多数摄影测量学教程将教给您的是，最好是在扫描物体时避免直射，刺眼的光线和阴影。 如果您要捕获的是一块小石头，则可以将其放在阴影下甚至在工作室中，并使用柔光箱和转盘。 但是，您不能真正使用树木来做到这一点，因此我一直在观察天气预报，并希望能有多云的天气。 但是，即使在阴天条件下，也存在一些阴影和环境光遮挡。 这可以通过Unity的DeLighting工具解决。 它所需要的只是一个法线贴图，一个弯曲的法线贴图和烘焙的AO。 在消除阴影的同时，它可以保持漫反射值不变。

The resulted assets were then imported into Unity to test the dynamic lighting and shaders.

然后将生成的资产导入Unity以测试动态照明和着色器。

There are times when it is just not possible to capture every single part of your model. Either there’s an obstacle and you can’t get all the angles. Other times you are in a hurry or your battery is dying and you miss something and you don’t realize until you get home and start processing the data. I made a lot of mistakes like that, but then I was able to salvage some of my work by using Substance Painter to clone stamp and try to fix the missing data.

有时甚至无法捕获模型的每个部分。 要么有障碍物，要么就无法获得所有角度。 在其他时间，您急着或电池快要耗尽了，您错过了一些东西，直到回家并开始处理数据时您才意识到。 我犯了很多类似的错误，但是后来我得以通过使用Substance Painter克隆图章并尝试修复丢失的数据来挽救一些工作。

实际游戏资产 (The actual game assets)

For most of the duration of the Book of the Dead production, the Demo team didn’t have an environment artist on staff and we were looking to find one. Some work was contracted out to an external environment artist, Tihomir Nyagolov, who had done the initial explorations and white boxed the environment, but the main load of the work fell on the Creative and Art Director, Veselin Efremov, and myself. Each of us would go out to our nearby forests to capture photogrammetry data, and the work naturally transitioned into producing the final game assets that were needed. I don’t have a background in environment art, and I had zero experience in dealing with game optimizations, LODs etc. At that point there were some placeholder trees already created by Tihomir with the help of GrowFx, so I took over from there, learning as I go.

在《死者之书》制作的大部分时间内，演示团队都没有一名环境艺术家在工作，我们一直在寻找一名。 某些工作被外包给了外部环境艺术家Tihomir Nyagolov，他做了初步的探索，对环境进行了白盒装，但工作的主要工作落在创意和艺术总监Veselin Efremov和我本人身上。 我们每个人都会到附近的森林中捕获摄影测量数据，然后工作自然过渡为产生所需的最终游戏资产。 我没有环境艺术方面的背景，并且在游戏优化，LOD等方面的处理经验为零。那时，Tihomir已经在GrowFx的帮助下创建了一些占位符树，所以我从那里接手了，边走边学。

GrowFX proved to be really powerful and versatile tool for creating all kinds of vegetation. It interacts with other objects in your scene so you can achieve all kinds of unique and natural looking results. It isn’t exactly built with game assets creation in mind, but it is controllable enough and can be used for the task. It is a 3DS Max plugin. I’ve been a 3DS Max user for 20+ years and I really feel at home there. Unfortunately GrowFX relies on some of the outdated 3DS Max components like the curves editing dialogs, which aren’t very convenient, but it still was a good tool for the task at hand so I just had to deal with it.

事实证明，GrowFX是创建各种植被的强大工具，用途广泛。 它与场景中的其他对象进行交互，因此您可以获得各种独特自然的效果。 它并不是完全以创建游戏资产为目的而构建的，但是它是可控制的，可以用于任务。 这是一个3DS Max插件。 我成为3DS Max用户已有20多年了，在那里我真的很宾至如归。 不幸的是，GrowFX依赖于一些过时的3DS Max组件，例如曲线编辑对话框，虽然不太方便，但是它仍然是完成当前任务的好工具，因此我只需要处理它。

The forest in Book of the Dead was intended to be primarily conifer. There are some beautiful forests and parks near my home, so I went on a “hunt” and scanned some of those. Then I proceeded with stitching my GrowFX creations onto the scanned models. The final tree trunk was composed out of scanned geometry and unique texture for the lower part stitched to a procedurally generated trunk with tileable texture for the rest of it, all the way to the top.

《死者之书》中的森林原本主要是针叶树。 我家附近有一些美丽的森林和公园，所以我进行了一次“狩猎”并扫描了其中一些。 然后，我将我的GrowFX作品缝合到扫描的模型上。 最终的树干由扫描的几何形状和独特的纹理组成，下部缝合到程序生成的树干上，其余部分一直拼接到顶部，一直到顶部。

A small patch of the bottom was clone stamped to the top of texture to make it tileable

将底部的一小块克隆克隆到纹理的顶部，使其可平铺

It is one thing to do photogrammetry on rocks and tree trunks, but scanning pine needles is a whole new deal. This is where Quixel stepped in and provided us with their beautifully scanned atlases. They collaborated with the Demo Team and did numerous small assets like grass, shrubs, debris, etc. specially created for “Book Of The Dead”.

在岩石和树干上进行摄影测量是一回事，但是扫描松针则是全新的事情。 这就是Quixel介入的地方，为我们提供了精美扫描的地图集。 他们与演示团队合作，完成了许多专门为“死者之书”创建的小资产，例如草，灌木，残骸等。

As I mentioned in the beginning, my background is in CG productions and I’ve made large forests before, using Multiscatter or Forest Pack Pro and rendering in V-ray. In such tasks, you can use the Quixel Megascans atlases as they are, but for a realtime project like Book of the Dead we needed to do some optimization. It included building larger elements (branches, treetops etc.) and arranging those into new textures, transferring the initial scanned data for the normal maps, displacement, transmission and so on.

正如我在开始时提到的，我的背景是CG制作，之前我使用Multiscatter或Forest Pack Pro制作大型森林，并在V-ray中进行渲染。 在此类任务中，您可以按原样使用Quixel Megascans地图集，但是对于像《死者之书 》这样的实时项目，我们需要进行一些优化。 它包括构建较大的元素(分支，树梢等)，并将它们排列为新的纹理，传输法线贴图的初始扫描数据，位移，传输等。

The existing Megascans normal data was slightly modified to give a fake overall volume impression.

现有的Megascans正常数据经过了稍微修改，以给人以假的总体印象。

I used different normals editing techniques such as Normal Thief and other custom built 3DSMax scripts to blend the branches with the trunk.

我使用了不同的法线编辑技术，例如“法线小偷”和其他自定义构建的3DSMax脚本，将分支与主干混合在一起。

Altering the vertex normals so that they can blend with the trunk

更改顶点法线，使其可以与躯干融合

Using this approach I was able to produce different types of pine trees.

使用这种方法，我能够生产出不同类型的松树。

风 (Wind)

We wanted the forest to feel “alive” and the wind was a crucial element for us. The trees were set up for our vertex shader based wind animation solution by our Environment Artist Julien Heijmans.
There are many different ways of creating a vector field and I looked up several different options. Being familiar with Chaosgroup’s fluid solver, PhoenixFD, I decided to see what kind of usable data I can get out of it and bring it into Unity. I was able to export the scene geometry, bring it in 3DS Max as an FBX and run some fluid through it, that swirls around the vegetation and creates the turbulent wind effect. The bigger trees were shielding the smaller vegetation and the effect there was less prominent.

我们希望森林能感觉到“活跃”，而风对我们而言至关重要。 这些树木是由我们的环境艺术家Julien Heijmans为基于顶点着色器的风动画解决方案设置的。
创建矢量场的方法有很多，我查看了几种不同的选项。 熟悉Chaosgroup的流体求解器PhoenixFD之后，我决定看看可以从中得到什么样的可用数据并将其引入Unity。 我能够导出场景几何体，将其作为FBX引入3DS Max中，并在其中流过一些流体，这些流体绕着植被旋转并产生湍动的风效果。 较大的树木遮蔽了较小的植被，其影响不那么明显。

I looped the simulated sequence using the integrated PhoenixFD playback controls.

我使用集成的PhoenixFD播放控件循环播放了模拟序列。

The vector information was then read through a PhoenixFD Texmap, normalized and plugged as a diffuse texture over the procedurally created isosurface.

然后，通过PhoenixFD Texmap读取矢量信息，将其标准化并作为漫反射纹理插入在程序创建的等值面上方。

The rendered image sequence was then imported back in Unity, where the final texture atlas was assembled. I used to do that in After Effects in the past, but now Unity has a very convenient Image Sequencer tool, that can do that pretty much automatically. It is one of the new VFX tools that is being developed by Unity’s GFX  team in Paris.

然后将渲染的图像序列导入到Unity中，在那里组装了最终的纹理图集。 我过去曾经在After Effects中这样做，但是现在Unity具有非常方便的Image Sequencer工具，该工具几乎可以自动完成。 它是Unity的GFX团队在巴黎开发的新VFX工具之一。

The created texture atlas was placed in the scene. I made a simple box to define my simulation boundaries and used that as a position reference.

创建的纹理图集已放置在场景中。 我制作了一个简单的框来定义我的模拟边界，并将其用作位置参考。

To be clear, this was an experiment that allowed us to push the visuals of some of the shots in the cinematic teaser that we showed. It’s a method that I can recommend if you are using Unity for film production. It plugs into the main procedural vertex shader based wind animation solution, which was developed for the project by our Tech Lead Torbjorn Laedre and was used in most of the scenes of the teaser, as well as for the console version of the project that we showed at GDC.

明确地说，这是一个实验，使我们能够在展示的电影预告片中推动某些镜头的视觉效果。 如果您使用Unity进行电影制作，我可以推荐这种方法。 它插入了基于程序顶点着色器的主要基于风的动画解决方案，该解决方案是由我们的技术负责人Torbjorn Laedre为该项目开发的，并用于预告片的大多数场景，以及我们展示的项目的控制台版本在GDC。

In an upcoming blog post, Julien and Torbjorn will explain more about how we handled the Wind and the final solution we adopted.

在即将发布的博客文章中，Julien和Torbjorn将解释有关我们如何处理Wind以及我们采用的最终解决方案的更多信息。

蜂巢 (The Hive)

I started to block some of the ideas about the Hive early on.

我从一开始就阻止了有关Hive的一些想法。

After the initial design, I started building various game ready elements in order to build the Unity assets.

进行初始设计之后，我开始构建各种可用于游戏的元素，以构建Unity资产。

螺丝钉 (The Screwies)

For the screwies crowd, I did some exploration for the body variations. Again I used Chaosgroup’s PhoenixFD and ran a fluid smoke simulation. Then I cut out the screwie shape and created an isosurface based on the fluid  temperature

对于那些苦苦挣扎的人群，我对身体的变化做了一些探索。 我再次使用Chaosgroup的PhoenixFD并进行了烟雾模拟。 然后我切出小螺钉形状，并根据流体温度创建等值面

Some shape exploration made with PhoenixFD

PhoenixFD进行的一些形状探索

This method allowed us to quickly preview different shapes and it was used as a general reference. The final screwie character model was created by Plamen (Paco) Tamnev and you can read all about it in his incredibly detailed blog post.

这种方法使我们可以快速预览不同的形状，并用作一般参考。 最终的螺丝钉角色模型是由Plamen(Paco)Tamnev创建的，您可以在他非常详细的博客文章中阅读有关它的全部内容。

滴液效果 (The dripping sap effect)

To achieve the dripping sap on the screwie’s face, I used PhoenixFD again. I started with making a little proof of concept showing the capabilities and what we can achieve with a dense viscous liquid.

为了获得小螺钉上滴落的汁液，我再次使用了PhoenixFD。 我首先进行了一些概念验证，以展示其功能以及使用稠密粘性液体可以实现的目标。

I was quite happy with the overall result and the fluid motion, so I proceeded with setting up the real model. The goal was to prevent the simulation from forming too many separated pieces and droplets.

我对整体结果和流体运动感到非常满意，因此我继续建立真实模型。 目的是防止模拟形成太多分离的碎片和液滴。

That allowed me to get a single frame from the generated geometry sequence, retopologize it, make UVs, and use WRAP3 to project it over the rest of the shapes in the sequence. As a result, I’ve got a series of blend shapes that use the same topology.

这样一来，我就可以从生成的几何图形序列中获得单个框架，对其重新拓扑，制作UV，然后使用WRAP3将其投影到该序列的其余形状上。 结果，我得到了一系列使用相同拓扑的混合形状。

I also tried running a sap simulation over some of the tree trunks.

我还尝试过在一些树干上进行树液模拟。

We didn’t end up using those in the final project. However, I still find it as a nice way to add some detail over the scanned models.
—

我们最终并没有在最终项目中使用它们。 但是，我仍然发现它是在扫描的模型上添加一些细节的好方法。
-

Stay tuned for the next blog post in the series. We’ll be diving further into the environment art created for Book of the Dead with Julien Heijmans.

请继续关注本系列的下一篇博客文章。 我们将 与Julien Heijmans 进一步探讨为 《死者之书》 创作的环境艺术 。

Meet us at Unite Berlin on June 19 to walk through the Book of the Dead environment on a console yourself, and attend Julien Heijmans’s presentation about Environment art in the demo. See the full schedule here.

6月19日在柏林团结公园与我们会面，亲自在控制台上浏览《死亡之书》环境，并在演示中参加朱利安·海曼斯(Julien Heijmans)关于环境艺术的演讲。 请在此处查看完整时间表 。

More information on Book of the Dead

有关死者之书的更多信息

翻译自: https://blogs.unity3d.com/2018/06/15/book-of-the-dead-photogrammetry-assets-trees-vfx/

摄影测量计算机视觉领域