Computational photography has been at the forefront of smartphone camera innovation (e.g. Google’s HDR+ and Apple’s Smart HDR). Photographs captured by modern smartphones have higher dynamic range and lower noise than the small sensors appear capable of. With the introduction of the Camera2 API in Android 5.0 it has been possible to capture RAW images from the camera, bypassing the entire post processing pipeline of the device itself.

计算摄影一直是智能手机相机创新的前沿(例如Google的HDR +和苹果的Smart HDR)。 现代智能手机拍摄的照片具有比小型传感器似乎更高的动态范围和更低的噪声。 随着Android 5.0中Camera2 API的引入，可以从相机捕获RAW图像，而绕过设备本身的整个后处理流程。

Two years ago I started a side project to replace the camera processing pipeline on Android with my own. After months of research and trial and error, the result of my efforts is my app Motion Cam (in beta, free download). The app captures images from the camera in RAW format, robustly stacks multiple images to reduce noise and recovers highlights by blending an underexposed image. Below is a high-level summary of each step in the new camera pipeline. I intend to write a more detailed description of each step in separate posts.

两年前，我开始了一个附带项目，以用我自己的项目替换Android上的相机处理管道。 经过数月的研究和反复试验，我的努力结果是我的应用程序Motion Cam(处于beta版，免费下载) 。 该应用程序以RAW格式从相机捕获图像，牢固地堆叠多张图像以减少噪点，并通过混合曝光不足的图像来恢复高光。 以下是新相机管线中每个步骤的简要概述。 我打算在单独的文章中对每个步骤进行更详细的描述。

捕获数据 (Capturing the data)

Motion Cam uses the Camera2 C++ API from the Android NDK. At startup, I allocate 512 MB of buffers to store the RAW images in the background in a circular buffer. Android typically stores the RAW images in RAW10 or RAW12 format where each pixel is either 10 or 12 bits. Most devices capture at a resolution of 12 megapixels. If the device captures in RAW10 format, each buffer is 15 MB (4000*3000*1.25). With 512 MB of buffers it is possible to store around 34 images. There is usually at least 1/30th of a second between each capture which equates to around 1.1 seconds worth of image history. The app has zero shutter lag because the shutter button simply allows the user to pick an image from this history.

Motion Cam使用Android NDK中的Camera2 C ++ API 。 在启动时，我分配了512 MB的缓冲区，以将RAW图像在后台存储在循环缓冲区中。 Android通常以RAW10或RAW12格式存储RAW图像，其中每个像素为10位或12位。 大多数设备以12兆像素的分辨率捕获。 如果设备以RAW10格式捕获，则每个缓冲区为15 MB(4000 * 3000 * 1.25)。 借助512 MB的缓冲区，可以存储大约34张图像。 每次捕获之间通常至少有1/30秒的时间，相当于大约1.1秒的图像历史记录。 该应用具有零快门滞后，因为快门按钮仅允许用户从该历史记录中选择图像。

Motion Cam sets up the camera to capture an underexposed image every 5 frames. The purpose of this underexposed frame is to recover the missing highlights. Below is a sample of 5 captured frames:

Motion Cam将相机设置为每5帧捕获曝光不足的图像。 此曝光不足的框的目的是恢复丢失的高光。 以下是5个捕获帧的示例：

The fifth frame is much darker than the rest. The sky is not visible in the first four frames but can be seen in the underexposed frame. Below I’ll describe how the images are blended to recover the blown highlights.

第五帧比其余的要暗得多。 在前四个帧中看不到天空，但在曝光不足的帧中可以看到天空。 下面我将描述如何融合图像以恢复高光。

合并影像以减少杂讯 (Noise reduction by merging images)

The denoising algorithm uses the RAW images as input. Motion Cam treats the bayer RAW data as four colour channels (red, blue and two green channels). It starts by creating an optical flow map between a set of images and the reference image utilising Fast Optical Flow using Dense Inverse Search. Then, each colour channel is transformed into the wavelet domain using a dual tree wavelet transform. The wavelet coefficients are fused with the low pass subband acting as a guide to minimize artifacts due to errors in the optical flow map from occlusion or alignment failure.

去噪算法使用RAW图像作为输入。 Motion Cam将Bayer RAW数据视为四个颜色通道(红色，蓝色和两个绿色通道)。 首先，通过使用密集逆搜索的快速光流，在一组图像和参考图像之间创建光流图 。 然后，使用双树小波变换将每个颜色通道转换到小波域。 将小波系数与低通子带融合在一起，以最小化由于阻塞或对准失败引起的光流图中的错误而导致的伪影。

The amount of noise present in the image is used to determine how many RAW images are merged together. It is estimated from the high frequency subband of the wavelet transform. A well-lit scene may not need much noise reduction whereas a low light scene will have greater noise and require more images. Below is an example of a low light scene and the result of merging 16 images:

图像中存在的噪波量用于确定将多少个RAW图像合并在一起。 根据小波变换的高频子带进行估计。 光线充足的场景可能不需要太多的降噪，而光线较暗的场景会产生更大的噪点并需要更多图像。 下面是一个昏暗场景的示例以及合并16张图像的结果：

In addition to stacking multiple images (temporal noise reduction), the algorithm will also shrink the wavelet coefficients to further reduce noise (spatial noise reduction).

除了堆叠多个图像(时间噪声降低)之外，该算法还将缩小小波系数以进一步降低噪声(空间噪声降低)。

将RAW数据转换为图像 (Transforming the RAW data into an image)

Most modern cameras use a bayer filter. This means the RAW image is subsampled and consists of 25% red, 25% blue and 50% green pixels. There are more green pixels because human vision is most sensitive to green light. The output from the denoising algorithm is demosaiced and colour corrected into an sRGB image. There are many different demoisaicing algorithms. After experimenting with a few different methods I settled on the algorithm Color filter array demosaicking: New method and performance measures by Lu and Tan (2003). Alternative algorithms exist but the low computational complexity and implementation simplicity was the deciding factor for me. The output of the demosaic step is shown below:

大多数现代相机都使用拜耳滤镜 。 这意味着RAW图像被二次采样，并且由25％的红色，25％的蓝色和50％的绿色像素组成。 绿色像素更多，因为人类视觉对绿光最敏感。 去噪算法的输出被去马赛克，并将颜色校正为sRGB图像。 有许多不同的去马赛克算法。 在尝试了几种不同的方法之后，我开始研究算法滤色镜阵列的去马赛克：Lu和Tan(2003)提出的新方法和性能指标 。 存在替代算法，但是低计算复杂度和实现简单性是我的决定因素。 去马赛克步骤的输出如下所示：

Without any further post processing the initial output from the camera would not be considered a good photo.

没有任何进一步的后期处理，相机的初始输出将不被视为好照片。

固定亮点 (Fixing the highlights)

In the image above, the sky is white and missing detail. The reason for that is the camera sensor was not capable of capturing the full dynamic range of this scene. If you recall above, one of the images that is captured is an underexposed image. It is used to recover the highlights. The algorithm starts by matching the exposure of the reference image to the underexposed image. The two images are then aligned to each other. The brightest parts of the underexposed image are blended to the reference with a check for structural consistency. If there is too much motion between the two images, they are not blended to avoid introducing artifacts into the final output. Below is the mask used to blend in the highlights into the output image:

在上图中，天空是白色且缺少细节。 原因是相机传感器无法捕获该场景的全部动态范围。 如果您还记得上面的内容，则捕获的图像之一是曝光不足的图像。 用于恢复高光。 该算法首先将参考图像的曝光与曝光不足的图像进行匹配。 然后将两个图像彼此对齐。 曝光不足图像的最亮部分会与参考混合，并检查结构一致性。 如果两个图像之间的运动过多，则不会将它们融合在一起，以避免将伪像引入最终输出中。 以下是用于将高光混合到输出图像中的蒙版：

It is not necessary to perform noise reduction on the underexposed image because only well exposed pixels are blended and they are not noisy.

无需对曝光不足的图像进行降噪处理，因为仅混合了曝光良好的像素且它们没有噪点。

色调映射 (Tone mapping)

At this point the reference image matches the exposure of the underexposed image. It is very dark and would benefit from dynamic range compression. I settled on the exposure fusion algorithm shared by Google’s HDR+. It produces pleasing and natural images. The algorithm blends multiple different exposures to produce an HDR image. Instead of capturing multiple exposures, I artificially generate the overexposed image and use the image generated above as the second exposure to the algorithm. The shadows slider in the app controls the overexposed image. The output of tone mapping is shown below:

此时，参考图像与曝光不足图像的曝光匹配。 它非常暗，可以从动态范围压缩中受益。 我选择了Google HDR +共享的曝光融合算法 。 它产生令人愉悦的自然图像。 该算法混合了多种不同的曝光以生成HDR图像。 我没有捕获多次曝光，而是人为地生成了曝光过度的图像，并将上面生成的图像用作算法的第二次曝光。 应用程序中的阴影滑块可控制曝光过度的图像。 音调映射的输出如下所示：

锐化 (Sharpening)

The next step in the pipeline is to sharpen the output after tone mapping. After some experimentation I ended up using a guided filter to enhance the details of the image. It is computationally efficient and produces pleasing results. The output is shown below:

流水线的下一步是在色调映射后提高输出的清晰度。 经过一些实验，我最终使用了导向滤镜来增强图像的细节。 它计算效率高，并产生令人愉悦的结果。 输出如下所示：

整理起来 (Finishing up)

The final stage of the pipeline is to increase the contrast, saturation and gamma correct the image. I use a simple S-curve to boost the contrast. In the output below I’ve also increased the saturation of the blues and greens by converting the image to the HSV colour space.

流水线的最后一步是增加对比度，饱和度和伽玛校正图像。 我使用一个简单的S曲线来增强对比度。 在下面的输出中，我还通过将图像转换为HSV颜色空间来增加了蓝色和绿色的饱和度。

结论 (Conclusion)

One of the most interesting aspects of this project is just how much processing is required to get a visually pleasing photograph out of a smartphone camera. There are hundreds of decisions made by the manufacturer that end up impacting the style of the final output. In Motion Cam I largely avoid that issue by giving the user the controls to adjust the image after pressing the shutter button. I hope you’ve found this blog post informative. I intend to expand on each step of the pipeline in more detail at a later time.

该项目最有趣的方面之一就是需要多少处理才能从智能手机相机中获得视觉上令人愉悦的照片。 制造商做出的数百个决定最终会影响最终输出的样式。 在Motion Cam中 ，通过在按下快门按钮后为用户提供调整图像的控件，我在很大程度上避免了该问题。 希望您发现此博客文章能为您提供帮助。 我打算稍后再详细介绍管道的每个步骤。

Please try out Motion Cam here:

请在此处试用Motion Cam：

https://play.google.com/store/apps/details?id=com.motioncam

https://play.google.com/store/apps/details?id=com.motioncam