视频插帧--Video Frame Interpolation via Adaptive Convolution

Video Frame Interpolation via Adaptive Convolution
CVPR2017
http://web.cecs.pdx.edu/~fliu/project/adaconv/

本文使用CNN网络完成 frame interpolation，这里我们将像素插值问题看作对相邻两帧中相应图像块的卷积，通过一个全卷积CNN网络来估计 spatially-adaptive convolutional kernel，这些核捕获运动信息和插值系数， capture both the motion and interpolation coefficients, and uses these kernels to directly convolve with input images to synthesize a middle video frame.，如下图所示：

Given two video frames I1and I2, our method aims to interpolate a frame ˆI temporally in the middle of the two input frames

3 Video Frame Interpolation
传统的帧插值方法是 two-step approach： first estimates motion between two frames and then interpolates the pixel color based on the motion
但是光流的计算很容易不稳定
optical flow is not reliable due to occlusion, motion blur, and lack of texture

我们这里的策略是 Interpolation by convolution

将 pixel interpolation 表示为 convolution 的优点：
1）将运动估计和像素合成变为一个步骤可以提高方法的鲁棒性 provides a more robust solution
2）卷积核对一些困难的情况提供了灵活性 the convolution kernel provides flexibility to account for and address difficult cases like occlusion
3）一旦得到卷积核，可以无缝接入advanced re-sampling techniques

3.1. Convolution kernel estimation 卷积核估计
这里我们使用一个 CNN 网络来 estimate a proper convolutional kernel to synthesize each output pixel in the interpolated images.

In our implementation, the default receptive field size is 79 × 79 pixels. The convolution patch size is 41×41 and the kernel size is 41 × 82 as it is used to convolve with two patches

Loss function
这里我们分别设计了 color loss 和 gradient loss，最终的损失函数是 combine the above color and gradient loss as our final loss

4 Experiments
Qualitative evaluation on blurry videos

Evaluation on the Middlebury testing set

Qualitative evaluation on video with abrupt brightness change

Qualitative evaluation with respect to occlusion

On a single Nvidia Titan X, this implementation takes about 2.8 seconds with 3.5 gigabytes of memory for a 640 × 480 image, and 9.1 seconds with 4.7 gigabytes for 1280×720, and 21.6 seconds with 6.8 gigabytes for 1920 × 1080.

视频插帧--Video Frame Interpolation via Adaptive Convolution相关推荐

视频插值--Video Frame Interpolation via Adaptive Separable Convolution
Video Frame Interpolation via Adaptive Separable Convolution ICCV2017 https://github.com/sniklaus/py ...
RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation——精读笔记
RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation 原文:https://arxiv.org/pdf/ ...
视频插帧（Video Frame interpolation）论文及源码
视频插帧论文列表(含Code) 蓝色突出表示热门方法 CCF A AAAI 2019 CyclicGen:Deep Video Frame Interpolation Using Cyclic Fra ...
干货 | 视频插帧的方案实现与对比~~
对于视频网站.电视厂商以及进行视频压制的用户来说,改变视频的帧率算是一个比较常见的需求. 视频网站改变帧率主要是为了向不同级别的网站用户提供差异化服务:电视厂商则是以提供更好的显示效果作为电视的卖点: ...
详解插帧算法DAIN论文和代码(Depth-Aware Video Frame Interpolation)
Project : https://sites.google.com/view/wenbobao/dain Paper: https://arxiv.org/abs/1904.00830 Code: ...
论文阅读笔记（1）Beyond Natural Motion: Exploring Discontinuity for Video Frame Interpolation——超越自然运动：探索视频帧
论文:Beyond Natural Motion: Exploring Discontinuity for Video Frame Interpolation 会议:2022CVPR February ...
2019-CVPR-上交-(DAIN)Depth-Aware Video Frame Interpolation
可以把30fps的进一步插帧到480fps(16倍插帧) 英伟达的Super SloMo只用在了真实拍摄的视频上,而这项研究却可以扩展到常见的任何类型视频:电影.定格动画. 动画卡通等等. DAIN的 ...
视频插帧—学习笔记（算法+配置+云服务+Google-Colab）
恰好碰到同学项目需要,了解了一下关于利用深度学习视频插帧的相关知识,在这里做一个简单的记录. 目录一.方法+论文 1.DAIN (Depth-Aware Video Frame Interpolat ...
CVPR2019｜Depth-Aware Video Frame Interpolation【论文阅读笔记】
CVPR2019|Depth-Aware Video Frame Interpolation[论文阅读笔记] 作者相关链接 1.前言 2.介绍 3.算法 4.实验作者 Wenbo Bao, Wei ...

视频插帧--Video Frame Interpolation via Adaptive Convolution

视频插帧--Video Frame Interpolation via Adaptive Convolution相关推荐

最新文章

热门文章