文章题目：

FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors

文章地址：FSRNet: End-to-End Learning Face Super-Resolution With Facial Priors (thecvf.com)

项目地址：github.com

Abstract

我们提出了一种新型的深端到端可训练人脸超分辨率网络（FSRNet），它利用了几何先验知识，即。facial landmark heatmaps和parsing maps，用于超分辨率极低分辨率（LR）人脸图像，无需对齐要求。

We present a novel deep end-to-end trainable Face Super-Resolution Network (FSRNet), which makes use of the geometry prior, i.e., facial landmark heatmaps and parsing maps, to superresolve very low-resolution (LR) face images without wellaligned requirement.

具体来说，我们首先构造一个粗略的SR网络来恢复粗略的高分辨率（HR）图像。然后，粗HR图像被发送到两个分支：一个精细SR编码器和一个先验信息估计网络，该网络提取图像特征，并分别估计黑点热图/解析图。图像特征和先验信息都被发送到精细SR解码器以恢复HR图像。为了生成真实的人脸，我们还提出了人脸超分辨率生成对抗网络（FSRGAN），将对抗性损失纳入FSRNet。

Specifically, we first construct a coarse SR network to recover a coarse high-resolution (HR) image. Then, the coarse HR image is sent to two branches: a fine SR encoder and a prior information estimation network, whichextractstheimagefeatures, andestimateslandmark heatmaps/parsing maps respectively. Both image features and prior information are sent to a fine SR decoder to recover the HR image. To generate realistic faces, we also propose the Face Super-Resolution Generative Adversarial Network (FSRGAN) to incorporate the adversarial loss into FSRNet.

此外，我们引入了两个相关的任务，人脸对齐和解析，作为新的人脸SR评估指标，解决了经典指标w.r.t.视觉感知。大量实验表明，FSRNet和FSRGAN在定量和定性方面都显著优于最新的LR生成的SR。

Further, we introduce two related tasks, face alignment and parsing, as the new evaluation metrics for face SR, which address the inconsistency of classic metrics w.r.t. visual perception. Extensive experiments show that FSRNet and FSRGAN significantly outperforms state of the arts for very LR face SR, both quantitatively and qualitatively.

1.introduce

本文基于深度卷积神经网络（CNN），提出了一种新的端到端可训练人脸超分辨率网络（FSRNet），该网络在训练过程中估计facial landmark heatmaps和parsing maps，然后利用这些先验信息更好地超分辨率非常低的人脸图像。

Based on deep Convolutional Neural Network (CNN), in this work, we propose a novelend-to-end trainable Face Super-Resolution Network (FSRNet), which estimates facial landmark heatmaps and parsing maps during training, and then uses these prior information to better super-resolve very LR face images.

下表是与以前最先进的超分辨率方法的比较：

总结：

第一个提出用人脸几何先验的知识进行端到端学习的人脸超分辨率方法。
同时引入了两种几何先验，facial landmark heatmaps 和parsing maps。
提出的FSRnet在模糊未对齐和非常低的分辨率的图像，通过8倍放大，是目前最好的水平。同时用FSRnetGAN网络可以进一步生成更加逼真的images。
采用人脸对齐和解析作为新的人脸超分辨率评价指标。进一步证明，该方法可以解决传统的视觉感知度量方法的不一致性。

3. Face Super-Resolution Network

3.1.Overview of FSRnet

我们的基本FSRNet由四部分组成：粗SR网络、精细SR编码器、先验估计网络和最终的精细SR解码器。表示X为低分辨率输入图像，Y和P为恢复的高分辨率图像，并通过FSRNet估计先验信息。

Our basic FSRNet F consists of four parts:coarse SR network,fine SR encoder,prior estimation networkand finally afine SR decoder. Denotexas the low-resolution input image,Y and P as the recovered high-resolution image and estimated prior information by FSRNet.

1.极低分辨率的输入图像对于先验估计可能过于模糊，我们首先构造coarse SR网络来恢复coarse SR图像：

2.将coarse SR images送入特征提取和先验估计网络中：

将f,p送入解码器网络去恢复SR images.

FSRnet的损失函数：

Θ表示参数集，α和β是粗略SR损失和先验损失的权重，y（i），p（i）分别是恢复的HR图像和估计的第i幅图像的先验信息。

x:低分辨率图像
y~:低分辨率图像对应的真实的高分辨率图像
p~：真实的图像对应的真实的先验信息

3.2. Details inside FSRNet

它由一个coarse SR网络和一个fine SR网络组成。

其中fine SR包括prior estimation network, fine SR encoder and fine SR decoder.

We now present the details of our FSRNet, which consists of a coarse and a fine SR network, where the fine SR network contains three parts: a prior estimation network, a fine SR encoder and a fine SR decoder.

整个流程：

Coarse SR network：

粗SR网络的结构如图2所示。它以3×3的卷积层开始，然后是3个剩余块。然后再利用另一个3×3的卷积层对粗HR图像进行重建。

It starts with a 3×3 convolution followed by 3 residual blocks. Then another 3×3 convolutional layer is used to reconstruct the coarse HR image.

在随后的精细SR网络中，粗HR图像被发送到两个分支，先验估计网络和精细编码器网络，以分别估计人脸先验值和提取特征。然后，解码器联合使用两个分支的结果来恢复精细的HR图像。

in the following fine SR network, the coarse HR image is sent to two branches, prior estimation network and fine encoder network, to estimate facial priors and extract features, respectively. Then the decoder jointly uses results of both branches to recover the fine HR image.

Prior Estimation Network：

我们采用沙漏（HG）结构来估计我们先验估计网络中的facial landmark heatmaps 和 parsing maps。

we adopt the HourGlass (HG) structure to estimate facial landmark heatmaps and parsing maps in our prior estimation network.

为了有效地整合跨比例的特征并保留不同比例的空间信息，沙漏块在对称层之间使用了跳过连接机制。随后使用1×1卷积层对获得的特征进行后处理。最后，将共享沙漏特征连接到两个分离的1×1卷积层，生成landmark heatmaps 和 the parsing maps。

To effectively consolidate features across scales and preserve spatial information in different scales, the hourglass block uses a skip connection mechanism between symmetrical layers. An1×1 convolution layer follows to post-process the obtained features. Finally, the shared hourglass feature is connected to two separate 1×1 convolution layers to generate the landmark heatmaps and the parsing maps.

Fine SR Encoder：

我们利用剩余块进行特征提取。考虑到计算成本，我们的先验特征的大小被下采样到64×64。为了使特征尺寸一致，精细SR编码器从一个3×3卷积层（步幅2）开始，将特征映射向下采样到64×64。然后利用ResNet结构提取图像特征。

we utilize the residual blocks for feature extraction. Considering the computation cost, the size of our prior features is down-sampled to 64×64. To make the feature size consistent, the fine SR encoder starts with a 3×3 convolutional layer of stride 2 to down-sample the feature map to 64×64. Then the ResNet structure is utilized to extract image features.

Fine SR Decoder：

精细SR解码器联合使用特征和先验来恢复最终精细HR图像。首先，将先前特征和图像特征串接为解码器的输入。然后，一个3×3卷积层将特征图的数量减少到64个。利用4×4反卷积层对特征图进行上采样，使其尺寸达到128×128。然后使用3个剩余块对特征进行解码。最后，使用3×3卷积层恢复精细HR图像。

The fine SR decoder jointly uses the features and priors to recover the final fine HR image. First, the prior featurepand image featurefare concatenated as the input of the decoder. Then a 3×3 convolutional layer reduces the number of feature maps to 64. A 4×4 deconvolutional layer is utilized to up-sample the feature map to size 128×128. Then 3 residual blocks are used to decode the features. Finally, a 3×3 convolutional layer is used to recover the fine HR image.

3.3. FSRGAN

其核心思想是利用判别网络来区分超分辨率图像和真实的高分辨率图像，并训练SR网络来欺骗鉴别器。

The key idea is to use a discriminative network to distinguish the super-resolved images and the real high-resolution images, and to train the SR network to deceive the discriminator.

对抗网络的目标函数（对抗性损失）：

C输出输入为真实的概率，E是概率分布的期望值。

感知损失：

φ表示固定的预训练VGG模型，并将图像Y/Y~映射到特征空间。

FSRGAN的最终目标函数：

γC和γP分别是GAN和感知损失的权重。

4. Prior Knowledge for Face Super-Resolution、

作者回答了两个问题：

（1）面部先验知识真的对面部超分辨率有用吗？

（2）不同的面部先验知识能带来多大的改善？

把先验信息估计网络移除以后，构建了一个 Baseline 网络。基于 Baseline 网络，引入 ground truth 人脸先验信息（landmark heatmap 和解析图）到拼接层，得到一个新的网络。

结论：

解析图比 landmark heatmap 含有更多人脸图像超分辨的信息，带来的提升更大；
全局的解析图比局部的解析图更有用；
landmark 数量增加所带来的提升很小

FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors相关推荐

论文翻译：2019_Speech Super Resolution Generative Adversarial Network
博客作者:凌逆战论文地址:基于GAN的音频超分辨率博客地址:https://www.cnblogs.com/LXP-Never/p/10874993.html 论文作者:Sefik Emre Es ...
论文翻译：Speech Super Resolution Generative Adversarial Network
博客作者:凌逆战论文地址:https://ieeexplore.ieee.org/document/8682215 博客地址:https://www.cnblogs.com/LXP-Never/p/ ...
漫谈深度学习在Super Resolution（超分辨率）领域上的应用
1.前言清晨,师兄推荐给我一篇文章,关于利用DeepLearning思想进行图像超分辨恢复的.超分辨这个话题几年之前还是比较火爆的,无论是BiCube.SP.A*都给出了令人振奋的结果.但是细节恢复 ...
Google Pixel 超分辨率--Super Resolution Zoom
Google Pixel 超分辨率–Super Resolution Zoom Google 的Super Res Zoom技术,主要用于在zoom时增强画面细节以及提升在夜景下的效果. 文章的主要贡 ...
CV之SR：超分辨率(Super resolution)的简介、使用方法、案例应用之详细攻略
CV之SR:超分辨率(Super resolution)的简介.使用方法.案例应用之详细攻略目录超分辨率(Super resolution)的简介超分辨率(Super resolution)的使 ...
Chapter7-7_Deep Learning for Coreference Resolution
文章目录 1 什么是coreference resolution 2 框架 2.1 Mention Detection 2.2 Mention Pair Detection 2.3 End-to-En ...
【Super Resolution】超分辨率——SRCNN
SRCNN 01 闲聊--图像的超分辨率 02 SRCNN--超分和DL的结合 02-1 双三次插值 02-2 SRCNN的网络结构 02-3 Training 训练阶段 03 EXPERIMENTS ...
Wavelet-SRNet: A Wavelet-based CNN for Multi-scale Face Super Resolution
Wavelet-SRNet: A Wavelet-based CNN for Multi-scale Face Super Resolution 2017 ICCV 1.引言 2.网络结构 3.损失函 ...
Unfolding the Alternating Optimization for Blind Super Resolution
Unfolding the Alternating Optimization for Blind Super Resolution 论文信息 Paper: [NeurIPS2020] Unfoldin ...
(NIPS2020)Unfolding the Alternating Optimization for Blind Super Resolution 笔记
(NIPS2020)Unfolding the Alternating Optimization for Blind Super Resolution https://github.com/great ...