A New Hope for the deleted scenes

删除场景的新希望

I’m a huge Star Wars fan. And like a lot of Star Wars fans I’ve been getting into Star Wars: The Clone Wars on Cartoon Network and Disney+. It’s a phenomenal show.

我是《星球大战》的忠实粉丝。 就像许多《星球大战》的粉丝一样，我也一直参与《星球大战：卡通网络和迪士尼+上的克隆人战争 》。 这是一个了不起的表演。

But I’m always annoyed by the drop in video quality when I watch the older stuff. For example, here are the deleted scenes from Star Wars: Episode IV: A New Hope (1977). This was the very first Star Wars to be created.

但是当我观看较旧的视频时，我总是对视频质量的下降感到恼火。 例如，以下是《星球大战：第四集：新希望》(1977)中已删除的场景。 这是第一个创建的《星球大战》。

There’s these weird black specs that keep popping up. Small wonder why these are the deleted scenes.

这些奇怪的黑色规格不断涌现。 难怪为什么这些都是被删除的场景。

Apparently those weird specs are called cue marks. They’re marks that come from scratches on film. Star Wars is a fantastic series, but it’s also fantastically old.

显然那些怪异的规格被称为提示标记 。 它们是来自胶片划痕的标记。 星球大战是一个梦幻般的系列，但它也充满了想象老了 。

Deep Learning has recently been used for video restoration. The results have been very promising. Deoldify for example, allows users to colorize old videos and images. NVIDIA’s Noise2Noise model allows people to restore old images to their former glory.

深度学习最近已用于视频还原。 结果非常有希望。 例如， Deoldify允许用户为旧视频和图像着色。 NVIDIA的Noise2Noise模型使人们可以将旧图像还原到以前的辉煌。

But so far there’s nothing I know of that can specifically remove ‘cue marks’ and grainy spots from old film. So let’s build it!

但是到目前为止，我所知还没有什么能从旧电影中特别去除“提示标记”和颗粒状斑点。 因此，让我们来构建它！

创建数据集 (Creating the Dataset)

Creating the dataset was tricky — but still doable. Here’s what I did. I downloaded high quality videos into from youtube. Then I ruined them. I added black specs and reduced the resolution of the video. Ffmpeg was very useful in doing this.

创建数据集很棘手，但仍然可行。 这就是我所做的。 我从youtube下载了高质量的视频。 然后我毁了他们。 我添加了黑色规格并降低了视频的分辨率。 Ffmpeg在执行此操作时非常有用。

First we’ll download the video.

首先，我们将下载视频。

youtube-dl --format best -o seinfeld.mp4 https://www.youtube.com/watch?v=nEAO60ON7yo

I’m using this video. I’m using a clip from Seinfeld. Cause why not?

我正在使用这部影片。 我正在使用Seinfeld的剪辑。 原因何在？

Then we’ll need to ruin it. To do this I downloaded a grainy film overlay from youtube. Then I overlayed the video using ffmpeg with the blend setting set to softlight. Finding the right blend setting took a lot of trial and error. The ffmpeg docs don’t have a lot of examples.

然后，我们需要破坏它。 为此，我从youtube下载了颗粒状的胶片覆盖层。 然后，我将ffmpeg的视频叠加到了混合设置为柔光的地方。 找到正确的混合设置需要大量的反复试验。 ffmpeg 文档没有很多示例。

Now we have two videos. One in perfect quality and another in crappy quality.

现在我们有两个视频。 一个以完美的质量而另一个以cr脚的质量。

Now we’ll extract frames from each video. Initially, I adopted a naive approach for doing this. Where I would do through the video in python and scrape each frame individually. But that took too long. I eventually realised we can use multi-processing here to really speed things up. This was adapted from Hayden Faulker’s script.

现在，我们将从每个视频中提取帧。 最初，我采用了幼稚的方法。 我将在python中通过视频进行操作并分别刮取每个帧。 但这花了太长时间。 我最终意识到我们可以在这里使用多处理来真正加快速度。 这是改编自海登·福克(Hayden Faulker)的脚本的。

Great. Now we have two datasets. One of crappy quality images (taken from the ruined video) and one of good quality images (taken from the high quality video). To make the crappy images crappier, I’ll downscale them (this isn’t a necessary step though).

大。 现在我们有两个数据集。 一张质量低劣的图像(从损坏的视频中获取)和一张高质量的图像(从高质量的视频中获取)。 为了缩小图像，我会缩小图像的尺寸(但这不是必需的步骤)。

def resize_one(img, size):   targ_sz = resize_to(img, size, use_min = True)   img = img.resize(targ_sz, resample =  PIL.Image.BILINEAR).convert('RGB')   return img

This is what the crappy and normal images looked like now. Side note: this is a great scene from seinfeld.

这就是现在的the脚和正常图像。 旁注：这是塞恩费尔德的绝佳风景。

A quick check shows that we have a dataset of about 10014 files. Pretty good.

快速检查表明，我们有大约10014文件的数据集。 非常好。

神经网络 (Neural Network)

Let’s make the most of those 10014 files by using transforms.

让我们通过使用转换来充分利用这10014文件。

I added horizontal and vertical flips, zoom changes, lighting changes and rotation changes. With Fastai this is really easy to do.

我添加了水平和垂直翻转，缩放更改，照明更改和旋转更改。 使用Fastai，这真的很容易做到。

Here are some of the image transforms.

这是一些图像变换。

Not bad!

不错！

We’ll use the NoGAN network pioneered by fastai and jason antic on this data. This code was inspired by lesson 7 of the fastai course.

我们将在此数据上使用由fastai和jason antic率先开发的NoGAN网络 。 该代码的灵感来自fastai课程的第7课。

I trained the model on google colab’s free gpus. They’re a great resource and I can’t believe they are free.

我在Google colab的免费GPU上训练了模型。 它们是一种巨大的资源，我无法相信它们是免费的。

训练 (Training)

The interesting thing that fastai recommends is increasing the size of your images gradually.

fastai 推荐的有趣的事情是逐渐增加图像的大小。

So at first, you train on small size images, then you upscale your images and retrain on the larger images. It saves you a lot of time. Pretty smart.

因此，首先，您在小尺寸图像上进行训练，然后放大图像，然后在较大图像上进行重新训练。 它为您节省了大量时间。 很聪明

First, we’ll train on images of size 128x128. Because the images are so small I can up the batch size to 64.

首先，我们将训练尺寸为128x128的图像。 由于图像太小，我可以将批处理大小增加到64。

I picked a learning rate of 1e-2 for this. I wanted something aggressive, but still on the safe side of explosion. This has been shown to be very useful.

为此，我选择了1e-2的学习率。 我想要一些具有侵略性的东西，但仍然可以放心。 已经证明这是非常有用的。

The network will print the results during training. The input is on the left, the prediction in the middle and the target on the right. The results look very promising!

网络将在训练过程中打印结果。 输入在左边，预测在中间，目标在右边。 结果看起来很有希望！

I resized and trained again. And again. Every time I make the resize slightly larger than it was previously. I moved from 128x128 to 480x480 to the original size of the video frames.

我调整了大小并再次训练。 然后再次。 每次我调整大小都比以前大一些时。 我从128x128移到了480x480的原始视频帧大小。

This was the final train. For this I used pct_start = 0.3. I wanted to learning rate to reduce 70% of the time during training. I prefer a lower learning rate when fine tuning models. The results from this piece of training look really good.

这是最后一班火车。 为此，我使用pct_start = 0.3 。 我想提高学习率，以减少培训期间70％的时间。 微调模型时，我更喜欢较低的学习率。 这一训练的结果看起来非常好。

推论：适用于《星球大战》 (Inference: Applying to Star Wars)

Once this network had trained, I ran inference. This was more involved than I originally thought.

一旦该网络训练完毕，我就进行推断。 这比我最初想象的要复杂得多。

I had to download the Star Wars deleted scenes (using youtube-dl) and then extract all the frames in this video. I extracted the frames using the same method previously.

我必须下载“星球大战”删除的场景(使用youtube-dl )，然后提取该视频中的所有帧。 我以前使用相同的方法提取了帧。

Then I had to run inference from the learner on each individual frame of the video. That takes a long time.

然后，我不得不在视频的每个单独的帧上从学习者那里进行推断。 这需要很长时间。

I added some hacks here.

我在这里添加了一些技巧。

First, I added a render factor. This was taken from Deoldify. The idea is that I downscale the image and convert it to a square. Then I run inference on that image. The model is more receptive to images that are square shaped. This has been shown to reduce ‘glitches’ considerably.

首先，我添加了渲染因子。 这取自Deoldify 。 这个想法是我缩小图像并将其转换为正方形。 然后，我对该图像进行推断。 该模型更易于接收正方形图像。 事实证明 ，这可以大大减少“毛刺”。

After running inference on the square shaped image I convert it back to its original shape. I found this to reduce glitches and generally result in a smoother video output. I set the render_factor to 40, although it can be higher if we want higher res output. I may need a larger RAM for that though.

在对方形图像进行推断后，我将其转换回其原始形状。 我发现这可以减少毛刺，通常可以使视频输出更平滑。 我将render_factor设置为40 ，尽管如果我们想要更高的res输出，它可以更高。 不过，我可能需要更大的RAM。

Second, I adjust brightness. This isn’t really a hack. Seems like more of a mistake that I’m correctly manually. For some reason, the model inference results in images that are very low in brightness.

其次，我调整亮度。 这并不是真正的hack。 似乎更像是我正确地手动执行的错误。 由于某种原因，模型推断会导致图像的亮度非常低。

I suspect it’s something to with the softlight filter we used for ffmpeg earlier. But I'm having to manually correct that here. I'll need to look into this further.

我怀疑这与我们之前用于ffmpeg的softlight滤镜有关。 但是我必须在这里手动更正。 我需要进一步研究。

Third, I’m using matplotlib’s save functionality. I found fastai’s save image functionality to give me very weird results (Luke’s clothes were fluroscent blue and red). But strangely, matplotlib’s save functionality gives me okay results. I’ll need to look into this. I suspect that I may be losing quality on the image because I’m using matplotlib’s savefig functionality.

第三，我正在使用matplotlib的保存功能。 我发现fastai的保存图像功能给我非常奇怪的效果(Luke的衣服是柔软的蓝色和红色)。 但是奇怪的是，matplotlib的保存功能给了我不错的结果。 我需要调查一下。 我怀疑由于使用了matplotlib的savefig功能，可能会降低图像质量。

Here’s some of the outputs from the model.

这是模型的一些输出。

Then I had to stitch all these frames together to create a video. To do that I initially used ffmpeg but I ended up overloading my RAM. Instead, I used opencv2’s VideoWriter.

然后，我不得不将所有这些帧缝合在一起以创建视频。 为此，我最初使用ffmpeg，但最终导致RAM超载。 相反，我使用了opencv2的VideoWriter 。

Here is the final output.

这是最终的输出。

And the original video

和原始视频

改进之处 (Improvements)

1. The sky needs a bit more work. But I like the vibrancy of the background. That is an interesting (and completely unplanned) effect. The goal was to remove the ‘cue marks’ (annoying black specs) from the video. I think it’s done okay in that respect — but there’s still more to do.

   天空需要更多的工作。 但是我喜欢背景的活力。 这是一个有趣的(而且完全是计划外的)效果。 目的是从视频中删除“提示标记”(讨厌黑标)。 在这方面，我认为这样做还不错，但还有更多工作要做。

   I like how the network has intensified the sun though. It completely changes the the scene between Luke and Biggs when Biggs says he’s joining the rebellion.

   我喜欢网络如何增强阳光的作用。 当Biggs说他加入叛乱时，这完全改变了Luke和Biggs之间的场景。

2. There’s a weird horizontal bar line that shows up around the 22 second mark. I didn't add any horizontal bars in the training set so it's completely understandable that the network didn't remove that at all. But in the future, I'll need to add more horizontal bars to my training set to fix these.

2.在22秒标记附近出现一条奇怪的水平线。 我没有在训练集中添加任何单杠，因此完全不删除网络完全是可以理解的。 但是将来，我需要在训练集中添加更多的单杠以解决这些问题。

3. I’m also thinking of doing more super-resolution on the video. It would be nice to show a young Luke Skywalker in high quality. To do that I could resize the images before training further. I’ve already downscaled the image, but potentially I could downscale it further.Alternatively, to achieve superres I could potentially use a ready-made upscaler such as VapourSynth. This is probably the best option as the original video is already in poor quality.

3.我也在考虑在视频上做更多的超分辨率。 高质量地展示年轻的卢克·天行者很高兴。 为此，我可以在进一步训练之前调整图像的大小。 我已经缩小了图像的尺寸，但可能会进一步缩小图像。或者，要获得超分辨率，我可以使用现成的升频器，例如VapourSynth 。 这可能是最好的选择，因为原始视频的质量已经很差。

4. Inference is also an issue. It tends to overload memory and crash. The result is that 42 seconds is the longest I could get for this video. I'm not completely sure how to solve this problem. But I'll need to solve it if I'm going to be using this further.

4.推论也是一个问题。 它往往使内存过载并崩溃。 结果是42秒是我能获得的最长视频。 我不确定如何解决此问题。 但是，如果我要进一步使用它，则需要解决它。

So much to do!

需要做的事情还很多！

Originally published at https://spiyer99.github.io on June 19, 2020.

最初于 2020年6月19日 发布在 https://spiyer99.github.io 。