Contributions

We propose a new self-supervised CNN pre-training technique based on a novel auxiliary task called odd-one-out learning. In this task, we sample subsequences from videos and ask the network to learn to predict the odd video subsequence. The odd video subsequence is sampled such that it has wrong temporal order of frames while the even ones have the correct temporal order. Our learning machine is implemented as multi-stream convolutional neural network, which is learned end-to-end. Using odd-one-out networks, we learn temporal representations for videos that generalizes to other related tasks such as action recognition.

Method

1、Model

O3N is composed of (N+1) input branches, each contains five Convolutional layers and weights are shared across the input layers. Configuration of each input branch is identical to AlexNet architecture up to the first fully connected layer. We then introduce a fusion layer which merges the information from (N+1) branches after the first fully connected layer. We experiment with two fusion models, the Concatenation model and sum of difference model leading to two different network architectures as shown in Fig. 2.

Concatenation model: The first fully connected layers from each branch are concatenated to give a (N + 1) × d dimensional vector, where d is the dimensionality of the first fully connected layer.
Sum of difference model: The first fully connected layers from each branch are summed after taking the pair-wise activation difference leading to a d dimensional vector, where d is the dimensionality of the first fully connected layer. Mathematically, let vi be the activation vector of the i-th branch of the network. The output of the sum of difference layer is given by

2、Three sampling strategies.

Consecutive sampling: We sample W number of consecutive frames N times from video X to generate N number of even (related) elements. Each sampled even element of the odd-one-out question is a valid video sub-clip consisting of W consecutive frames from the original video. However, the odd video sequence of length W is constructed by random ordering of frames and therefore does not satisfy the order constraints.
Random sampling: We randomly sample W frames N times from the video X to generate N number of even (related) elements. Each of these N elements are sequences that has the correct temporal order and satisfy the original order constraints of X. However, the frames are not consecutive as in the case of consecutive sampling. The odd video sequence of length W is also constructed by randomly sampling frames. Similar to consecutive sampling strategy, the odd sequence does not satisfy the order constraints. Specifically, we randomly shuffled the frames of the odd element (sequence).
Constrained consecutive sampling: First we sub select a video clip of size 1.5 ×W from the original video which we denote by Xˆ. We randomly sample W consecutive frames N times from Xˆ to generate N number of even (related) elements. Each of these N elements are subsequences that have the correct temporal order and satisfy the original order constraints of X. At the same time each of the sampled even video clips of size W overlaps more than 50% with each other. The odd video sequence of length W is also constructed by randomly sampling frames from Xˆ. Similar to other sampling strategies, the odd sequence does not satisfy the order constraints. Specifically, we randomly shuffled the frames of the odd element (sequence).

3、Video frame encoding.

Each element (video-clip or subsequence) in an odd-one-out question is encoded to extract temporal information before presenting to the first convolutional filters of the network. There are several ways to capture the temporal structure of a video sequence. For example, one can use 3D-convolutions, recurrent encoders, rank-pooling encoders or simply concatenate frames. Odd-one-out networks can use any of the above methods to learn video representations in self-supervised manner using video data. Next, we discuss three technique that is used in our experiments to encode video-frame-clips using the differences of RGB frames into a single tensor Xd.

Sum of differences of frames video-clip encoder: We take the difference of frames and then sum the differences to obtain a single image Xd. This single image captures the structure of the sequence. Precisely, this is exactly same as the equation 2 but now applied over frames. It is interesting to note that this equation boils down to a weighted average of frames such that Xd=∑wtXt where the weight of frame at index t is given by

Dynamic image encoder: This method is similar to the sum of differences of frames method, however the only difference is that now the input sequence is pre-processed to obtain a smoothed sequence M=<M1,M2,⋯MW>. Smoothing is obtain using the mean at index t. The smoothed frame at index t denoted by Mt is given by

where Xj is the frame at index j of the sub-video.

Stack of differences of frames video-clip encoder: We also stack the difference of frames. However, now the resulting image is not any more a standard RGB image with three channels. Instead, we obtain (N − 1) × 3 channel image.

Results

More Reference to Follow

traditionally unsupervised feature learning (e.g.[6, 20])【看看 video 怎么做 unsupervised】
There has also been un- supervised temporal feature encoding methods to capture the structure of videos for action classification [13, 14, 15, 29, 36].

论文阅读：Self-Supervised Video Representation Learning With Odd-One-Out Networks相关推荐

论文阅读：Self-supervised video representation learning with space-time cubic puzzles
论文名称:Self-supervised video representation learning with space-time cubic puzzles(2019 AAAI) 论文作者:Dah ...
论文阅读：Self-supervised Video Representation Learning with Cross-Stream Prototypical Contrasting
题目:Self-supervised Video Representation Learning with Cross-Stream Prototypical Contrasting 作者:Marti ...
【论文阅读】InfoGAN: Interpretable Representation Learning by Information Maximizing GAN
论文下载 bib: @inproceedings{chenduan2016infogan,author = {Xi Chen and Yan Duan and Rein Houthooft and J ...
论文阅读之Improved Word Representation Learning with Sememes(2017)
文章目录论文介绍 Conventional Skip-gram Model Simple Sememe Aggregation Model(SSA) Sememe Attention over Co ...
论文阅读——Mockingjay: unsupervised speech representation learning
<Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Enco ...
论文阅读GraphSAGE《Inductive Representation Learning on Large Graphs》
目录研究背景算法模型采样邻居顶点生成向量的伪代码聚合函数的选取参数的学习实验结果 GraphSAGE的核心: 改进方向: 其他补充学习知识归纳式与直推式为什么GCN是transduc ...
【论文阅读-NRE】Self-Supervised Representation Learning via Neighborhood-Relational Encoding
Abs: 自监督表示 NRE:neighborhood-relational encoding邻居相关编码旧的unsupervised方法一般只注重用deep net抓取data中的重点,而忽视了他 ...
T-PAMI-2021论文Semi-Supervised Multi-View Deep Discriminant Representation Learning阅读笔记
提示:文 0.论文信息题目:Semi-Supervised Multi-View Deep Discriminant Representation Learning 期刊: IEEE Transac ...
【论文阅读】Rethinking Spatiotemporal Feature Learning For Video Understanding
[论文阅读]Rethinking Spatiotemporal Feature Learning For Video Understanding 这是一篇google的论文,它和之前介绍的一篇face ...

论文阅读：Self-Supervised Video Representation Learning With Odd-One-Out Networks

目录