目录

Contributions

Method

1、Model

2、Three sampling strategies.

3、Video frame encoding.

Results

More Reference to Follow


论文名称:Self-Supervised Video Representation Learning With Odd-One-Out Networks(2017 CVPR)

论文作者:Basura Fernando, Hakan Bilen, Efstratios Gavves, Stephen Gould

下载地址:https://openaccess.thecvf.com/content_cvpr_2017/html/Fernando_Self-Supervised_Video_Representation_CVPR_2017_paper.html


Contributions

We propose a new self-supervised CNN pre-training technique based on a novel auxiliary task called odd-one-out learning. In this task, we sample subsequences from videos and ask the network to learn to predict the odd video subsequence. The odd video subsequence is sampled such that it has wrong temporal order of frames while the even ones have the correct temporal order. Our learning machine is implemented as multi-stream convolutional neural network, which is learned end-to-end. Using odd-one-out networks, we learn temporal representations for videos that generalizes to other related tasks such as action recognition.


Method

1、Model

O3N is composed of (N+1) input branches, each contains five Convolutional layers and weights are shared across the input layers. Configuration of each input branch is identical to AlexNet architecture up to the first fully connected layer. We then introduce a fusion layer which merges the information from (N+1) branches after the first fully connected layer. We experiment with two fusion models, the Concatenation model and sum of difference model leading to two different network architectures as shown in Fig. 2.

  1. Concatenation model: The first fully connected layers from each branch are concatenated to give a (N + 1) × d dimensional vector, where d is the dimensionality of the first fully connected layer.
  2. Sum of difference model: The first fully connected layers from each branch are summed after taking the pair-wise activation difference leading to a d dimensional vector, where d is the dimensionality of the first fully connected layer. Mathematically, let vi be the activation vector of the i-th branch of the network. The output of the sum of difference layer is given by

2、Three sampling strategies.

  1. Consecutive sampling: We sample W number of consecutive frames N times from video X to generate N number of even (related) elements. Each sampled even element of the odd-one-out question is a valid video sub-clip consisting of W consecutive frames from the original video. However, the odd video sequence of length W is constructed by random ordering of frames and therefore does not satisfy the order constraints.
  2. Random sampling: We randomly sample W frames N times from the video X to generate N number of even (related) elements. Each of these N elements are sequences that has the correct temporal order and satisfy the original order constraints of X. However, the frames are not consecutive as in the case of consecutive sampling. The odd video sequence of length W is also constructed by randomly sampling frames. Similar to consecutive sampling strategy, the odd sequence does not satisfy the order constraints. Specifically, we randomly shuffled the frames of the odd element (sequence).
  3. Constrained consecutive sampling: First we sub select a video clip of size 1.5 ×W from the original video which we denote by Xˆ. We randomly sample W consecutive frames N times from Xˆ to generate N number of even (related) elements. Each of these N elements are subsequences that have the correct temporal order and satisfy the original order constraints of X. At the same time each of the sampled even video clips of size W overlaps more than 50% with each other. The odd video sequence of length W is also constructed by randomly sampling frames from Xˆ. Similar to other sampling strategies, the odd sequence does not satisfy the order constraints. Specifically, we randomly shuffled the frames of the odd element (sequence).

3、Video frame encoding.

Each element (video-clip or subsequence) in an odd-one-out question is encoded to extract temporal information before presenting to the first convolutional filters of the network. There are several ways to capture the temporal structure of a video sequence. For example, one can use 3D-convolutions, recurrent encoders, rank-pooling encoders or simply concatenate frames. Odd-one-out networks can use any of the above methods to learn video representations in self-supervised manner using video data. Next, we discuss three technique that is used in our experiments to encode video-frame-clips using the differences of RGB frames into a single tensor Xd.

  • Sum of differences of frames video-clip encoder: We take the difference of frames and then sum the differences to obtain a single image Xd. This single image captures the structure of the sequence. Precisely, this is exactly same as the equation 2 but now applied over frames. It is interesting to note that this equation boils down to a weighted average of frames such that Xd=∑wtXt where the weight of frame at index t is given by

  • Dynamic image encoder: This method is similar to the sum of differences of frames method, however the only difference is that now the input sequence is pre-processed to obtain a smoothed sequence M=<M1,M2,⋯MW>. Smoothing is obtain using the mean at index t. The smoothed frame at index t denoted by Mt is given by

where Xj is the frame at index j of the sub-video.

  • Stack of differences of frames video-clip encoder: We also stack the difference of frames. However, now the resulting image is not any more a standard RGB image with three channels. Instead, we obtain (N − 1) × 3 channel image.

Results


More Reference to Follow

  1. traditionally unsupervised feature learning (e.g.[6, 20])【看看 video 怎么做 unsupervised】
  2. There has also been un- supervised temporal feature encoding methods to capture the structure of videos for action classification [13, 14, 15, 29, 36].

论文阅读:Self-Supervised Video Representation Learning With Odd-One-Out Networks相关推荐

  1. 论文阅读:Self-supervised video representation learning with space-time cubic puzzles

    论文名称:Self-supervised video representation learning with space-time cubic puzzles(2019 AAAI) 论文作者:Dah ...

  2. 论文阅读:Self-supervised Video Representation Learning with Cross-Stream Prototypical Contrasting

    题目:Self-supervised Video Representation Learning with Cross-Stream Prototypical Contrasting 作者:Marti ...

  3. 【论文阅读】InfoGAN: Interpretable Representation Learning by Information Maximizing GAN

    论文下载 bib: @inproceedings{chenduan2016infogan,author = {Xi Chen and Yan Duan and Rein Houthooft and J ...

  4. 论文阅读之Improved Word Representation Learning with Sememes(2017)

    文章目录 论文介绍 Conventional Skip-gram Model Simple Sememe Aggregation Model(SSA) Sememe Attention over Co ...

  5. 论文阅读——Mockingjay: unsupervised speech representation learning

    <Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Enco ...

  6. 论文阅读GraphSAGE《Inductive Representation Learning on Large Graphs》

    目录 研究背景 算法模型 采样邻居顶点 生成向量的伪代码 聚合函数的选取 参数的学习 实验结果 GraphSAGE的核心: 改进方向: 其他补充学习知识 归纳式与直推式 为什么GCN是transduc ...

  7. 【论文阅读-NRE】Self-Supervised Representation Learning via Neighborhood-Relational Encoding

    Abs: 自监督表示 NRE:neighborhood-relational encoding邻居相关编码 旧的unsupervised方法一般只注重用deep net抓取data中的重点,而忽视了他 ...

  8. T-PAMI-2021论文Semi-Supervised Multi-View Deep Discriminant Representation Learning阅读笔记

    提示:文 0.论文信息 题目:Semi-Supervised Multi-View Deep Discriminant Representation Learning 期刊: IEEE Transac ...

  9. 【论文阅读】Rethinking Spatiotemporal Feature Learning For Video Understanding

    [论文阅读]Rethinking Spatiotemporal Feature Learning For Video Understanding 这是一篇google的论文,它和之前介绍的一篇face ...

最新文章

  1. Datawhale与Tianchi助力产学研融合-华北电力大学专场
  2. 移动端制作公共样式reset
  3. python培训学校-如何找到靠谱的Python培训学校?
  4. python No-ASCII character ,编码错误
  5. mysql myisam 去掉表锁a_MyISAM表锁的解决方案
  6. a12处理器和骁龙855_【性能】骁龙855最新跑分曝光 多核竟超苹果A12?
  7. Flowable 数据库表结构 ACT_ID_GROUP
  8. android 自定义窗口,Android studio如何自定义设置窗口布局?
  9. python的第三方库是干什么用的-quot;别人造好的轮子”,python常用第三方库——python自动办公7...
  10. OpenGL学习一:OpenGL简介
  11. paip。java 高级特性 类默认方法,匿名方法+多方法连续调用, 常量类型
  12. 第三章:SQL Server2019数据库 之 开启 SQL 语言之旅
  13. Python爬虫系列(二)——Python爬虫批量下载百度图片
  14. python可以手眼定标吗_北京高碑店小学生Python编程周末班
  15. Java二叉树基础操作常见代码例题
  16. iOS中微信语音动画
  17. 【智力题】小环绕大环
  18. Spiking neural networks 2017 进展
  19. 帝国CMS插件自动采集发布文章插件
  20. 攻防世界--看雪看雪看雪

热门文章

  1. Qt实现的宠物小精灵对战游戏阶段二-用户的联网注册和登录
  2. Java中的html和css语言
  3. 200条装修小常识(结婚,不结婚的都要看一下,很有用的)
  4. win10浏览器网页打开慢
  5. python淘宝爬虫_python爬虫爬取淘宝商品信息
  6. Javascript数据类型及函数
  7. Arduino ESP8266自定义配置分区表
  8. php使用mysqli连接mysql数据库_PHP使用mysqli扩展连接MySQL数据库
  9. 纯干货 | 《团体标准管理规定》新旧对照表
  10. 3、halcon图像预处理:基本变换、滤波和人脸祛斑