背景介绍

在现有的的行为分类数据集（UCF-101 and HMDB-51）中，视频数据的缺乏使得确定一个好的视频结构很困难，大部分方法在小规模数据集上取得差不多的效果。这篇文章根据Kinetics人类行为动作来重新评估这些先进的结构。Kinetics有两个数量级的数据，400类人类行为，每一类有超过400剪辑，并且这些都是从现实的，有挑战性的YouTube视频中收集的。我们分析了现有的结构在这个数据集上进行行为分类任务的过程以及在Kinetics上预训练后是如何提高在小样本集上的表现。

我们引入了一个基于二维卷积膨胀网络的Two-Stream Inflated 三维卷积网络（I3D）：深度图像分类卷积网络中的滤波器和pooling卷积核推广到了3D的情况，这样能够学到从视频中提取好的时空特征的能力，同时可以利用ImageNet结构的设计以及参数；我们发现在Kinetics上预训练之后，I3D模型在行为分类上提高了很多，在HMDB-51上达到了80.7%的正确率，在UCF-101上达到了98.0%的正确率。

具体参考前面的介绍《Qua Vadis, Action Recognition? A New Model and the Kinetics Dataset》论文解读之Two-Stream I3D

今天主要介绍在UCF-101上的I3D finetune

相关的项目资源也是很多，这里简单列一下：

1.https://github.com/LossNAN/I3D-Tensorflow

2.https://github.com/piergiaj/pytorch-i3d

3.https://github.com/USTC-Video-Understanding/I3D_Finetune

4.https://github.com/hassony2/kinetics_i3d_pytorch

All this code is based on Deepmind's Kinetics-I3D. Including PyTorch versions of their models.

下面我们介绍下操作步骤。

# how to run this demo?about train,test .Follow these steps please.
1.Clone this repo:https://github.com/LossNAN/I3D-Tensorflow
2. Download kinetics pretrained I3D modelsIn order to finetune I3D network on UCF101, you have to download Kinetics pretrained I3D models provided by DeepMind at here.
https://github.com/deepmind/kinetics-i3d/tree/master/dataSpecifically, download the repo kinetics-i3d and put the data/checkpoints folder into data subdir of our I3D_Finetune repo:git clone https://github.com/deepmind/kinetics-i3d
cp -r kinetics-i3d/data/checkpoints I3D-Tensorflow/ps:if u have model files in I3D-Tensorflow/checkpoints,u don't need to download again.3.Data_process1>download UCF101 and HMDB51 dataset by yourself
2>extract RGB and FLOW frames by denseFlow_GPU, (or https://github.com/wanglimin/dense_flow)
you will get rgb frames and x_flow,y_flow frames in the video path.Then,run
python3 split_flow.py
to get files such as:~PATH/UCF-101/ApplyEyeMakeup/v_ApplyEyeMakeup_g01_c01/i for all rgb frames
~PATH/UCF-101/ApplyEyeMakeup/v_ApplyEyeMakeup_g01_c01/x for all x_flow frames
~PATH/UCF-101/ApplyEyeMakeup/v_ApplyEyeMakeup_g01_c01/y for all y_flow frames3>convert images to list for train and testcd ./list/ucf_list/
bash ./convert_images_to_list.sh ~path/UCF-101 4you will get train.list and test.list for your own dataset
such as: ~PATH/UCF-101/ApplyEyeMakeup/v_ApplyEyeMakeup_g01_c01 04.Train your own dataset(UCF101 as example)
1>if you get path errors, please modify by yourselfcd ./experiments/ucf-101
python3 train_ucf_rgb.py
python3 train_ucf_flow.pytest_flow.list 2004 /home/gavin/Dataset/UCF-101_Flow/PlayingDaf/v_PlayingDaf_g17_c03 59result :
1.flow ：0.9136422.rgb:TOP_1_ACC in test: 0.9496493.rgb+ flow:TOP_1_ACC in test: 0.969555

https://github.com/Gavin666Github/dense_flow

提取代码如下：extract_flow_rgb.py

import numpy as np
import osroot_path = "/home/gavin/Dataset/actiondata" #"/home/gavin/Downloads/UCF-101"
flow_path = "/home/gavin/Dataset/action_flow" #"/home/gavin/Dataset/UCF-101_Flow"# ./denseFlow -f /home/gavin/Downloads/UCF-101/ApplyEyeMakeup/v_ApplyEyeMakeup_g01_c01.avi
# -x /home/gavin/Dataset/UCF-101_Flow/ApplyEyeMakeup/v_ApplyEyeMakeup_g01_c01/x
# -y /home/gavin/Dataset/UCF-101_Flow/ApplyEyeMakeup/v_ApplyEyeMakeup_g01_c01/y
# -i /home/gavin/Dataset/UCF-101_Flow/ApplyEyeMakeup/v_ApplyEyeMakeup_g01_c01/i -b 20def extract_flow_rgb(root_dirs):video_dirs = os.listdir(root_dirs) # ApplyEyeMakeup,...for video_dir in video_dirs:tmp = video_dirvideo_dir = os.path.join(root_dirs, video_dir) # root_path/ApplyEyeMakeupflow_dir = os.path.join(flow_path, tmp) # flow_path/ApplyEyeMakeupvideo_list = os.listdir(video_dir) # v_ApplyEyeMakeup_g01_c01.avi,...for video in video_list:video_name = video.split('.')[0]# tmp_dir = os.path.join(video_dir, video_name)tmpFlow_dir = os.path.join(flow_dir, video_name)tmp_file = os.path.join(video_dir, video)print(tmp_file) # path/UCF-101/ApplyEyeMakeup/v_ApplyEyeMakeup_g01_c01.aviif not os.path.exists(tmpFlow_dir):os.makedirs(tmpFlow_dir)i_dir = os.path.join(tmpFlow_dir,'i')x_dir = os.path.join(tmpFlow_dir, 'x')y_dir = os.path.join(tmpFlow_dir, 'y')cmd = './denseFlow -f %s -x %s -y %s -i %s -b 20' % (tmp_file,x_dir,y_dir,i_dir)if len(tmp_file.split('.')) > 1:os.system(cmd)print("extract rgb and flow from %s done." % (video))def splitflow(root_dirs):video_dirs = os.listdir(root_dirs)for video_dir in video_dirs:tmp = video_dirvideo_dir = os.path.join(root_dirs, video_dir)video_list = os.listdir(video_dir)for video in video_list:print(os.path.join(video_dir, video))image_list = os.listdir(os.path.join(video_dir, video))i_dir = os.path.join(video_dir, video, 'i')x_dir = os.path.join(video_dir, video, 'x')y_dir = os.path.join(video_dir, video, 'y')if not os.path.exists(i_dir):os.makedirs(i_dir)if not os.path.exists(x_dir):os.makedirs(x_dir)if not os.path.exists(y_dir):os.makedirs(y_dir)for image in image_list:classic = image.split('_')[0]cmd = 'mv %s %s' % (os.path.join(video_dir, video, image), os.path.join(video_dir, video, classic))if len(image.split('_')) > 1:os.system(cmd)if __name__ == '__main__':#extract_flow_rgb(root_path)splitflow(flow_path)

对于自定义的行为，我们同样采取上述方法，比如我们训练两个类的行为

文件夹下面就是具体一个个的视频了。

提取的rgb和flow存在action_flow下

进去一个看看：

再进：

即整理好的i,x，y三类图片截图。

接下来可以进行训练了。

参考：

1.https://www.jianshu.com/p/0c26a33cefd0

2.https://www.jianshu.com/p/71d35fda32d3

3.github:https://github.com/LossNAN/I3D-Tensorflow

I3D Finetune相关推荐

i3d finetune(微调)
主要参考 https://github.com/deepmind/kinetics-i3d/blob/master/evaluate_sample.py 来恢复网络图(基本可以在提供的文件包中的用来p ...
I3D视频分类论文梗概及代码解读Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
论文https://arxiv.org/pdf/1705.07750.pdf,from DeepMind ,CVPR2017 代码https://github.com/LossNAN/I3D-Tens ...
《Qua Vadis, Action Recognition? A New Model and the Kinetics Dataset》论文解读之Two-Stream I3D
本文是CVPR 2017的一篇文章 Paper:Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset github: ...
增量学习不只有finetune，三星AI提增量式少样本目标检测算法 | CVPR 2020
作者 | VincentLee 来源 | 晓飞的算法工程笔记该论文研究了非常有意义的增量式少样本目标检测场景iFSD(Incremental Few-Shot Detection),场景设置如下: ...
干货 | 谷歌BERT模型fine-tune终极实践教程
作者 | 奇点机智从11月初开始,Google Research就陆续开源了BERT的各个版本.Google此次开源的BERT是通过TensorFlow高级API-- tf.estimator进行封 ...
Fine-tune之后的NLP新范式：Prompt越来越火，CMU华人博士后出了篇综述文章
视学算法报道机器之心编辑部 CMU 博士后研究员刘鹏飞:近代自然语言处理技术发展的第四范式可能是预训练语言模型加持下的 Prompt Learning. 近几年,NLP 技术发展迅猛,特别是 BER ...
caffe finetune predict and classify the lung nodule（肺结节的分类）
引言--做了什么?如上一篇博文所提到用ImageNet图像来帮助医学图像的识别,但是ImageNet里面的图像(二维,彩色)没有医学图像,包含一些诸如鸟类.猫.狗.直升机等物体的识别,与医学图像(二维 ...
【深度学习】keras框架使用预训练模型进行Finetune的应用
文章目录 1 概述 2 Keras 3 VGG16 (VGG16) 4 MobileNetV2 (MobileNetV2) 5 盗版ResNetV2 (InceptionResNetV2) 6 盗梦空 ...
Caffe学习系列(13):对训练好的模型进行fine-tune
使用http://www.cnblogs.com/573177885qq/p/5804863.html中的图片进行训练和测试. 整个流程差不多,fine-tune命令: ./build/tools/c ...

I3D Finetune

背景介绍

I3D Finetune相关推荐

最新文章

热门文章