Controllable Music Transformer

Official code for our paper Video Background Music Generation with Controllable Music Transformer (ACM MM 2021 Best Paper Award)

[Paper] [Demos] [Bibtex]

Introduction

在这项工作中，我们解决了视频背景音乐生成的任务。以前的一些作品实现了有效的音乐生成，但无法为给定的视频专门生成悦耳的音乐，也没有考虑到视频音乐的节奏一致性。为了生成与给定视频匹配的背景音乐，我们首先建立视频和背景音乐之间的节奏关系。特别是，我们将视频中的时间、运动速度和运动显着性分别与音乐中的节拍、模拟音符密度和模拟音符强度联系起来。然后，我们提出了 CMT，这是一种可控音乐转换器，可以对上述节奏特征进行本地控制，以及对音乐流派和用户指定使用的乐器进行全局控制。客观和主观评价表明，生成的背景音乐与输入视频的兼容性令人满意，同时音乐质量令人印象深刻。

We address the unexplored task – video background music generation. We first establish three rhythmic relations between video and background music. We then propose a Controllable Music Transformer (CMT) to achieve local and global control of the music generation process. Our proposed method does not require paired video and music data for training while generates melodious and compatible music with the given video.

Directory Structure

src/: code of the whole pipeline
- train.py: training script, take a npz as input music data to train the model
- model.py: code of the model
- gen_midi_conditional.py: inference script, take a npz (represents a video) as input to generate several songs
- src/video2npz/: convert video into npz by extracting motion saliency and motion speed
dataset/: processed dataset for training, in the format of npz
logs/: logs that automatically generate during training, can be used to track training process
exp/: checkpoints, named after val loss (e.g. loss_8_params.pt)
inference/: processed video for inference (.npz), and generated music(.mid)

Preparation

clone this repo
download the processed data lpd_5_prcem_mix_v8_10000.npz from HERE and put it under dataset/
download the pretrained model loss_8_params.pt from HERE and put it under exp/
install ffmpeg=3.2.4
prepare a Python3 conda environment
- ```
conda create -n mm21_py3 python=3.7
conda activate mm21_py3
pip install -r py3_requirements.txt
```
- choose the correct version of torch and pytorch-fast-transformers based on your CUDA version (see fast-trainsformers repo and this issue)
prepare a Python2 conda environment (for extracting visbeat)
- ```
conda create -n mm21_py2 python=2.7
conda activate mm21_py2
pip install -r py2_requirements.txt
```
- open visbeat package directory (e.g. anaconda3/envs/XXXX/lib/python2.7/site-packages/visbeat), replace the original Video_CV.py with src/video2npz/Video_CV.py

Training

Note: use the mm21_py3 environment: conda activate mm21_py3

A quick start using the processed data lpd_5_prcem_mix_v8_10000.npz (1~2 days on 8x 1080Ti GPUs):
```
python train.py --name train_default -b 8 --gpus 0 1 2 3 4 5 6 7
```

If you want to reproduce the whole process:
1. download the lpd-5-cleansed dataset from HERE and put the extracted files under dataset/lpd_5_cleansed/
2. go to src/ and convert the pianoroll files (.npz) to midi files (~3 files / sec):
```
python pianoroll2midi.py --in_dir ../dataset/lpd_5_cleansed/ --out_dir ../dataset/lpd_5_cleansed_midi/
```
3. convert midi files to .npz files with our proposed representation (~5 files / sec):
```
python midi2numpy_mix.py --midi_dir ../dataset/lpd_5_cleansed_midi/ --out_name data.npz
```
4. train the model (1~2 days on 8x 1080Ti GPUs):
```
python train.py --name train_exp --train_data ../dataset/data.npz -b 8 --gpus 0 1 2 3 4 5 6 7
```

Note: If you want to train with another MIDI dataset, please ensure that each track belongs to one of the five instruments (Drums, Piano, Guitar, Bass, or Strings) and is named exactly with its instrument. You can check this with Muspy:

import muspymidi = muspy.read_midi('xxx.mid')
print([track.name for track in midi.tracks]) # Should be like ['Drums', 'Guitar', 'Bass', 'Strings']

Inference

convert input video (MP4 format) into npz (use the mm21_py2 environment):

conda activate mm21_py2
cd src/video2npz
# try resizing the video if this takes a long time
sh video2npz.sh ../../videos/xxx.mp4

run model to generate .mid (use the mm21_py3 environment) :

conda activate mm21_py3
python gen_midi_conditional.py -f "../inference/xxx.npz" -c "../exp/loss_8_params.pt"# if using another training set, change `decoder_n_class` and `init_n_class` in `gen_midi_conditional` to the ones in `train.py`

convert midi into audio: use GarageBand (recommended) or midi2audio
- set tempo to the value of tempo in video2npz/metadata.json (generated when running video2npz.sh)

combine original video and audio into video with BGM:

ffmpeg -i 'xxx.mp4' -i 'yyy.mp3' -c:v copy -c:a aac -strict experimental -map 0:v:0 -map 1:a:0 'zzz.mp4'# xxx.mp4: input video
# yyy.mp3: audio file generated in the previous step
# zzz.mp4: output video

Matching Method

The matching method finds the five most matching music pieces from the music library for a given video (use the mm21_py3 environment).
```
conda activate mm21_py3
python src/match.py inference/xxx.npz dataset/lpd_5_prcem_mix_v8_10000.npz
```

Citation

@inproceedings{di2021video,title={Video Background Music Generation with Controllable Music Transformer},author={Di, Shangzhe and Jiang, Zeren and Liu, Si and Wang, Zhaokai and Zhu, Leyan and He, Zexin and Liu, Hongming and Yan, Shuicheng},booktitle={Proceedings of the 29th ACM International Conference on Multimedia},pages={2037--2045},year={2022}
}

Acknowledgements

Our code is based on Compound Word Transformer.

山东大学项目实训十六——可控音乐变压器Controllable Music Transformer相关推荐

山东大学项目实训——地图圈系统——微信小程序（16）
地图圈系统--微信小程序(16) 一.社区功能二.项目代码三.效果展示一.社区功能这次项目实训的社区功能是一个很大的模块,包括显示用户发布的文字及图片.用户选择的地点.发布的时间.点赞.评论等 ...
山东大学项目实训——6月29日
总结项目实训近两天的工作. 首先在第一天上午,参加项目实训开幕仪式,了解实训相关规定和内容.然后与李学庆老师见面,由他像我们讲解项目内容,介绍项目需求. 同时我主动提出担任项目组长. 通过与李学庆老师 ...
山东大学项目实训——简易微信小游戏（一）
一.项目背景微信小程序接触真实用户快,对用户反馈快速体现,是一个非常好的实践平台.同时就行业背景来看,人们对于无需安装打开即玩的休闲类小游戏的需求依然存在. 二.工作内容开发绳套奶牛.大鱼吃小鱼. ...
实训十六：路由器RIPV2配置
一.实验目的掌握动态路由的配置方法理解 RIP-2 协议的工作过程理解 RIP-2 对变长子网掩码的支持二.应用环境在不连续的变长子网掩码环境中,RIP-1 的自动汇总会得出错误的路由 RI ...
山东大学项目实训——地图圈系统——微信小程序（18）
地图圈系统--微信小程序(18) 一.轨迹记录功能二.项目代码三.效果展示一.轨迹记录功能轨迹记录功能的实现思路相对简单,设置一个定时器,每隔1s获取一下当前位置,将当前的经纬度坐标加入到数组 ...
山东大学项目实训（四）—— 微信小程序扫描web端二维码实现web端登录
效果点击登录后,显示二维码→打开"探古"(本项目)微信小程序,扫描二维码确认登录→web端登录成功主要流程因为本人主要负责web前端的开发,所以本文仅介绍web前端的实现方法 ...
山东大学项目实训小组一——基于深度学习的AI视频剪辑器“易剪”
技术要点:图像处理计算机视觉深度学习多媒体前端一.项目研究背景: 随着短视频热潮的兴起,越来越多的人投入精力到了视频剪辑视频制作之中.然而利用现有的视频剪辑工具,剪辑一段视频是非常麻烦的,尤其 ...
山东大学项目实训——地图圈系统——微信小程序（17）
地图圈系统--微信小程序(17) 一.社区附加功能二.项目代码三.效果展示一.社区附加功能在上次社区功能的基础上,我增加了几个附加功能,首先点击用户头像可以看到此用户的用户信息,然后点击查看动 ...
山东大学项目实训——地图圈系统——微信小程序（5）
地图圈系统--微信小程序(5) 一.腾讯位置服务路线规划api 二.结果响应实例三.项目代码四.效果展示一.腾讯位置服务路线规划api 提供路线规划能力. 1.驾车(driving):支持结合实 ...

山东大学项目实训十六——可控音乐变压器Controllable Music Transformer