Skip to content

VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models (CVPR 2024)

License

Notifications You must be signed in to change notification settings

HyeonHo99/Video-Motion-Customization

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

27 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Video-Motion-Customization (CVPR 2024)

This repository is the official implementation of VMC.
[CVPR 2024] VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models
Hyeonho Jeong*, Geon Yeong Park*, Jong Chul Ye,

Project Website arXiv


Given an input video with any type of motion patterns, our framework, VMC fine-tunes only the Keyframe Generation Module within hierarchical Video Diffusion Models for motion-customized video generation.

News

  • [2023.11.30] Initial Code Release (Additional codes will be uploaded.)

Setup

Requirements

pip install -r requirements.txt

Usage

The following command will run "train & inference" at the same time:

accelerate launch train_inference.py --config configs/car_forest.yml

*Additional scripts of 'train_only' and 'inference_with_pretrained' will be uploaded too.

Data

Results

Input Videos Output Videos

Video Style Transfer

Backward Motion Customization

Reversed Videos Output Videos

Citation

If you find our work interesting, please cite our paper.

@article{jeong2023vmc,
  title={VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models},
  author={Jeong, Hyeonho and Park, Geon Yeong and Ye, Jong Chul},
  journal={arXiv preprint arXiv:2312.00845},
  year={2023}
}

Shoutouts


Thanks all for open-sourcing!