MotionCtrl Model Card
Project Page | Paper (ArXiv) | Code
π€ Gradio demo (MotionCtrl+VideoCrafter) | π€ Gradio demo (MotionCtrl+SVD)
Introduction
MotionCtrl, when provided with specific camera movements indicated by camera poses or several trajectories, can independently control both the camera and object motion in a generated video. It can be deployed on both Text-to-Video generation models, such as LVDM/VideoCrafter and AnimateDiff, as well as Image-to-Video generation models like SVD.
MotionCtrl + SVD
- Model: motionctrl.pth
- Results:
β
MotionCtrl + LVDM/VideoCrafter
- Model:
- Results:
β
MotionCtrl + AnimateDiff
- Model:
- Results:
β
Usage
- Download directly in this repository OR Download with python script
from huggingface_hub import hf_hub_download
motionctrl_lvdm_ckpt = hf_hub_download(repo_id="TencentARC/MotionCtrl", filename="motionctrl.ckpt", repo_type="model")
motionctrl_svd_ckpt = hf_hub_download(repo_id="TencentARC/MotionCtrl", filename="motionctrl_svd.ckpt", repo_type="model")
- Then generate the controllable video following the instructions in our GitHub repository.
Citation
@inproceedings{wang2023motionctrl,
title={MotionCtrl: A Unified and Flexible Motion Controller for Video Generation},
author={Wang, Zhouxia and Yuan, Ziyang and Wang, Xintao and Chen, Tianshui and Xia, Menghan and Luo, Ping and Shan, Yin},
booktitle={arXiv preprint arXiv:2312.03641},
year={2023}
}