| Model Name |
CogVideoX-2B |
CogVideoX-5B (Current Repository) |
| Model Description |
Entry-level model, balancing compatibility, operation, and low cost of secondary development. |
A larger model that generates higher-quality videos with better visual effects. |
| Inference Precision |
FP16, FP32, does not support BF16.
Can run on mainstream NVIDIA GPUs. |
BF16, FP32, does not support FP16.
Requires NVIDIA GPUs with Ampere architecture or higher (e.g., A100, H100). |
Inference Speed
(Single A100, Step = 50) |
FP16: ~90 s |
BF16: ~200 s |
| Single GPU Inference Memory Usage |
18GB using SAT
12GB (with tied VAE) using diffusers
24GB (without tied VAE) using diffusers |
26GB using SAT
21GB (with tied VAE) using diffusers
41GB (without tied VAE) using diffusers |
| Multi-GPU Inference Memory Usage |
20GB minimum per GPU using diffusers |
| Fine-tuning Memory Usage (per GPU) |
47 GB (bs=1, LORA)
61 GB (bs=2, LORA)
62GB (bs=1, SFT) |
63 GB (bs=1, LORA)
80 GB (bs=2, LORA)
75GB (bs=1, SFT)
|
| Prompt Language |
English* |
| Maximum Prompt Length |
226 Tokens |
| Video Length |
6 seconds |
| Frame Rate |
8 frames/second |
| Video Resolution |
720 x 480, does not support other resolutions (including fine-tuning) |
**Note** Using [SAT](https://github.com/THUDM/SwissArmyTransformer) for inference and fine-tuning of SAT version models. Feel free to visit our GitHub for more information.
## Quick Start 🤗
This model supports deployment using the huggingface diffusers library. You can deploy it by following these steps.
**We recommend that you visit our [GitHub](https://github.com/THUDM/CogVideo) and check out the relevant prompt
optimizations and conversions to get a better experience.**
1. Install the required dependencies
```shell
pip install --upgrade opencv-python transformers diffusers # Must using diffusers>=0.30.0
```
2. Run the code
```python
import gc
import torch
from diffusers import CogVideoXPipeline
from diffusers.utils import export_to_video
prompt = "A panda, dressed in a small, red jacket and a tiny hat, sits on a wooden stool in a serene bamboo forest. The panda's fluffy paws strum a miniature acoustic guitar, producing soft, melodic tunes. Nearby, a few other pandas gather, watching curiously and some clapping in rhythm. Sunlight filters through the tall bamboo, casting a gentle glow on the scene. The panda's face is expressive, showing concentration and joy as it plays. The background includes a small, flowing stream and vibrant green foliage, enhancing the peaceful and magical atmosphere of this unique musical performance."
pipe = CogVideoXPipeline.from_pretrained(
"THUDM/CogVideoX-5b",
torch_dtype=torch.bfloat16
)
pipe.enable_model_cpu_offload()
gc.collect()
torch.cuda.empty_cache()
torch.cuda.reset_accumulated_memory_stats()
torch.cuda.reset_peak_memory_stats()
pipe.vae.enable_tiling()
video = pipe(
prompt=prompt,
num_videos_per_prompt=1,
num_inference_steps=50,
num_frames=49,
guidance_scale=6,
generator=torch.Generator(device="cuda").manual_seed(42),
).frames[0]
export_to_video(video, "output.mp4", fps=8)
```
**Using a single A100 GPU, generating a video with the above configuration takes approximately 200 seconds**
If the generated model appears “all green” and not viewable in the default MAC player, it is a normal phenomenon (due to
OpenCV saving video issues). Simply use a different player to view the video.
## Explore the Model
Welcome to our [github](https://github.com/THUDM/CogVideo), where you will find:
1. More detailed technical details and code explanation.
2. Optimization and conversion of prompt words.
3. Reasoning and fine-tuning of SAT version models, and even pre-release.
4. Project update log dynamics, more interactive opportunities.
5. CogVideoX toolchain to help you better use the model.
## Model License
This model is released under the [CogVideoX LICENSE](LICENSE).
## Citation
```
@article{yang2024cogvideox,
title={CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer},
author={Zhuoyi Yang and Jiayan Teng and Wendi Zheng and Ming Ding and Shiyu Huang and JiaZheng Xu and Yuanming Yang and Xiaohan Zhang and Xiaotao Gu and Guanyu Feng and Da Yin and Wenyi Hong and Weihan Wang and Yean Cheng and Yuxuan Zhang and Ting Liu and Bin Xu and Yuxiao Dong and Jie Tang},
year={2024},
}
```
