Xora️

This is the official repository for Xora.

Introduction
Installation
Inference
- Inference Code
Acknowledgement

Introduction

The performance of Diffusion Transformers is heavily influenced by the number of generated latent pixels (or tokens). In video generation, the token count becomes substantial as the number of frames increases. To address this, we designed a carefully optimized VAE that compresses videos into a smaller number of tokens while utilizing a deeper latent space. This approach enables our model to generate high-quality 768x512 videos at 24 FPS, achieving near real-time speeds.

Installation

Setup

The codebase currently uses Python 3.10.5, CUDA version 12.2, and supports PyTorch >= 2.1.2.

git clone https://github.com/LightricksResearch/xora-core.git
cd xora-core

# create env
python -m venv env
source env/bin/activate
python -m pip install -e .\[inference-script\]

Then, download the model from Hugging Face

from huggingface_hub import snapshot_download

model_path = 'PATH'   # The local directory to save downloaded checkpoint
snapshot_download("Lightricks/Xora", local_dir=model_path, local_dir_use_symlinks=False, repo_type='model')

Inference

Inference Code

To use our model, please follow the inference code in inference.py at https://github.com/LightricksResearch/xora-core/blob/main/inference.py:

For text-to-video generation:

python inference.py --ckpt_dir 'PATH' --prompt "PROMPT" --height HEIGHT --width WIDTH

For image-to-video generation:

python inference.py --ckpt_dir 'PATH' --prompt "PROMPT" --input_image_path IMAGE_PATH --height HEIGHT --width WIDTH

Acknowledgement

We are grateful for the following awesome projects when implementing Xora:

DiT and PixArt-alpha: vision transformers for image generation.

Spaces:

openfree
/

ginigen-sora

Paused