Final code cleanings

README.md

@@ -1,13 +1,463 @@
+
+
+
+# Text2Video-Zero
+
+This repository is the official implementation of [Text2Video-Zero](https://arxiv.org/abs/2303.13439).
+
+
+**[Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://arxiv.org/abs/2303.13439)**
+</br>
+Levon Khachatryan,
+Andranik Movsisyan,
+Vahram Tadevosyan,
+Roberto Henschel,
+[Zhangyang Wang](https://www.ece.utexas.edu/people/faculty/atlas-wang), Shant Navasardyan, [Humphrey Shi](https://www.humphreyshi.com)
+</br>
+
+[Paper](https://arxiv.org/abs/2303.13439) | [Video](https://www.dropbox.com/s/uv90mi2z598olsq/Text2Video-Zero.MP4?dl=0) | [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/PAIR/Text2Video-Zero) | [Project](https://text2video-zero.github.io/)
+
+
+<p align="center">
+<img src="__assets__/github/teaser/teaser_final.png" width="800px"/>  
+<br>
+<em>Our method Text2Video-Zero enables zero-shot video generation using (i) a textual prompt (see rows 1, 2),  (ii) a prompt combined with guidance from poses or edges (see lower right), and  (iii)  Video Instruct-Pix2Pix, i.e., instruction-guided video editing (see lower left). 
+    Results are temporally consistent and follow closely the guidance and textual prompts.</em>
+</p>
+
+## News
+
+* [03/23/2023] Paper [Text2Video-Zero](https://arxiv.org/abs/2303.13439) released!
+* [03/25/2023] The [first version](https://huggingface.co/spaces/PAIR/Text2Video-Zero) of our huggingface demo (containing `zero-shot text-to-video generation` and  `Video Instruct Pix2Pix`) released!
+* [03/27/2023] The [full version](https://huggingface.co/spaces/PAIR/Text2Video-Zero) of our huggingface demo released! Now also included: `text and pose conditional video generation`, `text and canny-edge conditional video generation`, and 
+`text, canny-edge and dreambooth conditional video generation`.
+* [03/28/2023] Code for all our generation methods released! We added a new low-memory setup. Minimum required GPU VRAM is currently **12 GB**. It will be further reduced in the upcoming releases. 
+* [03/29/2023] Improved [Huggingface demo](https://huggingface.co/spaces/PAIR/Text2Video-Zero)! (i) For text-to-video generation, **any base model for stable diffusion** and **any dreambooth model** hosted on huggingface can now be loaded! (ii) We improved the quality of Video Instruct-Pix2Pix. (iii) We added two longer examples for Video Instruct-Pix2Pix.   
+* [03/30/2023] New code released! It includes all improvements of our latest huggingface iteration. See the news update from `03/29/2023`. In addition, generated videos (text-to-video) can have **arbitrary length**. 
+
+
+## Contribute
+We are on a journey to democratize AI and empower the creativity of everyone, and we believe Text2Video-Zero is a great research direction to unleash the zero-shot video generation and editing capacity of the amazing text-to-image models!
+
+To achieve this goal, all contributions are welcome. Please check out these external implementations and extensions of Text2Video-Zero. We thank the authors for their efforts and contributions:
+* https://github.com/JiauZhang/Text2Video-Zero
+* https://github.com/camenduru/text2video-zero-colab
+* https://github.com/SHI-Labs/Text2Video-Zero-sd-webui
+
+
+
+## Setup
+
+
+1. Clone this repository and enter:
+
+```shell
+git clone https://github.com/Picsart-AI-Research/Text2Video-Zero.git
+cd Text2Video-Zero/
+```
+2. Install requirements using Python 3.9 and CUDA >= 11.6
+```shell
+virtualenv --system-site-packages -p python3.9 venv
+source venv/bin/activate
+pip install -r requirements.txt
+```
+
+
+<!--- Installing [xformers](https://github.com/facebookresearch/xformers) is highly recommended for more efficiency and speed on GPUs. 
+
+### Weights
+
+#### Text-To-Video with Pose Guidance
+
+Download the pose model weights used in [ControlNet](https://arxiv.org/abs/2302.05543):
+```shell
+wget -P annotator/ckpts https://huggingface.co/lllyasviel/ControlNet/resolve/main/annotator/ckpts/hand_pose_model.pth
+wget -P annotator/ckpts https://huggingface.co/lllyasviel/ControlNet/resolve/main/annotator/ckpts/body_pose_model.pth
+```
+
+
+<!---
+#### Text-To-Video
+Any [Stable Diffusion](https://arxiv.org/abs/2112.10752) v1.4 model weights in huggingface format can be used and must be placed in `models/text-to-video`.
+For instance:
+
+```shell
+git lfs install
+git clone https://huggingface.co/CompVis/stable-diffusion-v1-4 model_weights
+mv model_weights models/text-to-video
+```
+
+#### Video Instruct-Pix2Pix
+From [Instruct-Pix2Pix](https://arxiv.org/pdf/2211.09800.pdf) download pretrained model files:
+```shell
+git lfs install
+git clone https://huggingface.co/timbrooks/instruct-pix2pix models/instruct-pix2pix
+``` 
+
+#### Text-To-Video with Pose Guidance
+From [ControlNet](https://arxiv.org/abs/2302.05543), download the open pose model file:
+```shell
+mkdir -p models/control
+wget -P models/control https://huggingface.co/lllyasviel/ControlNet/resolve/main/models/control_sd15_openpose.pth
+```
+#### Text-To-Video with Edge Guidance
+From [ControlNet](https://arxiv.org/abs/2302.05543), download the Canny edge model file:
+```shell
+mkdir -p models/control
+wget -P models/control https://huggingface.co/lllyasviel/ControlNet/resolve/main/models/control_sd15_canny.pth 
+```
+
+
+### Weights
+
+
+#### Text-To-Video with Edge Guidance and Dreambooth
+
+
+
+We provide already prepared model files derived from CIVITAI for `anime` (keyword `1girl`), `arcane style` (keyword `arcane style`) `avatar` (keyword `avatar style`) and `gta-5 style`  (keyword `gtav style`). 
+--->
+
+<!---
+To this end, download the model files from [google drive](https://drive.google.com/drive/folders/1uwXNjJ-4Ws6pqyjrIWyVPWu_u4aJrqt8?usp=share_link) and extract them into `models/control_db/`.
+--->
+
+
+
+## Inference API
+
+To run inferences create an instance of `Model` class
+```python
+import torch
+from model import Model
+
+model = Model(device = "cuda", dtype = torch.float16)
+```
+
+---
+
+
+### Text-To-Video
+To directly call our text-to-video generator, run this python command which stores the result in `tmp/text2video/A_horse_galloping_on_a_street.mp4` :
+```python
+prompt = "A horse galloping on a street"
+params = {"t0": 44, "t1": 47 , "motion_field_strength_x" : 12, "motion_field_strength_y" : 12, "video_length": 8}
+
+out_path, fps = f"./text2video_{prompt.replace(' ','_')}.mp4", 4
+model.process_text2video(prompt, fps = fps, path = out_path, **params)
+```
+
+To use a different stable diffusion base model run this python command:
+```python
+from hf_utils import get_model_list
+model_list = get_model_list()
+for idx, name in enumerate(model_list):
+  print(idx, name)
+idx = int(input("Select the model by the listed number: ")) # select the model of your choice
+model.process_text2video(prompt, model_name = model_list[idx], fps = fps, path = out_path, **params)
+```
+
+
+#### Hyperparameters (Optional)
+
+You can define the following hyperparameters:
+* **Motion field strength**:   `motion_field_strength_x` = $\delta_x$  and `motion_field_strength_y` = $\delta_x$ (see our paper, Sect. 3.3.1). Default: `motion_field_strength_x=motion_field_strength_y= 12`.
+* $T$ and $T'$ (see our paper, Sect. 3.3.1). Define values `t0` and `t1` in the range `{0,...,50}`. Default: `t0=44`, `t1=47` (DDIM steps). Corresponds to timesteps `881` and `941`, respectively. 
+* **Video length**: Define the number of frames `video_length` to be generated. Default: `video_length=8`.
+
+
+---
+
+### Text-To-Video with Pose Control
+To directly call our text-to-video generator with pose control, run this python command:
+```python
+prompt = 'an astronaut dancing in outer space'
+motion_path = '__assets__/poses_skeleton_gifs/dance1_corr.mp4'
+out_path = f"./text2video_pose_guidance_{prompt.replace(' ','_')}.gif"
+model.process_controlnet_pose(motion_path, prompt=prompt, save_path=out_path)
+```
+
+
+---
+
+### Text-To-Video with Edge Control
+To directly call our text-to-video generator with edge control, run this python command:
+```python
+prompt = 'oil painting of a deer, a high-quality, detailed, and professional photo'
+video_path = '__assets__/canny_videos_mp4/deer.mp4'
+out_path = f'./text2video_edge_guidance_{prompt}.mp4'
+model.process_controlnet_canny(video_path, prompt=prompt, save_path=out_path)
+```
+
+#### Hyperparameters
+
+You can define the following hyperparameters for Canny edge detection:
+* **low threshold**. Define value `low_threshold` in the range $(0, 255)$. Default: `low_threshold=100`.
+* **high threshold**. Define value `high_threshold` in the range $(0, 255)$. Default: `high_threshold=200`. Make sure that `high_threshold` > `low_threshold`.
+
+You can give hyperparameters as arguments to `model.process_controlnet_canny`
+
+---
+
+
+### Text-To-Video with Edge Guidance and Dreambooth specialization
+Load a dreambooth model then proceed as described in `Text-To-Video with Edge Guidance`
+```python
+
+prompt = 'your prompt'
+video_path = 'path/to/your/video'
+dreambooth_model_path = 'path/to/your/dreambooth/model'
+out_path = f'./text2video_edge_db_{prompt}.gif'
+model.process_controlnet_canny_db(dreambooth_model_path, video_path, prompt=prompt, save_path=out_path)
+```
+
+The value `video_path` can be the path to a `mp4` file. To use one of the example videos provided, set `video_path="woman1"`, `video_path="woman2"`, `video_path="woman3"`, or `video_path="man1"`. 
+ 
+
+The value `dreambooth_model_path` can either be a link to a diffuser model file, or the name of one of the dreambooth models provided. To this end, set `dreambooth_model_path = "Anime DB"`, `dreambooth_model_path = "Avatar DB"`, `dreambooth_model_path = "GTA-5 DB"`, or `dreambooth_model_path = "Arcane DB"`.  The corresponding keywords are: `1girl` (for `Anime DB`), `arcane style` (for `Arcane DB`) `avatar style` (for `Avatar DB`) and `gtav style`  (for `GTA-5 DB`).
+
+
+To load custom Dreambooth models, [transfer](https://github.com/lllyasviel/ControlNet/discussions/12) control to the custom model and  [convert](https://github.com/huggingface/diffusers/blob/main/scripts/convert_original_stable_diffusion_to_diffusers.py) it to diffuser format. Then, the value of `dreambooth_model_path` must link to the folder containing the diffuser file. Dreambooth models can be obtained, for instance, from [CIVITAI](https://civitai.com). 
+
+
+
+---
+
+
+
+### Video Instruct-Pix2Pix
+
+To perform pix2pix video editing, run this python command:
+```python
+prompt = 'make it Van Gogh Starry Night'
+video_path = '__assets__/pix2pix video/camel.mp4'
+out_path = f'./video_instruct_pix2pix_{prompt}.mp4'
+model.process_pix2pix(video_path, prompt=prompt, save_path=out_path)
+```
+
+---
+
+### Low Memory Inference
+Each of the above introduced interface can be run in a low memory setup. In the minimal setup, a GPU with **12 GB VRAM** is sufficient. 
+
+To reduce the memory usage, add `chunk_size=k` as additional parameter when calling one of the above defined inference APIs. The integer value `k` must be in the range `{2,...,video_length}`. It defines the number of frames that are processed at once (without any loss in quality). The lower the value the less memory is needed.
+
+When using the gradio app, set `chunk_size` in the `Advanced options`. 
+
+
+We plan to release soon a new version that further reduces the memory usage. 
+
+
 ---
-title: Text2Video-Zero
-emoji: 🚀
-colorFrom: green
-colorTo: blue
-sdk: gradio
-sdk_version: 3.23.0
-app_file: app.py
-pinned: false
-pipeline_tag: text-to-video
+
+
+### Ablation Study
+To replicate the ablation study, add additional parameters when calling the above defined inference APIs.
+*  To deactivate `cross-frame attention`: Add `use_cf_attn=False` to the parameter list.
+* To deactivate enriching latent codes with `motion dynamics`: Add `use_motion_field=False` to the parameter list.
+
+
+Note: Adding `smooth_bg=True` activates background smoothing. However, our  code does not include the salient object detector necessary to run that code.
+
+
 ---
 
-Paper: https://arxiv.org/abs/2303.13439
+## Inference using Gradio
+From the project root folder, run this shell command:
+```shell
+python app.py
+```
+
+Then access the app [locally](http://127.0.0.1:7860) with a browser.
+
+To access the app remotely, run this shell command:
+```shell
+python app.py --public_access
+```
+For security information about public access we refer to the documentation of [gradio](https://gradio.app/sharing-your-app/#security-and-file-access).
+
+
+
+## Results
+
+### Text-To-Video
+<table class="center">
+<tr>
+  <td><img src="__assets__/github/results/t2v/cat_running.gif" raw=true></td>
+  <td><img src="__assets__/github/results/t2v/playing.gif"></td>
+  <td><img src="__assets__/github/results/t2v/running.gif"></td>              
+  <td><img src="__assets__/github/results/t2v/skii.gif"></td>
+</tr>
+<tr>
+  <td width=25% align="center">"A cat is running on the grass"</td>
+  <td width=25% align="center">"A panda is playing guitar on times square"</td>
+  <td width=25% align="center">"A man is running in the snow"</td>
+  <td width=25% align="center">"An astronaut is skiing down the hill"</td>
+</tr>
+
+<tr>
+  <td><img src="__assets__/github/results/t2v/panda_surfing.gif" raw=true></td>
+  <td><img src="__assets__/github/results/t2v/bear_dancing.gif"></td>
+  <td><img src="__assets__/github/results/t2v/bicycle.gif"></td>              
+  <td><img src="__assets__/github/results/t2v/horse_galloping.gif"></td>
+</tr>
+<tr>
+  <td width=25% align="center">"A panda surfing on a wakeboard"</td>
+  <td width=25% align="center">"A bear dancing on times square"</td>
+  <td width=25% align="center">"A man is riding a bicycle in the sunshine"</td>
+  <td width=25% align="center">"A horse galloping on a street"</td>
+</tr>
+
+<tr>
+  <td><img src="__assets__/github/results/t2v/tiger_walking.gif" raw=true></td>
+  <td><img src="__assets__/github/results/t2v/panda_surfing_2.gif"></td>
+  <td><img src="__assets__/github/results/t2v/horse_galloping_2.gif"></td>              
+  <td><img src="__assets__/github/results/t2v/cat_walking.gif"></td>
+</tr>
+<tr>
+  <td width=25% align="center">"A tiger walking alone down the street"</td>
+  <td width=25% align="center">"A panda surfing on a wakeboard"</td>
+  <td width=25% align="center">"A horse galloping on a street"</td>
+  <td width=25% align="center">"A cute cat running in a beatiful meadow"</td>
+</tr>
+
+
+<tr>
+  <td><img src="__assets__/github/results/t2v/horse_galloping_3.gif" raw=true></td>
+  <td><img src="__assets__/github/results/t2v/panda_walking.gif"></td>
+  <td><img src="__assets__/github/results/t2v/dog_walking.gif"></td>              
+  <td><img src="__assets__/github/results/t2v/astronaut.gif"></td>
+</tr>
+<tr>
+  <td width=25% align="center">"A horse galloping on a street"</td>
+  <td width=25% align="center">"A panda walking alone down the street"</td>
+  <td width=25% align="center">"A dog is walking down the street"</td>
+  <td width=25% align="center">"An astronaut is waving his hands on the moon"</td>
+</tr>
+
+
+</table>
+
+### Text-To-Video with Pose Guidance
+
+
+<table class="center">
+<tr>
+  <td><img src="__assets__/github/results/pose2v/img_bot_left.gif" raw=true><img src="__assets__/github/results/pose2v/pose_bot_left.gif"></td>
+  <td><img src="__assets__/github/results/pose2v/img_bot_right.gif" raw=true><img src="__assets__/github/results/pose2v/pose_bot_right.gif"></td>
+  <td><img src="__assets__/github/results/pose2v/img_top_left.gif" raw=true><img src="__assets__/github/results/pose2v/pose_top_left.gif"></td>
+  <td><img src="__assets__/github/results/pose2v/img_top_right.gif" raw=true><img src="__assets__/github/results/pose2v/pose_top_right.gif"></td>
+</tr>
+<tr>
+  <td width=25% align="center">"A bear dancing on the concrete"</td>
+  <td width=25% align="center">"An alien dancing under a flying saucer"</td>
+  <td width=25% align="center">"A panda dancing in Antarctica"</td>
+  <td width=25% align="center">"An astronaut dancing in the outer space"</td>
+
+</tr>
+</table>
+
+### Text-To-Video with Edge Guidance
+
+
+
+<table class="center">
+<tr>
+  <td><img src="__assets__/github/results/edge2v/butterfly.gif" raw=true><img src="__assets__/github/results/edge2v/butterfly_edge.gif"></td>
+  <td><img src="__assets__/github/results/edge2v/head.gif" raw=true><img src="__assets__/github/results/edge2v/head_edge.gif"></td>
+  <td><img src="__assets__/github/results/edge2v/jelly.gif" raw=true><img src="__assets__/github/results/edge2v/jelly_edge.gif"></td>
+  <td><img src="__assets__/github/results/edge2v/mask.gif" raw=true><img src="__assets__/github/results/edge2v/mask_edge.gif"></td>
+</tr>
+<tr>
+  <td width=25% align="center">"White butterfly"</td>
+  <td width=25% align="center">"Beautiful girl"</td>
+    <td width=25% align="center">"A jellyfish"</td>
+  <td width=25% align="center">"beautiful girl halloween style"</td>
+</tr>
+
+<tr>
+  <td><img src="__assets__/github/results/edge2v/fox.gif" raw=true><img src="__assets__/github/results/edge2v/fix_edge.gif"></td>
+  <td><img src="__assets__/github/results/edge2v/head_2.gif" raw=true><img src="__assets__/github/results/edge2v/head_2_edge.gif"></td>
+  <td><img src="__assets__/github/results/edge2v/santa.gif" raw=true><img src="__assets__/github/results/edge2v/santa_edge.gif"></td>
+  <td><img src="__assets__/github/results/edge2v/dear.gif" raw=true><img src="__assets__/github/results/edge2v/dear_edge.gif"></td>
+</tr>
+<tr>
+  <td width=25% align="center">"Wild fox is walking"</td>
+  <td width=25% align="center">"Oil painting of a beautiful girl close-up"</td>
+    <td width=25% align="center">"A santa claus"</td>
+  <td width=25% align="center">"A deer"</td>
+</tr>
+
+</table>
+
+
+### Text-To-Video with Edge Guidance and Dreambooth specialization
+
+
+
+
+<table class="center">
+<tr>
+  <td><img src="__assets__/github/results/canny_db/anime_style.gif" raw=true><img src="__assets__/github/results/canny_db/anime_edge.gif"></td>
+  <td><img src="__assets__/github/results/canny_db/arcane_style.gif" raw=true><img src="__assets__/github/results/canny_db/arcane_edge.gif"></td>
+  <td><img src="__assets__/github/results/canny_db/gta-5_man_style.gif" raw=true><img src="__assets__/github/results/canny_db/gta-5_man_edge.gif"></td>
+  <td><img src="__assets__/github/results/canny_db/img_bot_right.gif" raw=true><img src="__assets__/github/results/canny_db/edge_bot_right.gif"></td>
+</tr>
+<tr>
+  <td width=25% align="center">"anime style"</td>
+  <td width=25% align="center">"arcane style"</td>
+    <td width=25% align="center">"gta-5 man"</td>
+  <td width=25% align="center">"avatar style"</td>
+</tr>
+
+</table>
+
+
+## Video Instruct Pix2Pix
+
+<table class="center">
+<tr>
+  <td><img src="__assets__/github/results/Video_InstructPix2Pix/frame_1/up_left.gif" raw=true><img src="__assets__/github/results/Video_InstructPix2Pix/frame_1/bot_left.gif"></td>
+  <td><img src="__assets__/github/results/Video_InstructPix2Pix/frame_1/up_mid.gif" raw=true><img src="__assets__/github/results/Video_InstructPix2Pix/frame_1/bot_mid.gif"></td>
+  <td><img src="__assets__/github/results/Video_InstructPix2Pix/frame_1/up_right.gif" raw=true><img src="__assets__/github/results/Video_InstructPix2Pix/frame_1/bot_right.gif"></td>
+</tr>
+<tr>
+  <td width=25% align="center">"Replace man with chimpanze"</td>
+  <td width=25% align="center">"Make it Van Gogh Starry Night style"</td>
+    <td width=25% align="center">"Make it Picasso style"</td>
+</tr>
+
+<tr>
+  <td><img src="__assets__/github/results/Video_InstructPix2Pix/frame_2/up_left.gif" raw=true><img src="__assets__/github/results/Video_InstructPix2Pix/frame_2/bot_left.gif"></td>
+  <td><img src="__assets__/github/results/Video_InstructPix2Pix/frame_2/up_mid.gif" raw=true><img src="__assets__/github/results/Video_InstructPix2Pix/frame_2/bot_mid.gif"></td>
+  <td><img src="__assets__/github/results/Video_InstructPix2Pix/frame_2/up_right.gif" raw=true><img src="__assets__/github/results/Video_InstructPix2Pix/frame_2/bot_right.gif"></td>
+</tr>
+<tr>
+  <td width=25% align="center">"Make it Expressionism style"</td>
+  <td width=25% align="center">"Make it night"</td>
+    <td width=25% align="center">"Make it autumn"</td>
+</tr>
+</table>
+
+
+## Related Links 
+
+* [High-Resolution Image Synthesis with Latent Diffusion Models (a.k.a. LDM & Stable Diffusion)](https://ommer-lab.com/research/latent-diffusion-models/)
+* [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://www.timothybrooks.com/instruct-pix2pix/)
+* [Adding Conditional Control to Text-to-Image Diffusion Models (a.k.a ControlNet)](https://github.com/lllyasviel/ControlNet)
+* [Diffusers](https://github.com/huggingface/diffusers)
+
+## License
+Our code is published under the CreativeML Open RAIL-M license. The license provided in this repository applies to all additions and contributions we make upon the original stable diffusion code. The original stable diffusion code is under the CreativeML Open RAIL-M license, which can found [here](https://github.com/CompVis/stable-diffusion/blob/main/LICENSE).
+
+
+## BibTeX
+If you use our work in your research, please cite our publication:
+```
+@article{text2video-zero,
+    title={Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators},
+    author={Khachatryan, Levon and Movsisyan, Andranik and Tadevosyan, Vahram and Henschel, Roberto and Wang, Zhangyang and Navasardyan, Shant and Shi, Humphrey},
+    journal={arXiv preprint arXiv:2303.13439},
+    year={2023}
+}
+```

app.py

@@ -2,15 +2,21 @@ import gradio as gr
 import torch
 
 from model import Model, ModelType
-
 from app_canny import create_demo as create_demo_canny
 from app_pose import create_demo as create_demo_pose
 from app_text_to_video import create_demo as create_demo_text_to_video
 from app_pix2pix_video import create_demo as create_demo_pix2pix_video
 from app_canny_db import create_demo as create_demo_canny_db
+import argparse
+import os
 
-
+on_huggingspace = os.environ.get("SPACE_AUTHOR_NAME") == "PAIR"
 model = Model(device='cuda', dtype=torch.float16)
+parser = argparse.ArgumentParser()
+parser.add_argument('--public_access', action='store_true',
+                    help="if enabled, the app can be access from a public url", default=False)
+args = parser.parse_args()
+
 
 with gr.Blocks(css='style.css') as demo:
     gr.HTML(
@@ -38,13 +44,13 @@ with gr.Blocks(css='style.css') as demo:
         </div>
         """)
 
-    
-    gr.HTML("""
-    <p>For faster inference without waiting in queue, you may duplicate the space and upgrade to GPU in settings.
-    <br/>
-    <a href="https://huggingface.co/spaces/PAIR/Text2Video-Zero?duplicate=true">
-    <img style="margin-top: 0em; margin-bottom: 0em" src="https://bit.ly/3gLdBN6" alt="Duplicate Space"></a>
-    </p>""")
+    if on_huggingspace:
+        gr.HTML("""
+        <p>For faster inference without waiting in queue, you may duplicate the space and upgrade to GPU in settings.
+        <br/>
+        <a href="https://huggingface.co/spaces/PAIR/Text2Video-Zero?duplicate=true">
+        <img style="margin-top: 0em; margin-bottom: 0em" src="https://bit.ly/3gLdBN6" alt="Duplicate Space"></a>
+        </p>""")
 
     with gr.Tab('Zero-Shot Text2Video'):
         create_demo_text_to_video(model)
@@ -78,6 +84,7 @@ with gr.Blocks(css='style.css') as demo:
         </div>
         """)
 
-demo.queue(max_size=20)
-demo.launch(debug=True)
-# demo.queue(api_open=False).launch(file_directories=['temporal'], share=True)
+
+_, _, link = demo.queue(api_open=False).launch(
+    file_directories=['temporal'], share=args.public_access or on_huggingspace)
+print(link)

app_canny.py

@@ -1,16 +1,26 @@
 import gradio as gr
 from model import Model
+import os
+on_huggingspace = os.environ.get("SPACE_AUTHOR_NAME") == "PAIR"
+
 
 def create_demo(model: Model):
 
     examples = [
-        ["__assets__/canny_videos_edge_2fps/butterfly.mp4", "white butterfly, a high-quality, detailed, and professional photo"],
-        ["__assets__/canny_videos_edge_2fps/deer.mp4", "oil painting of a deer, a high-quality, detailed, and professional photo"],
-        ["__assets__/canny_videos_edge_2fps/fox.mp4", "wild red fox is walking on the grass, a high-quality, detailed, and professional photo"],
-        ["__assets__/canny_videos_edge_2fps/girl_dancing.mp4", "oil painting of a girl dancing close-up, masterpiece, a high-quality, detailed, and professional photo"],
-        ["__assets__/canny_videos_edge_2fps/girl_turning.mp4", "oil painting of a beautiful girl, a high-quality, detailed, and professional photo"],
-        ["__assets__/canny_videos_edge_2fps/halloween.mp4", "beautiful girl halloween style, a high-quality, detailed, and professional photo"],
-        ["__assets__/canny_videos_edge_2fps/santa.mp4", "a santa claus, a high-quality, detailed, and professional photo"],
+        ["__assets__/canny_videos_edge_2fps/butterfly.mp4",
+            "white butterfly, a high-quality, detailed, and professional photo"],
+        ["__assets__/canny_videos_edge_2fps/deer.mp4",
+            "oil painting of a deer, a high-quality, detailed, and professional photo"],
+        ["__assets__/canny_videos_edge_2fps/fox.mp4",
+            "wild red fox is walking on the grass, a high-quality, detailed, and professional photo"],
+        ["__assets__/canny_videos_edge_2fps/girl_dancing.mp4",
+            "oil painting of a girl dancing close-up, masterpiece, a high-quality, detailed, and professional photo"],
+        ["__assets__/canny_videos_edge_2fps/girl_turning.mp4",
+            "oil painting of a beautiful girl, a high-quality, detailed, and professional photo"],
+        ["__assets__/canny_videos_edge_2fps/halloween.mp4",
+            "beautiful girl halloween style, a high-quality, detailed, and professional photo"],
+        ["__assets__/canny_videos_edge_2fps/santa.mp4",
+            "a santa claus, a high-quality, detailed, and professional photo"],
     ]
 
     with gr.Blocks() as demo:
@@ -21,7 +31,7 @@ def create_demo(model: Model):
                 """
                 <div style="text-align: left; auto;">
                 <h2 style="font-weight: 450; font-size: 1rem; margin: 0rem">
-                    Description: For performance purposes, our current preview release supports any input videos but caps output videos to no longer than 15 seconds and the input videos are scaled down before processing.
+                    Description: For performance purposes, our current preview release supports any input videos but caps output videos after 80 frames and the input videos are scaled down before processing.
                 </h3>
                 </div>
                 """)
@@ -34,8 +44,10 @@ def create_demo(model: Model):
                 prompt = gr.Textbox(label='Prompt')
                 run_button = gr.Button(label='Run')
                 with gr.Accordion('Advanced options', open=False):
-                    watermark = gr.Radio(["Picsart AI Research", "Text2Video-Zero", "None"], label="Watermark", value='Picsart AI Research')
-                    chunk_size = gr.Slider(label="Chunk size", minimum=2, maximum=8, value=8, step=1)
+                    watermark = gr.Radio(["Picsart AI Research", "Text2Video-Zero",
+                                         "None"], label="Watermark", value='Picsart AI Research')
+                    chunk_size = gr.Slider(
+                        label="Chunk size", minimum=2, maximum=16, value=12 if on_huggingspace else 8, step=1, visible=not on_huggingspace)
             with gr.Column():
                 result = gr.Video(label="Generated Video").style(height="auto")
 
@@ -50,7 +62,7 @@ def create_demo(model: Model):
                     inputs=inputs,
                     outputs=result,
                     fn=model.process_controlnet_canny,
-                    cache_examples = True,
+                    cache_examples=on_huggingspace,
                     run_on_click=False,
                     )
 

app_canny_db.py

@@ -1,6 +1,9 @@
 import gradio as gr
 from model import Model
 import gradio_utils
+import os
+on_huggingspace = os.environ.get("SPACE_AUTHOR_NAME") == "PAIR"
+
 
 examples = [
     ['Anime DB', "woman1", "Portrait of detailed 1girl, feminine, soldier cinematic shot on canon 5d ultra realistic skin intricate clothes accurate hands Rory Lewis Artgerm WLOP Jeremy Lipking Jane Ansell studio lighting"],
@@ -25,7 +28,8 @@ def create_demo(model: Model):
 
     with gr.Blocks() as demo:
         with gr.Row():
-            gr.Markdown('## Text, Canny-Edge and DreamBooth Conditional Video Generation')
+            gr.Markdown(
+                '## Text, Canny-Edge and DreamBooth Conditional Video Generation')
         with gr.Row():
             gr.HTML(
                 """
@@ -35,7 +39,6 @@ def create_demo(model: Model):
                 </h3>
                 </div>
                 """)
-
         with gr.Row():
             with gr.Column():
                 # input_video_path = gr.Video(source='upload', format="mp4", visible=False)
@@ -45,25 +48,30 @@ def create_demo(model: Model):
                 prompt = gr.Textbox(label='Prompt')
                 run_button = gr.Button(label='Run')
                 with gr.Accordion('Advanced options', open=False):
-                    watermark = gr.Radio(["Picsart AI Research", "Text2Video-Zero", "None"], label="Watermark", value='Picsart AI Research')
-                    chunk_size = gr.Slider(label="Chunk size", minimum=2, maximum=8, value=8, step=1)
+                    watermark = gr.Radio(["Picsart AI Research", "Text2Video-Zero",
+                                         "None"], label="Watermark", value='Picsart AI Research')
+                    chunk_size = gr.Slider(
+                        label="Chunk size", minimum=2, maximum=16, value=12 if on_huggingspace else 8, step=1, visible=not on_huggingspace)
             with gr.Column():
                 result = gr.Image(label="Generated Video").style(height=400)
 
         with gr.Row():
-            gallery_db = gr.Gallery(label="Db models", value=[('__assets__/db_files/anime.jpg', "anime"), ('__assets__/db_files/arcane.jpg', "Arcane"), ('__assets__/db_files/gta.jpg', "GTA-5 (Man)"), ('__assets__/db_files/avatar.jpg', "Avatar DB")]).style(grid=[4], height=50)
+            gallery_db = gr.Gallery(label="Db models", value=[('__assets__/db_files/anime.jpg', "anime"), ('__assets__/db_files/arcane.jpg', "Arcane"), (
+                '__assets__/db_files/gta.jpg', "GTA-5 (Man)"), ('__assets__/db_files/avatar.jpg', "Avatar DB")]).style(grid=[4], height=50)
         with gr.Row():
-            gallery_canny = gr.Gallery(label="Motions", value=[('__assets__/db_files/woman1.gif', "woman1"), ('__assets__/db_files/woman2.gif', "woman2"), ('__assets__/db_files/man1.gif', "man1"), ('__assets__/db_files/woman3.gif', "woman3")]).style(grid=[4], height=50)
-            predefined_motion = gr.Textbox(visible=False, label='One of the above defined motions')
+            gallery_canny = gr.Gallery(label="Motions", value=[('__assets__/db_files/woman1.gif', "woman1"), ('__assets__/db_files/woman2.gif', "woman2"), (
+                '__assets__/db_files/man1.gif', "man1"), ('__assets__/db_files/woman3.gif', "woman3")]).style(grid=[4], height=50)
 
         db_selection = gr.Textbox(label="DB Model", visible=False)
-        canny_selection = gr.Textbox(label="One of the above defined motions", visible=False)
+        canny_selection = gr.Textbox(
+            label="One of the above defined motions", visible=False)
 
         gallery_db.select(load_db_model, None, db_selection)
         gallery_canny.select(canny_select, None, canny_selection)
 
-        db_selection.change(on_db_selection_update,None,db_text_field)
-        canny_selection.change(on_canny_selection_update,None,canny_text_field)
+        db_selection.change(on_db_selection_update, None, db_text_field)
+        canny_selection.change(on_canny_selection_update,
+                               None, canny_text_field)
 
         inputs = [
             db_selection,
@@ -77,7 +85,7 @@ def create_demo(model: Model):
                     inputs=inputs,
                     outputs=result,
                     fn=model.process_controlnet_canny_db,
-                    cache_examples = True,
+                    cache_examples=on_huggingspace,
                     )
 
         run_button.click(fn=model.process_controlnet_canny_db,
@@ -86,7 +94,7 @@ def create_demo(model: Model):
     return demo
 
 
-def on_db_selection_update(evt : gr.EventData):
+def on_db_selection_update(evt: gr.EventData):
 
     return f"DB model: **{evt._data}**"
 

app_pix2pix_video.py

@@ -1,15 +1,23 @@
 import gradio as gr
 from model import Model
+import os
+on_huggingspace = os.environ.get("SPACE_AUTHOR_NAME") == "PAIR"
 
 
 def create_demo(model: Model):
     examples = [
-        ['__assets__/pix2pix_video_2fps/camel.mp4', 'make it Van Gogh Starry Night style', 512, 0, 1.0],
-        ['__assets__/pix2pix_video_2fps/mini-cooper.mp4', 'make it Picasso style', 512, 0, 1.5],
-        ['__assets__/pix2pix_video_2fps/snowboard.mp4', 'replace man with robot', 512, 0, 1.0],
-        ['__assets__/pix2pix_video_2fps/white-swan.mp4', 'replace swan with mallard', 512, 0, 1.5],
-        ['__assets__/pix2pix_video_2fps/boat.mp4', 'add city skyline in the background', 512, 0, 1.5],
-        ['__assets__/pix2pix_video_2fps/ballet.mp4', 'make her a golden sculpture', 512, 0, 1.0],
+        ['__assets__/pix2pix_video_2fps/camel.mp4',
+            'make it Van Gogh Starry Night style', 512, 0, 1.0],
+        ['__assets__/pix2pix_video_2fps/mini-cooper.mp4',
+            'make it Picasso style', 512, 0, 1.5],
+        ['__assets__/pix2pix_video_2fps/snowboard.mp4',
+            'replace man with robot', 512, 0, 1.0],
+        ['__assets__/pix2pix_video_2fps/white-swan.mp4',
+            'replace swan with mallard', 512, 0, 1.5],
+        ['__assets__/pix2pix_video_2fps/boat.mp4',
+            'add city skyline in the background', 512, 0, 1.5],
+        ['__assets__/pix2pix_video_2fps/ballet.mp4',
+            'make her a golden sculpture', 512, 0, 1.0],
     ]
     with gr.Blocks() as demo:
         with gr.Row():
@@ -19,19 +27,21 @@ def create_demo(model: Model):
                 """
                 <div style="text-align: left; auto;">
                 <h2 style="font-weight: 450; font-size: 1rem; margin: 0rem">
-                    Description: For performance purposes, our current preview release supports any input videos but caps output videos to no longer than 15 seconds and the input videos are scaled down before processing. For faster inference you can choose lower output frames per seconds from Advanced Options.
+                    Description: For performance purposes, our current preview release supports any input videos but caps output videos after 80 frames and the input videos are scaled down before processing. For faster inference you can choose lower output frames per seconds from Advanced Options.
                 </h3>
                 </div>
                 """)
 
         with gr.Row():
             with gr.Column():
-                input_image = gr.Video(label="Input Video",source='upload', type='numpy', format="mp4", visible=True).style(height="auto")
+                input_image = gr.Video(label="Input Video", source='upload',
+                                       type='numpy', format="mp4", visible=True).style(height="auto")
             with gr.Column():
                 prompt = gr.Textbox(label='Prompt')
                 run_button = gr.Button(label='Run')
                 with gr.Accordion('Advanced options', open=False):
-                    watermark = gr.Radio(["Picsart AI Research", "Text2Video-Zero", "None"], label="Watermark", value='Picsart AI Research')
+                    watermark = gr.Radio(["Picsart AI Research", "Text2Video-Zero",
+                                         "None"], label="Watermark", value='Picsart AI Research')
                     image_resolution = gr.Slider(label='Image Resolution',
                                                  minimum=256,
                                                  maximum=1024,
@@ -43,18 +53,18 @@ def create_demo(model: Model):
                                      value=0,
                                      step=1)
                     image_guidance = gr.Slider(label='Image guidance scale',
-                                     minimum=0.5,
-                                     maximum=2,
-                                     value=1.0,
-                                     step=0.1)
+                                               minimum=0.5,
+                                               maximum=2,
+                                               value=1.0,
+                                               step=0.1)
                     start_t = gr.Slider(label='Starting time in seconds',
                                         minimum=0,
-                                        maximum=15,
+                                        maximum=10,
                                         value=0,
                                         step=1)
                     end_t = gr.Slider(label='End time in seconds (-1 corresponds to uploaded video duration)',
                                       minimum=0,
-                                      maximum=15,
+                                      maximum=10,
                                       value=-1,
                                       step=1)
                     out_fps = gr.Slider(label='Output video fps (-1 corresponds to uploaded video fps)',
@@ -62,7 +72,8 @@ def create_demo(model: Model):
                                         maximum=30,
                                         value=-1,
                                         step=1)
-                    chunk_size = gr.Slider(label="Chunk size", minimum=2, maximum=8, value=8, step=1)
+                    chunk_size = gr.Slider(
+                        label="Chunk size", minimum=2, maximum=16, value=12 if on_huggingspace else 8, step=1, visible=not on_huggingspace)
             with gr.Column():
                 result = gr.Video(label='Output', show_label=True)
         inputs = [
@@ -82,7 +93,7 @@ def create_demo(model: Model):
                     inputs=inputs,
                     outputs=result,
                     fn=model.process_pix2pix,
-                    cache_examples=True,
+                    cache_examples=on_huggingspace,
                     run_on_click=False,
                     )
 

app_pose.py

@@ -1,7 +1,7 @@
+from model import Model
 import gradio as gr
 import os
-
-from model import Model
+on_huggingspace = os.environ.get("SPACE_AUTHOR_NAME") == "PAIR"
 
 examples = [
     ['Motion 1', "An astronaut dancing in the outer space"],
@@ -11,29 +11,38 @@ examples = [
     ['Motion 5', "An astronaut dancing in the outer space"],
 ]
 
+
 def create_demo(model: Model):
     with gr.Blocks() as demo:
         with gr.Row():
             gr.Markdown('## Text and Pose Conditional Video Generation')
 
         with gr.Row():
-            gr.Markdown('Selection: **one motion** and a **prompt**, or use the examples below.')
+            gr.Markdown(
+                'Selection: **one motion** and a **prompt**, or use the examples below.')
             with gr.Column():
-                gallery_pose_sequence = gr.Gallery(label="Pose Sequence", value=[('__assets__/poses_skeleton_gifs/dance1.gif', "Motion 1"), ('__assets__/poses_skeleton_gifs/dance2.gif', "Motion 2"), ('__assets__/poses_skeleton_gifs/dance3.gif', "Motion 3"), ('__assets__/poses_skeleton_gifs/dance4.gif', "Motion 4"), ('__assets__/poses_skeleton_gifs/dance5.gif', "Motion 5")]).style(grid=[2], height="auto")
-                input_video_path = gr.Textbox(label="Pose Sequence",visible=False,value="Motion 1")
+                gallery_pose_sequence = gr.Gallery(label="Pose Sequence", value=[('__assets__/poses_skeleton_gifs/dance1.gif', "Motion 1"), ('__assets__/poses_skeleton_gifs/dance2.gif', "Motion 2"), (
+                    '__assets__/poses_skeleton_gifs/dance3.gif', "Motion 3"), ('__assets__/poses_skeleton_gifs/dance4.gif', "Motion 4"), ('__assets__/poses_skeleton_gifs/dance5.gif', "Motion 5")]).style(grid=[2], height="auto")
+                input_video_path = gr.Textbox(
+                    label="Pose Sequence", visible=False, value="Motion 1")
                 gr.Markdown("## Selection")
-                pose_sequence_selector = gr.Markdown('Pose Sequence: **Motion 1**')
+                pose_sequence_selector = gr.Markdown(
+                    'Pose Sequence: **Motion 1**')
             with gr.Column():
                 prompt = gr.Textbox(label='Prompt')
                 run_button = gr.Button(label='Run')
                 with gr.Accordion('Advanced options', open=False):
-                    watermark = gr.Radio(["Picsart AI Research", "Text2Video-Zero", "None"], label="Watermark",value='Picsart AI Research')
-                    chunk_size = gr.Slider(label="Chunk size", minimum=2, maximum=8, value=8, step=1)
+                    watermark = gr.Radio(["Picsart AI Research", "Text2Video-Zero",
+                                         "None"], label="Watermark", value='Picsart AI Research')
+                    chunk_size = gr.Slider(
+                        label="Chunk size", minimum=2, maximum=16, value=12 if on_huggingspace else 8, step=1, visible=not on_huggingspace)
             with gr.Column():
                 result = gr.Image(label="Generated Video")
 
-        input_video_path.change(on_video_path_update, None, pose_sequence_selector)
-        gallery_pose_sequence.select(pose_gallery_callback, None, input_video_path)
+        input_video_path.change(on_video_path_update,
+                                None, pose_sequence_selector)
+        gallery_pose_sequence.select(
+            pose_gallery_callback, None, input_video_path)
         inputs = [
             input_video_path,
             prompt,
@@ -45,7 +54,7 @@ def create_demo(model: Model):
                     inputs=inputs,
                     outputs=result,
                     fn=model.process_controlnet_pose,
-                    cache_examples = True,
+                    cache_examples=on_huggingspace,
                     run_on_click=False,
                     )
 
@@ -61,4 +70,4 @@ def on_video_path_update(evt: gr.EventData):
 
 
 def pose_gallery_callback(evt: gr.SelectData):
-    return f"Motion {evt.index+1}"
+    return f"Motion {evt.index+1}"

app_text_to_video.py

@@ -1,8 +1,9 @@
 import gradio as gr
 from model import Model
-from functools import partial
-from bs4 import BeautifulSoup
-import requests
+import os
+from hf_utils import get_model_list
+
+on_huggingspace = os.environ.get("SPACE_AUTHOR_NAME") == "PAIR"
 
 examples = [
     ["an astronaut waving the arm on the moon"],
@@ -14,40 +15,8 @@ examples = [
     ["a gorilla walking alone down the street"],
     ["a gorilla dancing on times square"],
     ["A panda dancing dancing like crazy on Times Square"],
-    ]
-
-
-def model_url_list():
-    url_list = []
-    for i in range(0, 5):
-        url_list.append(f"https://huggingface.co/models?p={i}&sort=downloads&search=dreambooth")
-    return url_list
-
-def data_scraping(url_list):
-    model_list = []
-    for url in url_list:
-        response = requests.get(url)
-        soup = BeautifulSoup(response.text, "html.parser")
-        div_class = 'grid grid-cols-1 gap-5 2xl:grid-cols-2'
-        div = soup.find('div', {'class': div_class})
-        for a in div.find_all('a', href=True):
-            model_list.append(a['href'])
-    return model_list
-
-model_list = data_scraping(model_url_list())
-for i in range(len(model_list)):
-    model_list[i] = model_list[i][1:]
-
-best_model_list = [
-    "dreamlike-art/dreamlike-photoreal-2.0",
-    "dreamlike-art/dreamlike-diffusion-1.0",
-    "runwayml/stable-diffusion-v1-5",
-    "CompVis/stable-diffusion-v1-4",
-    "prompthero/openjourney",
 ]
 
-model_list = best_model_list + model_list
-
 
 def create_demo(model: Model):
 
@@ -59,7 +28,7 @@ def create_demo(model: Model):
                 """
                 <div style="text-align: left; auto;">
                 <h2 style="font-weight: 450; font-size: 1rem; margin: 0rem">
-                    Description: Simply input <b>any textual prompt</b> to generate videos right away and unleash your creativity and imagination! You can also select from the examples below. For performance purposes, our current preview release by default generates only 8 output frames and output 4s videos, but you can increase it from Advanced Options.
+                    Description: Simply input <b>any textual prompt</b> to generate videos right away and unleash your creativity and imagination! You can also select from the examples below. For performance purposes, our current preview release allows to generate up to 16 frames, which can be configured in the Advanced Options.
                 </h3>
                 </div>
                 """)
@@ -68,27 +37,39 @@ def create_demo(model: Model):
             with gr.Column():
                 model_name = gr.Dropdown(
                     label="Model",
-                    choices=model_list,
+                    choices=get_model_list(),
                     value="dreamlike-art/dreamlike-photoreal-2.0",
                 )
                 prompt = gr.Textbox(label='Prompt')
                 run_button = gr.Button(label='Run')
                 with gr.Accordion('Advanced options', open=False):
-                    watermark = gr.Radio(["Picsart AI Research", "Text2Video-Zero", "None"], label="Watermark", value='Picsart AI Research')
-
-                    video_length = gr.Number(label="Video length", value=8, min=2, precision=0)
-                    chunk_size = gr.Slider(label="Chunk size", minimum=2, maximum=32, value=8, step=1)
-
-                    motion_field_strength_x = gr.Slider(label='Global Translation $\delta_{x}$', minimum=-20, maximum=20, value=12, step=1)
-                    motion_field_strength_y = gr.Slider(label='Global Translation $\delta_{y}$', minimum=-20, maximum=20, value=12, step=1)
-
-                    t0 = gr.Slider(label="Timestep t0", minimum=0, maximum=49, value=44, step=1)
-                    t1 = gr.Slider(label="Timestep t1", minimum=0, maximum=49, value=47, step=1)
-
-                    n_prompt = gr.Textbox(label="Optional Negative Prompt", value='')
+                    watermark = gr.Radio(["Picsart AI Research", "Text2Video-Zero",
+                                         "None"], label="Watermark", value='Picsart AI Research')
+
+                    if on_huggingspace:
+                        video_length = gr.Slider(
+                            label="Video length", minimum=8, maximum=16, step=1)
+                    else:
+                        video_length = gr.Number(
+                            label="Video length", value=8, precision=0)
+                    chunk_size = gr.Slider(
+                        label="Chunk size", minimum=2, maximum=16, value=12 if on_huggingspace else 8, step=1, visible=not on_huggingspace)
+
+                    motion_field_strength_x = gr.Slider(
+                        label='Global Translation $\delta_{x}$', minimum=-20, maximum=20, value=12, step=1)
+                    motion_field_strength_y = gr.Slider(
+                        label='Global Translation $\delta_{y}$', minimum=-20, maximum=20, value=12, step=1)
+
+                    t0 = gr.Slider(label="Timestep t0", minimum=0,
+                                   maximum=49, value=44, step=1)
+                    t1 = gr.Slider(label="Timestep t1", minimum=0,
+                                   maximum=49, value=47, step=1)
+
+                    n_prompt = gr.Textbox(
+                        label="Optional Negative Prompt", value='')
             with gr.Column():
                 result = gr.Video(label="Generated Video")
-                
+
         inputs = [
             prompt,
             model_name,
@@ -103,12 +84,12 @@ def create_demo(model: Model):
         ]
 
         gr.Examples(examples=examples,
-                inputs=inputs,
-                outputs=result,
-                fn=model.process_text2video,
-                # cache_examples=True,
-                run_on_click=False,
-        )
+                    inputs=inputs,
+                    outputs=result,
+                    fn=model.process_text2video,
+                    run_on_click=False,
+                    cache_examples=on_huggingspace,
+                    )
 
         run_button.click(fn=model.process_text2video,
                          inputs=inputs,

environment.yaml

@@ -0,0 +1,45 @@
+name: T2VZeroNew
+channels:
+  - pytorch
+  - defaults
+dependencies:
+  - python=3.8.5
+  - pip=22.3.1
+  - cudatoolkit=11.3
+  - pytorch=1.12.1
+  - torchvision=0.13.1
+  - pip:
+      - gradio==3.22.1
+      - albumentations==1.3.0
+      - opencv-contrib-python==4.3.0.36
+      - imageio==2.9.0
+      - imageio-ffmpeg==0.4.2
+      - pytorch-lightning==1.5.0
+      - omegaconf==2.3.0
+      - test-tube>=0.7.5
+      - streamlit==1.12.1
+      - einops==0.6.0
+      - transformers==4.26.0
+      - webdataset==0.2.5
+      - kornia==0.6
+      - open_clip_torch==2.16.0
+      - invisible-watermark>=0.1.5
+      - streamlit-drawable-canvas==0.8.0
+      - torchmetrics==0.6.0
+      - timm==0.6.12
+      - addict==2.4.0
+      - yapf==0.32.0
+      - prettytable==3.6.0
+      - safetensors==0.2.7
+      - basicsr==1.4.2
+      - accelerate==0.16.0
+      - decord==0.6.0
+      - diffusers==0.14.0
+      - moviepy==1.0.3
+      - opencv_python==4.7.0.68
+      - Pillow==9.4.0
+      - scikit_image==0.19.3
+      - scipy==1.10.1
+      - tensorboardX==2.6
+      - tqdm==4.64.1
+      - numpy==1.24.1

gradio_utils.py

@@ -1,4 +1,8 @@
+import os
+
 # App Canny utils
+
+
 def edge_path_to_video_path(edge_path):
     video_path = edge_path
 
@@ -17,9 +21,12 @@ def edge_path_to_video_path(edge_path):
         video_path = "__assets__/canny_videos_mp4_2fps/halloween.mp4"
     elif vid_name == "santa.mp4":
         video_path = "__assets__/canny_videos_mp4_2fps/santa.mp4"
+
+    assert os.path.isfile(video_path)
     return video_path
 
 
+# App Pose utils
 def motion_to_video_path(motion):
     videos = [
         "__assets__/poses_skeleton_gifs/dance1_corr.mp4",
@@ -49,8 +56,9 @@ def get_video_from_canny_selection(canny_selection):
     elif canny_selection == "woman3":
         input_video_path = "__assets__/db_files_2fps/woman3.mp4"
     else:
-        raise Exception
+        input_video_path = canny_selection
 
+    assert os.path.isfile(input_video_path)
     return input_video_path
 
 
@@ -64,7 +72,8 @@ def get_model_from_db_selection(db_selection):
     elif db_selection == "Arcane DB":
         input_video_path = 'PAIR/controlnet-canny-arcane'
     else:
-        raise Exception
+        input_video_path = db_selection
+
     return input_video_path
 
 

hf_utils.py

@@ -0,0 +1,39 @@
+from bs4 import BeautifulSoup
+import requests
+
+
+def model_url_list():
+    url_list = []
+    for i in range(0, 5):
+        url_list.append(
+            f"https://huggingface.co/models?p={i}&sort=downloads&search=dreambooth")
+    return url_list
+
+
+def data_scraping(url_list):
+    model_list = []
+    for url in url_list:
+        response = requests.get(url)
+        soup = BeautifulSoup(response.text, "html.parser")
+        div_class = 'grid grid-cols-1 gap-5 2xl:grid-cols-2'
+        div = soup.find('div', {'class': div_class})
+        for a in div.find_all('a', href=True):
+            model_list.append(a['href'])
+    return model_list
+
+
+def get_model_list():
+    model_list = data_scraping(model_url_list())
+    for i in range(len(model_list)):
+        model_list[i] = model_list[i][1:]
+
+    best_model_list = [
+        "dreamlike-art/dreamlike-photoreal-2.0",
+        "dreamlike-art/dreamlike-diffusion-1.0",
+        "runwayml/stable-diffusion-v1-5",
+        "CompVis/stable-diffusion-v1-4",
+        "prompthero/openjourney",
+    ]
+
+    model_list = best_model_list + model_list
+    return model_list

model.py

@@ -3,15 +3,15 @@ import gc
 import numpy as np
 
 import torch
-import decord
+
 from diffusers import StableDiffusionInstructPix2PixPipeline, StableDiffusionControlNetPipeline, ControlNetModel, UNet2DConditionModel
 from diffusers.schedulers import EulerAncestralDiscreteScheduler, DDIMScheduler
-from text_to_video.text_to_video_pipeline import TextToVideoPipeline
+from text_to_video_pipeline import TextToVideoPipeline
 
 import utils
 import gradio_utils
-
-# decord.bridge.set_bridge('torch')
+import os
+on_huggingspace = os.environ.get("SPACE_AUTHOR_NAME") == "PAIR"
 
 
 class ModelType(Enum):
@@ -34,9 +34,12 @@ class Model:
             ModelType.ControlNetCannyDB: StableDiffusionControlNetPipeline,
             ModelType.ControlNetPose: StableDiffusionControlNetPipeline,
         }
-        self.controlnet_attn_proc = utils.CrossFrameAttnProcessor(unet_chunk_size=2)
-        self.pix2pix_attn_proc = utils.CrossFrameAttnProcessor(unet_chunk_size=3)
-        self.text2video_attn_proc = utils.CrossFrameAttnProcessor(unet_chunk_size=2)
+        self.controlnet_attn_proc = utils.CrossFrameAttnProcessor(
+            unet_chunk_size=2)
+        self.pix2pix_attn_proc = utils.CrossFrameAttnProcessor(
+            unet_chunk_size=3)
+        self.text2video_attn_proc = utils.CrossFrameAttnProcessor(
+            unet_chunk_size=2)
 
         self.pipe = None
         self.model_type = None
@@ -49,7 +52,8 @@ class Model:
         torch.cuda.empty_cache()
         gc.collect()
         safety_checker = kwargs.pop('safety_checker', None)
-        self.pipe = self.pipe_dict[model_type].from_pretrained(model_id, safety_checker=safety_checker, **kwargs).to(self.device).to(self.dtype)
+        self.pipe = self.pipe_dict[model_type].from_pretrained(
+            model_id, safety_checker=safety_checker, **kwargs).to(self.device).to(self.dtype)
         self.model_type = model_type
 
     def inference_chunk(self, frame_ids, **kwargs):
@@ -90,6 +94,8 @@ class Model:
         prompt = [kwargs.pop('prompt')] * f
         negative_prompt = [kwargs.pop('negative_prompt', '')] * f
 
+        frames_counter = 0
+
         # Processing chunk-by-chunk
         if split_to_chunks:
             chunk_ids = np.arange(0, f, chunk_size - 1)
@@ -104,6 +110,9 @@ class Model:
                                                    prompt=prompt,
                                                    negative_prompt=negative_prompt,
                                                    **kwargs).images[1:])
+                frames_counter += len(chunk_ids)-1
+                if on_huggingspace and frames_counter >= 80:
+                    break
             result = np.concatenate(result)
             return result
         else:
@@ -126,8 +135,10 @@ class Model:
                                  save_path=None):
         video_path = gradio_utils.edge_path_to_video_path(video_path)
         if self.model_type != ModelType.ControlNetCanny:
-            controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny")
-            self.set_model(ModelType.ControlNetCanny,model_id="runwayml/stable-diffusion-v1-5", controlnet=controlnet)
+            controlnet = ControlNetModel.from_pretrained(
+                "lllyasviel/sd-controlnet-canny")
+            self.set_model(ModelType.ControlNetCanny,
+                           model_id="runwayml/stable-diffusion-v1-5", controlnet=controlnet)
             self.pipe.scheduler = DDIMScheduler.from_config(
                 self.pipe.scheduler.config)
             if use_cf_attn:
@@ -140,7 +151,7 @@ class Model:
         negative_prompts = 'longbody, lowres, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality'
 
         video, fps = utils.prepare_video(
-            video_path, resolution, self.device, self.dtype, False, 0, 15)
+            video_path, resolution, self.device, self.dtype, False)
         control = utils.pre_process_canny(
             video, low_threshold, high_threshold).to(self.device).to(self.dtype)
         f, _, h, w = video.shape
@@ -180,8 +191,10 @@ class Model:
                                 save_path=None):
         video_path = gradio_utils.motion_to_video_path(video_path)
         if self.model_type != ModelType.ControlNetPose:
-            controlnet = ControlNetModel.from_pretrained("fusing/stable-diffusion-v1-5-controlnet-openpose")
-            self.set_model(ModelType.ControlNetPose, model_id="runwayml/stable-diffusion-v1-5", controlnet=controlnet)
+            controlnet = ControlNetModel.from_pretrained(
+                "fusing/stable-diffusion-v1-5-controlnet-openpose")
+            self.set_model(ModelType.ControlNetPose,
+                           model_id="runwayml/stable-diffusion-v1-5", controlnet=controlnet)
             self.pipe.scheduler = DDIMScheduler.from_config(
                 self.pipe.scheduler.config)
             if use_cf_attn:
@@ -242,8 +255,10 @@ class Model:
         video_path = gradio_utils.get_video_from_canny_selection(video_path)
         # Load db and controlnet weights
         if 'db_path' not in self.states or db_path != self.states['db_path']:
-            controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny")
-            self.set_model(ModelType.ControlNetCannyDB, model_id=db_path, controlnet=controlnet)
+            controlnet = ControlNetModel.from_pretrained(
+                "lllyasviel/sd-controlnet-canny")
+            self.set_model(ModelType.ControlNetCannyDB,
+                           model_id=db_path, controlnet=controlnet)
             self.pipe.scheduler = DDIMScheduler.from_config(
                 self.pipe.scheduler.config)
             self.states['db_path'] = db_path
@@ -320,7 +335,7 @@ class Model:
 
     def process_text2video(self,
                            prompt,
-                           model_name,
+                           model_name="dreamlike-art/dreamlike-photoreal-2.0",
                            motion_field_strength_x=12,
                            motion_field_strength_y=12,
                            t0=44,
@@ -340,8 +355,10 @@ class Model:
                            path=None):
 
         if self.model_type != ModelType.Text2Video:
-            unet = UNet2DConditionModel.from_pretrained(model_name, subfolder="unet")
-            self.set_model(ModelType.Text2Video, model_id=model_name, unet=unet)
+            unet = UNet2DConditionModel.from_pretrained(
+                model_name, subfolder="unet")
+            self.set_model(ModelType.Text2Video,
+                           model_id=model_name, unet=unet)
             self.pipe.scheduler = DDIMScheduler.from_config(
                 self.pipe.scheduler.config)
             if use_cf_attn:

models/cldm_v15.yaml

@@ -1,81 +0,0 @@
-model:
-  target: cldm.cldm.ControlLDM
-  params:
-    linear_start: 0.00085
-    linear_end: 0.0120
-    num_timesteps_cond: 1
-    log_every_t: 200
-    timesteps: 1000
-    first_stage_key: "jpg"
-    cond_stage_key: "txt"
-    control_key: "hint"
-    image_size: 64
-    channels: 4
-    cond_stage_trainable: false
-    conditioning_key: crossattn
-    monitor: val/loss_simple_ema
-    scale_factor: 0.18215
-    use_ema: False
-    only_mid_control: False
-
-    control_stage_config:
-      target: cldm.cldm.ControlNet
-      params:
-        image_size: 32 # unused
-        in_channels: 4
-        hint_channels: 3
-        model_channels: 320
-        attention_resolutions: [ 4, 2, 1 ]
-        num_res_blocks: 2
-        channel_mult: [ 1, 2, 4, 4 ]
-        num_heads: 8
-        use_spatial_transformer: True
-        transformer_depth: 1
-        context_dim: 768
-        use_checkpoint: True
-        use_cf_attn: True
-        legacy: False
-
-    unet_config:
-      target: cldm.cldm.ControlledUnetModel
-      params:
-        image_size: 32 # unused
-        in_channels: 4
-        out_channels: 4
-        model_channels: 320
-        attention_resolutions: [ 4, 2, 1 ]
-        num_res_blocks: 2
-        channel_mult: [ 1, 2, 4, 4 ]
-        num_heads: 8
-        use_spatial_transformer: True
-        transformer_depth: 1
-        context_dim: 768
-        use_checkpoint: True
-        legacy: False
-        use_cf_attn: True
-
-    first_stage_config:
-      target: ldm.models.autoencoder.AutoencoderKL
-      params:
-        embed_dim: 4
-        monitor: val/rec_loss
-        ddconfig:
-          double_z: true
-          z_channels: 4
-          resolution: 256
-          in_channels: 3
-          out_ch: 3
-          ch: 128
-          ch_mult:
-          - 1
-          - 2
-          - 4
-          - 4
-          num_res_blocks: 2
-          attn_resolutions: []
-          dropout: 0.0
-        lossconfig:
-          target: torch.nn.Identity
-
-    cond_stage_config:
-      target: ldm.modules.encoders.modules.FrozenCLIPEmbedder

models/cldm_v15_no_cf_attn.yaml

@@ -1,81 +0,0 @@
-model:
-  target: cldm.cldm.ControlLDM
-  params:
-    linear_start: 0.00085
-    linear_end: 0.0120
-    num_timesteps_cond: 1
-    log_every_t: 200
-    timesteps: 1000
-    first_stage_key: "jpg"
-    cond_stage_key: "txt"
-    control_key: "hint"
-    image_size: 64
-    channels: 4
-    cond_stage_trainable: false
-    conditioning_key: crossattn
-    monitor: val/loss_simple_ema
-    scale_factor: 0.18215
-    use_ema: False
-    only_mid_control: False
-
-    control_stage_config:
-      target: cldm.cldm.ControlNet
-      params:
-        image_size: 32 # unused
-        in_channels: 4
-        hint_channels: 3
-        model_channels: 320
-        attention_resolutions: [ 4, 2, 1 ]
-        num_res_blocks: 2
-        channel_mult: [ 1, 2, 4, 4 ]
-        num_heads: 8
-        use_spatial_transformer: True
-        transformer_depth: 1
-        context_dim: 768
-        use_checkpoint: True
-        use_cf_attn: False
-        legacy: False
-
-    unet_config:
-      target: cldm.cldm.ControlledUnetModel
-      params:
-        image_size: 32 # unused
-        in_channels: 4
-        out_channels: 4
-        model_channels: 320
-        attention_resolutions: [ 4, 2, 1 ]
-        num_res_blocks: 2
-        channel_mult: [ 1, 2, 4, 4 ]
-        num_heads: 8
-        use_spatial_transformer: True
-        transformer_depth: 1
-        context_dim: 768
-        use_checkpoint: True
-        legacy: False
-        use_cf_attn: False
-
-    first_stage_config:
-      target: ldm.models.autoencoder.AutoencoderKL
-      params:
-        embed_dim: 4
-        monitor: val/rec_loss
-        ddconfig:
-          double_z: true
-          z_channels: 4
-          resolution: 256
-          in_channels: 3
-          out_ch: 3
-          ch: 128
-          ch_mult:
-          - 1
-          - 2
-          - 4
-          - 4
-          num_res_blocks: 2
-          attn_resolutions: []
-          dropout: 0.0
-        lossconfig:
-          target: torch.nn.Identity
-
-    cond_stage_config:
-      target: ldm.modules.encoders.modules.FrozenCLIPEmbedder

models/cldm_v21.yaml

@@ -1,85 +0,0 @@
-model:
-  target: cldm.cldm.ControlLDM
-  params:
-    linear_start: 0.00085
-    linear_end: 0.0120
-    num_timesteps_cond: 1
-    log_every_t: 200
-    timesteps: 1000
-    first_stage_key: "jpg"
-    cond_stage_key: "txt"
-    control_key: "hint"
-    image_size: 64
-    channels: 4
-    cond_stage_trainable: false
-    conditioning_key: crossattn
-    monitor: val/loss_simple_ema
-    scale_factor: 0.18215
-    use_ema: False
-    only_mid_control: False
-
-    control_stage_config:
-      target: cldm.cldm.ControlNet
-      params:
-        use_checkpoint: True
-        image_size: 32 # unused
-        in_channels: 4
-        hint_channels: 3
-        model_channels: 320
-        attention_resolutions: [ 4, 2, 1 ]
-        num_res_blocks: 2
-        channel_mult: [ 1, 2, 4, 4 ]
-        num_head_channels: 64 # need to fix for flash-attn
-        use_spatial_transformer: True
-        use_linear_in_transformer: True
-        transformer_depth: 1
-        context_dim: 1024
-        legacy: False
-
-    unet_config:
-      target: cldm.cldm.ControlledUnetModel
-      params:
-        use_checkpoint: True
-        image_size: 32 # unused
-        in_channels: 4
-        out_channels: 4
-        model_channels: 320
-        attention_resolutions: [ 4, 2, 1 ]
-        num_res_blocks: 2
-        channel_mult: [ 1, 2, 4, 4 ]
-        num_head_channels: 64 # need to fix for flash-attn
-        use_spatial_transformer: True
-        use_linear_in_transformer: True
-        transformer_depth: 1
-        context_dim: 1024
-        legacy: False
-
-    first_stage_config:
-      target: ldm.models.autoencoder.AutoencoderKL
-      params:
-        embed_dim: 4
-        monitor: val/rec_loss
-        ddconfig:
-          #attn_type: "vanilla-xformers"
-          double_z: true
-          z_channels: 4
-          resolution: 256
-          in_channels: 3
-          out_ch: 3
-          ch: 128
-          ch_mult:
-          - 1
-          - 2
-          - 4
-          - 4
-          num_res_blocks: 2
-          attn_resolutions: []
-          dropout: 0.0
-        lossconfig:
-          target: torch.nn.Identity
-
-    cond_stage_config:
-      target: ldm.modules.encoders.modules.FrozenOpenCLIPEmbedder
-      params:
-        freeze: True
-        layer: "penultimate"

requirements.txt

@@ -14,14 +14,17 @@ moviepy==1.0.3
 numpy==1.24.1
 omegaconf==2.3.0
 open_clip_torch==2.16.0
-opencv_python==4.7.0.68
-opencv-contrib-python==4.3.0.36
+opencv_python==4.7.0.72
+opencv-contrib-python==4.7.0.72
 Pillow==9.4.0
 pytorch_lightning==1.5.0
 prettytable==3.6.0
 scikit_image==0.19.3
 scipy==1.10.1
 tensorboardX==2.6
+torch==1.13.1
+torchvision==0.14.1
+torchmetrics==0.6.0
 tqdm==4.64.1
 timm==0.6.12
 transformers==4.26.0
@@ -29,7 +32,5 @@ test-tube>=0.7.5
 webdataset==0.2.5
 yapf==0.32.0
 safetensors==0.2.7
-huggingface-hub==0.13.0
-torch==1.13.1
-torchvision==0.14.1
 beautifulsoup4
+bs4

text_to_video/text_to_video_pipeline.py → text_to_video_pipeline.py

@@ -89,23 +89,6 @@ class TextToVideoPipeline(StableDiffusionPipeline):
         latents = latents * self.scheduler.init_noise_sigma
         return latents
 
-    def warp_latents_from_f0(self, latents, reference_flow):
-        _, _, H, W = reference_flow.size()
-        b, c, f, h, w = latents.size()
-        coords0 = coords_grid(f, H, W, device=latents.device).to(latents.dtype)
-        coords_t0 = coords0 + reference_flow
-        coords_t0[:, 0] /= W
-        coords_t0[:, 1] /= H
-        coords_t0 = coords_t0 * 2.0 - 1.0
-        coords_t0 = T.Resize((h, w))(coords_t0)
-        coords_t0 = rearrange(coords_t0, 'f c h w -> f h w c')
-        latents_0 = latents[:, :, 0]
-        latents_0 = latents_0.repeat(f, 1, 1, 1)
-        warped = grid_sample(latents_0, coords_t0,
-                             mode='nearest', padding_mode='reflection')
-        warped = rearrange(warped, '(b f) c h w -> b c f h w', f=f)
-        return warped
-
     def warp_latents_independently(self, latents, reference_flow, inject_noise=False):
         _, _, H, W = reference_flow.size()
         b, _, f, h, w = latents.size()