File size: 3,028 Bytes
65d3a5e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 |
---
library_name: diffusers
base_model: stabilityai/stable-diffusion-xl-base-1.0
tags:
- text-to-image
license: openrail++
inference: false
---
# One More Step
One More Step (OMS) module was proposed in [One More Step: A Versatile Plug-and-Play Module for Rectifying Diffusion Schedule Flaws and Enhancing Low-Frequency Controls](https://github.com/mhh0318/OneMoreStep)
by *Minghui Hu, Jianbin Zheng, Chuanxia Zheng, Tat-Jen Cham et al.*
By **adding one small step** on the top the sampling process, we can address the issues caused by the current schedule flaws of diffusion models **without changing the original model parameters**. This also allows for some control over low-frequency information, such as color.
Our model is **versatile** and can be integrated into almost all widely-used Stable Diffusion frameworks. It's compatible with community favorites such as **LoRA, ControlNet, Adapter, and foundational models**.
## Usage
OMS now is supported 🤗 `diffusers` with a customized pipeline [github](https://github.com/mhh0318/OneMoreStep). To run the model (especially with `LCM` variant), first install the latest version of `diffusers` library as well as `accelerate` and `transformers`.
```bash
pip install --upgrade pip
pip install --upgrade diffusers transformers accelerate
```
And then we clone the repo
```bash
git clone https://github.com/mhh0318/OneMoreStep.git
cd OneMoreStep
```
### SD15 and SD21
Due to differences in the *VAE latent space* between SD1.5/SD2.1 and SDXL, the OMS module for SD1.5/SD2.1 cannot be shared with SDXL, **however, SD1.5/SD2.1 can share the same OMS module as well as with models like LCM that are based on SD1.5 or SD2.1.** For more details, please refer to our paper.
We have uploaded one OMS module for SD15/21 series at [h1t/oms_b_openclip_15_21](https://huggingface.co/h1t/oms_b_openclip_15_21), which has a base architecture, an OpenCLIP text encoder.
We simply put a demo here:
```python
import torch
from diffusers import StableDiffusionPipeline, LCMScheduler
sd_pipe = StableDiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-2-1", torch_dtype=torch.float16, variant="fp16", safety_checker=None).to('cuda')
pipe = OMSPipeline.from_pretrained('h1t/oms_b_openclip_15_21', sd_pipeline = sd_pipe, torch_dtype=torch.float16, variant="fp16", trust_remote_code=True)
pipe.to('cuda')
generator = torch.Generator(device=pipe.device).manual_seed(100)
prompt = "a starry night"
image = pipe(prompt, guidance_scale=7.5, num_inference_steps=20, oms_guidance_scale=2., generator=generator)
image['images'][0]
```
and without OMS:
```python
image = pipe(prompt, guidance_scale=7.5, num_inference_steps=20, oms_guidance_scale=2., generator=generator, oms_flag=False)
image['images'][0]
```
We found that the quality of the generative model has been greatly improved.
For more models and more functions like diverse prompt, please refer to [OMS Repo](https://github.com/mhh0318/OneMoreStep).
|