|
--- |
|
license: other |
|
license_name: flux-1-dev-non-commercial-license |
|
tags: |
|
- text-to-image |
|
- SVDQuant |
|
- FLUX.1-dev |
|
- INT4 |
|
- FLUX.1 |
|
- Diffusion |
|
- Quantization |
|
- LoRA |
|
language: |
|
- en |
|
base_model: |
|
- mit-han-lab/svdq-int4-flux.1-dev |
|
- XLabs-AI/flux-RealismLora |
|
- aleksa-codes/flux-ghibsky-illustration |
|
- alvdansen/sonny-anime-fixed |
|
- Shakker-Labs/FLUX.1-dev-LoRA-Children-Simple-Sketch |
|
- linoyts/yarn_art_Flux_LoRA |
|
pipeline_tag: text-to-image |
|
datasets: |
|
- mit-han-lab/svdquant-datasets |
|
library_name: diffusers |
|
--- |
|
|
|
<p align="center" style="border-radius: 10px"> |
|
<img src="https://github.com/mit-han-lab/nunchaku/raw/refs/heads/main/assets/logo.svg" width="50%" alt="logo"/> |
|
</p> |
|
<h4 style="display: flex; justify-content: center; align-items: center; text-align: center;">Quantization Library: <a href='https://github.com/mit-han-lab/deepcompressor'>DeepCompressor</a>   Inference Engine: <a href='https://github.com/mit-han-lab/nunchaku'>Nunchaku</a> |
|
</h4> |
|
<div style="display: flex; justify-content: center; align-items: center; text-align: center;"> |
|
<a href="https://arxiv.org/abs/2411.05007">[Paper]</a>  |
|
<a href='https://github.com/mit-han-lab/nunchaku'>[Code]</a>  |
|
<a href='https://hanlab.mit.edu/projects/svdquant'>[Website]</a>  |
|
<a href='https://hanlab.mit.edu/blog/svdquant'>[Blog]</a> |
|
</div> |
|
|
|
 |
|
SVDQuant seamlessly integrates with off-the-shelf LoRAs without requiring re-quantization. When applying LoRAs, it matches the image quality of the original 16-bit FLUX.1-dev. |
|
|
|
## Model Description |
|
|
|
<div> |
|
This reposity contains a converted LoRA collection for SVDQuant INT4 FLUX.1-dev. The LoRA style includes <a href="https://huggingface.co/XLabs-AI/flux-RealismLora">Realism</a>, |
|
<a href="https://huggingface.co/aleksa-codes/flux-ghibsky-illustration">Ghibsky Illustration</a>, |
|
<a href="https://huggingface.co/alvdansen/sonny-anime-fixed">Anime</a>, |
|
<a href="https://huggingface.co/Shakker-Labs/FLUX.1-dev-LoRA-Children-Simple-Sketch">Children Sketch</a>, and |
|
<a href="https://huggingface.co/linoyts/yarn_art_Flux_LoRA">Yarn Art</a>. |
|
</div> |
|
|
|
|
|
## Usage |
|
|
|
### Diffusers |
|
|
|
Please follow the instructions in [mit-han-lab/nunchaku](https://github.com/mit-han-lab/nunchaku) to set up the environment. Then you can run the model with |
|
|
|
```python |
|
import torch |
|
|
|
from nunchaku.pipelines import flux as nunchaku_flux |
|
|
|
pipeline = nunchaku_flux.from_pretrained( |
|
"black-forest-labs/FLUX.1-dev", |
|
torch_dtype=torch.bfloat16, |
|
qmodel_path="mit-han-lab/svdq-int4-flux.1-dev", # download from Huggingface |
|
).to("cuda") |
|
pipeline.transformer.nunchaku_update_params(mit-han-lab/svdquant-models/svdq-flux.1-dev-lora-anime.safetensors) |
|
pipeline.transformer.nunchaku_set_lora_scale(1) |
|
image = pipeline("a dog wearing a wizard hat", num_inference_steps=28, guidance_scale=3.5).images[0] |
|
image.save("example.png") |
|
``` |
|
|
|
### Comfy UI |
|
|
|
Work in progress. |
|
|
|
## Limitations |
|
|
|
- The model is only runnable on NVIDIA GPUs with architectures sm_86 (Ampere: RTX 3090, A6000), sm_89 (Ada: RTX 4090), and sm_80 (A100). See this [issue](https://github.com/mit-han-lab/nunchaku/issues/1) for more details. |
|
- You may observe some slight differences from the BF16 models in details. |
|
|
|
### Citation |
|
|
|
If you find this model useful or relevant to your research, please cite |
|
|
|
```bibtex |
|
@article{ |
|
li2024svdquant, |
|
title={SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models}, |
|
author={Li*, Muyang and Lin*, Yujun and Zhang*, Zhekai and Cai, Tianle and Li, Xiuyu and Guo, Junxian and Xie, Enze and Meng, Chenlin and Zhu, Jun-Yan and Han, Song}, |
|
journal={arXiv preprint arXiv:2411.05007}, |
|
year={2024} |
|
} |
|
``` |
