Diffusers?
Can this somehow be used in Diffusers, since the FluxPipeline currently does not support from_single_file?
No, the model structure is different, you would have to load the transformers weights in diffusers to fp8 and save that.
There is what looks like a conversion script on diffusers' github, but the crucial saving part does not seem to be implemented.
It might be a good reference for the file structure.
https://github.com/huggingface/diffusers/blob/main/scripts/convert_flux_to_diffusers.py
Apparently, Flux can be loaded by Diffusers if they are authenticated.
https://huggingface.co/spaces/gokaygokay/FLUX.1-Schnell-with-Captioner/blob/main/app.py
I think this can be made smaller with the following code.
from diffusers import DiffusionPipeline
hf_token = "hf_*****" # Your authenticated HF token
pipe = DiffusionPipeline.from_single_file("black-forest-labs/FLUX.1-schnell", use_safetensors=True, torch_dtype=torch.float8_e4m3fn, token=hf_token)
pipe.save_pretrained("FLUX1-schnell", safe_serialization=True, use_safetensors=True)
Please give complete code to use with FLUX schnell fp8 with diffusers.
Thanks
First of all, schnell, unlike dev, did not require a token, sorry.
If "schnell fp8" refers to loading the official schnell with fp8, the following code is sufficient.
import torch
from diffusers import FluxPipeline
pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", torch_dtype=torch.float8_e4m3fn)
pipe.enable_model_cpu_offload() #save some VRAM by offloading the model to CPU. Remove this if you have enough GPU power
prompt = "A cat holding a sign that says hello world"
image = pipe(
prompt,
guidance_scale=0.0,
num_inference_steps=4,
max_sequence_length=256,
generator=torch.Generator("cpu").manual_seed(0)
).images[0]
image.save("flux-schnell.png")
The problem is when you are referring to a file that has been compressed, split, quantized, etc., and the format has changed during conversion in ComfyUI.
The development version of Diffusers was recently fixed by sayakpaul, so it should now work if properly configured.
pip uninstall diffusers
pip install git+https://github.com/huggingface/diffusers
Note that the development version of Diffusers may have the same version number but different contents, so if you do not uninstall it once, it may remain out of date.
For use with Spaces, the latest version is installed every time a build is made anyway, so you don't need to worry about it...
For more practical use, it would be quicker to refer to popular spaces that are actually working.
https://huggingface.co/spaces/black-forest-labs/FLUX.1-schnell
https://huggingface.co/spaces/multimodalart/flux-lora-the-explorer
First of all, schnell, unlike dev, did not require a token, sorry.
If "schnell fp8" refers to loading the official schnell with fp8, the following code is sufficient.import torch from diffusers import FluxPipeline pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", torch_dtype=torch.float8_e4m3fn) pipe.enable_model_cpu_offload() #save some VRAM by offloading the model to CPU. Remove this if you have enough GPU power prompt = "A cat holding a sign that says hello world" image = pipe( prompt, guidance_scale=0.0, num_inference_steps=4, max_sequence_length=256, generator=torch.Generator("cpu").manual_seed(0) ).images[0] image.save("flux-schnell.png")The problem is when you are referring to a file that has been compressed, split, quantized, etc., and the format has changed during conversion in ComfyUI.
The development version of Diffusers was recently fixed by sayakpaul, so it should now work if properly configured.pip uninstall diffusers pip install git+https://github.com/huggingface/diffusersNote that the development version of Diffusers may have the same version number but different contents, so if you do not uninstall it once, it may remain out of date.
For use with Spaces, the latest version is installed every time a build is made anyway, so you don't need to worry about it...
got this error :
Traceback (most recent call last):
File "/workspace/flux8-2.py", line 4, in
pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", torch_dtype=torch.float8_e4m3fn)
File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 876, in from_pretrained
loaded_sub_model = load_sub_model(
File "/opt/conda/lib/python3.10/site-packages/diffusers/pipelines/pipeline_loading_utils.py", line 700, in load_sub_model
loaded_sub_model = load_method(os.path.join(cached_folder, name), **loading_kwargs)
File "/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3794, in from_pretrained
dtype_orig = cls._set_default_torch_dtype(torch_dtype)
File "/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1608, in _set_default_torch_dtype
torch.set_default_dtype(dtype)
File "/opt/conda/lib/python3.10/site-packages/torch/init.py", line 1009, in set_default_dtype
_C._set_default_dtype(d)
TypeError: couldn't find storage object Float8_e4m3fnStorage
Please suggest. Thanks
later version of torch is necessary.
pip install torch
pip install torchaudio
pip install torchvision
later version of torch is necessary.
pip install torch pip install torchaudio pip install torchvision
Got the same error
TypeError: couldn't find storage object Float8_e4m3fnStorage
even after
pip uninstall -y torch torchvision torchaudio;
pip install --upgrade torch torchvision torchaudio;
Thanks
Sorry, I forgot, transformers are a must.
I think it might work without accererate and xformers, just slows it down.
accelerate
git+https://github.com/huggingface/diffusers.git
invisible_watermark
torch
transformers==4.42.4
xformers
sentencepiece
Sorry, I forgot, transformers are a must.
I think it might work without accererate and xformers, just slows it down.accelerate git+https://github.com/huggingface/diffusers.git invisible_watermark torch transformers==4.42.4 xformers sentencepiece
I already had installed transformers-4.44.2
Thanks
Hmmm. Maybe, but I think the torch version is fixed to 2.2.0 or lower due to other library dependencies.
You have to have a fairly new (and potentially unstable) torch to support this data type.
It's treated as experimental.
Hmmm. Maybe, but I think the torch version is fixed to 2.2.0 or lower due to other library dependencies.
You have to have a fairly new (and potentially unstable) torch to support this data type.
It's treated as experimental.
I used this command :
pip uninstall -y torch torchvision torchaudio;pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118;pip install --upgrade diffusers transformers accelerate xformers omegaconf sentencepiece protobuf opencv-python torchao;sudo apt-get install -y libgl1-mesa-glx;
and it installed torch-2.4.0+cu118 torchaudio-2.4.0+cu118 torchvision-0.19.0+cu118
Please suggest. Thanks
It seems to have been a bug in the transformers.
I didn't notice because whenever I use Flux (too heavy for my PC) I always use the latest transformers in HF's VM.
https://github.com/huggingface/transformers/issues/32185
Solved it by run pip install transformers==4.43.2
Name: torch
Version: 2.4.0+cu124 OR Version: 2.3.1
Name: transformers
Version: 4.44.2
CUDA Version: 12.5
Error:
Traceback (most recent call last):
File "/home/ma/work/flux/demo/TotoroUI/flux-demo-hyper.py", line 552, in gen
img_path, image = gen_with_api(prompt, steps, guidance, width, height, seed)
File "/home/ma/work/flux/demo/TotoroUI/flux-demo-hyper.py", line 392, in gen_with_api
return generate2(prompt, steps, guidance, width, height, seed)
File "/home/ma/work/flux/demo/TotoroUI/flux-demo-hyper.py", line 513, in generate2
flux_pipe = FluxPipeline.from_pretrained(bfl_repo, transformer=transformer, torch_dtype=torch.float8_e4m3fn)
File "/home/ma/miniconda3/envs/flux/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/home/ma/miniconda3/envs/flux/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 859, in from_pretrained
loaded_sub_model = load_sub_model(
File "/home/ma/miniconda3/envs/flux/lib/python3.10/site-packages/diffusers/pipelines/pipeline_loading_utils.py", line 698, in load_sub_model
loaded_sub_model = load_method(os.path.join(cached_folder, name), **loading_kwargs)
File "/home/ma/miniconda3/envs/flux/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3794, in from_pretrained
dtype_orig = cls._set_default_torch_dtype(torch_dtype)
File "/home/ma/miniconda3/envs/flux/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1608, in _set_default_torch_dtype
torch.set_default_dtype(dtype)
File "/home/ma/miniconda3/envs/flux/lib/python3.10/site-packages/torch/init.py", line 1009, in set_default_dtype
_C._set_default_dtype(d)
TypeError: couldn't find storage object Float8_e4m3fnStorage
Hmmm... a mystery...
I have never run the Flux inference on my local PC, but have tried to save and load for conversion.
I had all sorts of glitches with the save at the time, but I didn't stumble through the load phase.
And I didn't do anything special about loading.
It's a little different from the actual code, but to summarize, here's what I did.
pip uninstall diffusers
pip install -U git+https://github.com/huggingface/diffusers
pip install -U git+https://github.com/huggingface/peft
pip install -U git+https://github.com/huggingface/transformers
pip install -U git+https://github.com/huggingface/accelerate
pip install -U git+https://github.com/huggingface/huggingface_hub
import torch
from diffusers import FluxPipeline
bfl_repo = "ChuckMcSneed/FLUX.1-dev"
dtype = torch.float8_e4m3fn
pipe = FluxPipeline.from_pretrained(bfl_repo, torch_dtype=dtype)
The following space may be helpful when loading special format Flux modules from Diffusers. Also lllyasviel and sayakpaul have a lot of information in the community section of their repos or in github.
How to use NF4 quantized FLUX.1 from Diffusers in Zero GPU space:
https://huggingface.co/spaces/nyanko7/flux1-dev-nf4/blob/main/app.py
https://huggingface.co/spaces/nyanko7/flux1-dev-nf4
I just thought of something, for example, can you load an SDXL or SD1.5 model with torch.float8_e4m3fn?
Normally it should be easy to do.
If you can do it but not with Flux, then it must be a bug in something Flux related.
@John6666
Earlier today, I was still able to use torch.float8_e4m3fn normally. However, I'm not sure what operation I performed that started causing errors. Even after reinstalling the entire environment, the error persists. It seems that I carried out the following actions:
pip uninstall diffusers
pip install git+https://github.com/huggingface/diffusers
So that's how it is!
The culprit is either the dev version, which is updated frequently, or HF itself if it is in HF's space.
HF and Diffusers in particular are highly suspect in the midst of the Zero GPU spec changes.
Dare we try to use the stable pip diffusers version of Diffusers?