madebyollin
commited on
Commit
•
583db6c
1
Parent(s):
8a11549
Update README.md
Browse files
README.md
CHANGED
@@ -9,7 +9,20 @@ inference: false
|
|
9 |
|
10 |
SDXL-VAE-FP16-Fix is the [SDXL VAE](https://huggingface.co/stabilityai/sdxl-vae), but modified to run in fp16 precision without generating NaNs.
|
11 |
|
12 |
-
| VAE | Decoding in `float32` precision | Decoding in `float16` precision |
|
13 |
-
| --------------------- |
|
14 |
-
| SDXL-VAE | ✅ ![](./images/orig-fp32.png)
|
15 |
-
| SDXL-VAE-FP16-Fix | ✅ ![](./images/fix-fp32.png)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
9 |
|
10 |
SDXL-VAE-FP16-Fix is the [SDXL VAE](https://huggingface.co/stabilityai/sdxl-vae), but modified to run in fp16 precision without generating NaNs.
|
11 |
|
12 |
+
| VAE | Decoding in `float32` / `bfloat16` precision | Decoding in `float16` precision |
|
13 |
+
| --------------------- | -------------------------------------------- | ------------------------------- |
|
14 |
+
| SDXL-VAE | ✅ ![](./images/orig-fp32.png) | ⚠️ ![](./images/orig-fp16.png) |
|
15 |
+
| SDXL-VAE-FP16-Fix | ✅ ![](./images/fix-fp32.png) | ✅ ![](./images/fix-fp16.png) |
|
16 |
+
|
17 |
+
|
18 |
+
## Details
|
19 |
+
|
20 |
+
SDXL-VAE generates NaNs in fp16 because the internal activation values are too big:
|
21 |
+
![](./images/activation-magnitudes.jpg)
|
22 |
+
|
23 |
+
SDXL-VAE-FP16-Fix was created by finetuning the SDXL-VAE to:
|
24 |
+
1. keep the final output the same, but
|
25 |
+
2. make the internal activation values smaller, by
|
26 |
+
3. scaling down weights and biases within the network
|
27 |
+
|
28 |
+
There are slight discrepancies between the output of SDXL-VAE-FP16-Fix and SDXL-VAE, but the decoded images should be close enough for most purposes.
|