| # Latent Upscale | |
| <a href="https://ko-fi.com/ziyueli"><img src="https://storage.ko-fi.com/cdn/brandasset/kofi_button_blue.png?_gl=1*1lrplqr*_ga*MjA3NzkyMTU3Mi4xNjgxMDg2MjQw*_ga_M13FZ7VQ2C*MTY4MTA4NjI0Ni4xLjEuMTY4MTA4NzYxNi40NC4wLjA." alt="Ko-fi" width="180px"></a> | |
| Enhance the latent upscale options in the img2img process in [Automatic1111](https://github.com/AUTOMATIC1111/stable-diffusion-webui) to provide more flexibility and better image quality. | |
| ## Motivation | |
| 1. The current img2img process lacks the ability to select different options for Latent Upscale. | |
| 2. The default latent upscale method ("bilinear") often produces blurry images. | |
| This plugin introduces alternative interpolation methods for upscaling and offers different schedulers for the diffusion process, resulting in superior upscaled images. | |
| Moreover, this plugin expands the upscale options available in the Latent Space, surpassing those offered by the "Hires Fix" for the txt2img process. | |
| ## Example Comparison | |
| Original Image: | |
|  | |
| The default latent upscale (choose Resize mode "latent upscale"): | |
|  | |
| Latent Upscale Plugin (Upscale method: "nearest-exact", Scheduler: "simple") | |
|  | |
| ## Installation | |
| You can find the "Latent Upscale" plugin in the **Available** section under the **Extensions** tab in the WebUI. | |
| Simply search for "Latent Upscale" in the extension search bar to locate it and click on "install". | |
| Alternatively, you can install this plugin by navigating to **Install from URL** under the **Extensions** tab, and then pasting this repo's URL and clicking on **Install**. | |
|  | |
| _Don't forget to go back to **Installed** and click on **Apply**._ | |
| ## Usage | |
|  | |
| At the bottom of the **img2img** tab, you can select the newly installed **Latent Upscale** script in the **Script** dropdown menu. | |
| To benefit from these enhancements, make sure you have the "Just resize (latent upscale)" option selected for **Resize mode**. | |
| Additionally, all the parameters present in the user interface remain applicable, alongside the new options provided by this plugin in "Upscale Method" and "Scheduler". | |
| ## Tips | |
| 1. Opting for "nearest", "nearest-exact", or "area" in the Upscale Method and "simple" or "normal" in the Scheduler can often lead to crisper images. | |
| However, don't hesitate to experiment with different choices to discover the best outcome. | |
| 2. If you want more details in the image, you can try to increase the number of steps in the diffusion process. | |
| However, this will also increase the time required to generate the image. | |
| ## Technical Details | |
| ### How Latent Upscale Works | |
| The original image is first encoded into the latent space, which is upscaled by the correct factor before being fed into the diffusion (de-noising) process, and then decoded to the upscaled image. | |
|  | |
| ### Automatic1111's Default Implementation | |
| There are two problems with the default implementation of the latent upscale process in Automatic111: | |
| 1. It uses the [“bilinear” method](https://en.wikipedia.org/wiki/Bilinear_interpolation), which often results in blurry upscaled images. | |
| It does not provide users with other options to choose from. | |
| 2. The number of steps users set was applied to the full scheduler, effectively reducing the steps for “de-noising”. | |
| For example, if the user sets the steps to be "30" and the de-noise strength is set to 0.4, the diffusion process will only run for 30*0.4 = 12 steps. | |
|  | |
| ### Improved Implementation | |
| 1. This plugin overrides the default `init` method for `StableDiffusionProcessingImg2Img` to include additional features: | |
| 1. It adds the option to choose the "Upscale Method" interpolation method when creating the latent image. | |
| 2. I turned on a fix that’s already in the code base: | |
| ```python | |
| opts.img2img_fix_steps = True | |
| ``` | |
| This ensures that the diffusion process runs for the correct number of steps, as specified by the user. | |
| It is unclear why this was not the default setting for img2img. | |
|  | |
| 2. This plugin assigns the `sampler_noise_scheduler_override` method for `StableDiffusionProcessingImg2Img` so that our custom schedulers can be used for the diffusion process. | |