Introduction

The Stable Diffusion XL model is finetuned on comtemporatory Chinese ink paintings.

Usage

Our inference process is speed up using LCM-LORA, please make sure all the necessary libraries are up to date.

pip install --upgrade pip
pip install --upgrade diffusers transformers accelerate peft

Text to Image

Text-to-Image

Here, we should load two adapters, LCM-LORA for sample accleration and Chinese_Ink_LORA for styled rendering with it's base model stabilityai/stable-diffusion-xl-base-1.0. Next, the scheduler needs to be changed to LCMScheduler and we can reduce the number of inference steps to just 2 to 8 steps(8 used in my experiment).

import torch
from diffusers import DiffusionPipeline, LCMScheduler

pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0",
                                         variant="fp16",
                                         torch_dtype=torch.float16
                                         ).to("cuda")
# set scheduler
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)

# load LoRAs
pipe.load_lora_weights("latent-consistency/lcm-lora-sdxl", adapter_name="lcm")
pipe.load_lora_weights("ming-yang/sdxl_chinese_ink_lora", adapter_name="Chinese Ink")

# Combine LoRAs
pipe.set_adapters(["lcm", "Chinese Ink"], adapter_weights=[1.0, 0.8])

prompts = ["Chinese Ink, mona lisa picture, 8k", "mona lisa, 8k"]
generator = torch.manual_seed(1)
images = [pipe(prompt, num_inference_steps=8, guidance_scale=1, generator=generator).images[0] for prompt in prompts]

fig, axs = plt.subplots(1, 2, figsize=(40, 20))

axs[0].imshow(images[0])
axs[0].axis('off')  # 不显示坐标轴

axs[1].imshow(images[1])
axs[1].axis('off')
plt.show()

!(images/comparison.png)

tags: - text-to-image - stable-diffusion - lora - diffusers widget: - text: Chinese Ink, The girl with a pearl earring, 8k output: url: images/Chinese Ink, The girl with a pearl earring, 8k.png - text: Chinese Ink,a cute fox output: url: images/Chinese Ink,a cute fox.png - text: Chinese Ink, Mona Lisa, 8k output: url: images/Chinese Ink, Mona Lisa, 8k.png - text: Chinese Ink,lotus pond in summer rain output: url: images/Chinese Ink,lotus pond in summer rain.png - text: Chinese Ink, Wild Geese Descending on a Sandbank, 8k output: url: images/Chinese Ink, Wild Geese Descending on a Sandbank, 8k.png - text: Chinese Ink, the Paris skyline and the Eiffel Tower output: url: images/Chinese Ink, the Paris skyline and the Eiffel Tower.png - text: Chinese Ink, a lovely rabbit parameters: negative prompt: blurry, extra limb, bad anatomy output: url: images/Chinese Ink, a lovely rabbit.png - text: Chinese Ink, a tree with colorful leaves in autumn, 8k parameters: negative prompt: blurry, extra limb, bad anatomy output: url: images/a tree with colorful leaves in autumn.png base_model: stabilityai/stable-diffusion-xl-base-1.0 instance_prompt: Chinese Ink license: creativeml-openrail-m pipeline_tag: text-to-image

Chinese_Ink_Painting

Prompt
Chinese Ink, The girl with a pearl earring, 8k

Prompt
Chinese Ink,lotus pond in summer rain

Prompt
Chinese Ink, Wild Geese Descending on a Sandbank, 8k

Prompt
Chinese Ink, the Paris skyline and the Eiffel Tower

Prompt
Chinese Ink, a tree with colorful leaves in autumn, 8k

Trigger words

You should use Chinese Ink to trigger the image generation.

Download model

Weights for this model are available in Safetensors format.

Download them in the Files & versions tab.