LoRA text2image fine-tuning - sylvainlapeyrade/kanji2english

This is a model ran on only one epoch on Google Colab free version for proof of concepts purpose. Do not expect extraordinary resutls.

These are LoRA adaptation weights for CompVis/stable-diffusion-v1-4. The weights were fine-tuned on the sylvainlapeyrade/kanji_english_meaning dataset to generate Kanji images based on English descriptions. This model provides a novel approach to visualizing the artistic representation of Kanji characters through a text-to-image generation process, using the powerful Stable Diffusion architecture enhanced with LoRA layers for fine-tuning.

Model Description

This model uses LoRA (Low-Rank Adaptation) layers to fine-tune the Stable Diffusion v1-4 model specifically for the task of generating Kanji images from English text descriptions. The fine-tuning process was conducted using the sylvainlapeyrade/kanji_english_meaning dataset, which contains a collection of Kanji characters and their corresponding English meanings.

How to Use

This model is intended to be used with the diffusers library. Here is an example of how to generate an image from text:

from diffusers import LoraDiffusionPipeline

# Load the pipeline
pipe = LoraDiffusionPipeline.from_pretrained("sylvainlapeyrade/kanji2english")

# Generate an image
prompt = "a kanji meaning a Doge"
image = pipe(prompt).images[0]

# Save or display the image
image.save("kanji_doge.png")

Limitations and Bias

This model has been trained specifically for generating Kanji characters from English text inputs with the "a kanji meaning" sequence and may not generalize well to other text-to-image tasks outside this scope. Additionally, the accuracy of the generated images may vary depending on the complexity and rarity of the Kanji.

Acknowledgements

Special thanks to the creators of the KANJIDIC and KanjiVG projects for providing the data sources used during training.

License

This model is released under the CreativeML Open RAIL-M license, which allows for non-commercial use, sharing, and adaptation with attribution.