keras-dreambooth/dreambooth_riffusion_model_currulao_v1

Model description

Riffusion-Currulao (WIP) is a fine-tuned version of the Riffusion model, which is based on the Stable Diffusion 1.5 framework. This model is specifically designed to generate spectrograms for Currulao music, a traditional Afro-Colombian genre characterized by the use of the Marimba de Chonta, a unique wooden percussion instrument.

Currulao is an essential part of the Afro-Colombian cultural heritage and represents the rich history, traditions, and artistic expressions of the Afro-Colombian community. By leveraging the power of the Riffusion model and the additional fine-tuning through Keras using the (DreamBooth)[https://dreambooth.github.io/] technique, Riffusion-Currulao is capable of generating spectrograms that capture the essence of Currulao music.

The model was further enhanced using DreamBooth, which allowed the integration of the concepts of Currulao and Marimba de Chonta, ensuring that the generated spectrograms somehow represent the musical characteristics of this genre.

Riffusion-Currulao can be used for various applications, including music synthesis, audio analysis, and research on Afro-Colombian music and culture. It can also serve as a creative tool for artists and musicians who wish to explore Currulao music and incorporate its elements into their compositions.

The model is designed to be accessible and user-friendly, making it suitable for both professionals and enthusiasts interested in generating spectrograms of Currulao music for their projects or studies.

Examples:

---

Prompt: A $currulao song, lo-fi, nostalgic.

Steps: 100

Guidance: 15

How to use it

You need to load the Riffusion weights to use this model. Example for Colab:

!git lfs install
!git clone https://huggingface.co/juancopi81/dreambooth_riffusion_model_currulao_v1

!pip install git+https://github.com/costiash/stable-diffusion-tensorflow --upgrade --quiet
!pip install tensorflow tensorflow_addons ftfy --upgrade --quiet
!pip install pytorch-lightning
!pip install pydub
!pip install huggingface-hub

!pip install -q -U keras_cv
!pip install -q -U tensorflow 

import tensorflow as tf 

from keras_cv.models.stable_diffusion.clip_tokenizer import SimpleTokenizer
from keras_cv.models.stable_diffusion.diffusion_model import DiffusionModel
from keras_cv.models.stable_diffusion.stable_diffusion import StableDiffusion as StableDiffusionKeras
from keras_cv.models.stable_diffusion.image_encoder import ImageEncoder
from keras_cv.models.stable_diffusion.noise_scheduler import NoiseScheduler
from keras_cv.models.stable_diffusion.text_encoder import TextEncoder
import keras

from stable_diffusion_tf.stable_diffusion import StableDiffusion as StableDiffusionPy

resolution=512
sd_dreambooth_model_1=StableDiffusionPy(resolution, resolution, download_weights=False, jit_compile=True)

diffusion_model_pytorch_weights = keras.utils.get_file(
    origin="https://huggingface.co/riffusion/riffusion-model-v1/resolve/main/riffusion-model-v1.ckpt",
    file_hash="99a6eb51c18e16a6121180f3daa69344e571618b195533f67ae94be4eb135a57",
)

sd_dreambooth_model_1.load_weights_from_pytorch_ckpt(diffusion_model_pytorch_weights)

sd_dreambooth_model_1.diffusion_model.load_weights("/content/dreambooth_riffusion_model_currulao_v1")

img = sd_dreambooth_model_1.generate(
    "A $currulao song, EDM",
    num_steps=100,
    unconditional_guidance_scale=12,
    temperature=1.2,
    batch_size=1,
)
pil_img = Image.fromarray(img[0])

Intended uses & limitations

Riffusion-Currulao (work in progress) will be intended for:

• Music synthesis: Generate new Currulao music pieces or augment existing compositions with Marimba de Chonta sounds.
• Audio analysis: Study the characteristics of Currulao music and its elements, such as rhythm, melody, and timbre, through the generated spectrograms.
• Research: Explore the Afro-Colombian cultural heritage and its musical expressions by analyzing the generated Currulao music.
• Creative inspiration: Use the model as a tool to inspire new musical ideas and compositions that incorporate the unique sounds and rhythms of Currulao music.

Always ensure that the use of generated spectrograms aligns with the intended purpose and respects the cultural significance of the Currulao music and Afro-Colombian heritage.

Training and evaluation data

This model was fine-tuned using the DreamBooth technique. The spectrograms for fine-tuning the model were generated from the song: "MUSICA PARA RELAJAR DEL PACÍFICO". The spectrograms were generated using the "Riffusion Manipulation Tools" repo.

Model Plot

View Model Plot