svjack/diffusers-sdxl-controlnet

AnimatedDiff ControlNet SDXL Example

This document provides a step-by-step guide to setting up and running the animatediff_controlnet_sdxl.py script from the Hugging Face repository. The script leverages the diffusers-sdxl-controlnet library to generate animated images using ControlNet and SDXL models.

Prerequisites

Before running the script, ensure you have the necessary dependencies installed. You can install them using the following commands:

System Dependencies

sudo apt-get update && sudo apt-get install git-lfs cbm ffmpeg

Python Dependencies

pip install git+https://huggingface.co/svjack/diffusers-sdxl-controlnet
pip install transformers peft sentencepiece moviepy controlnet_aux

Clone the Repository

git clone https://huggingface.co/svjack/diffusers-sdxl-controlnet
cp diffusers-sdxl-controlnet/girl-pose.gif . 

Script Modifications

The script requires some modifications to work correctly. Specifically, you need to comment out certain lines related to LoRA processors:

'''
drop     #LoRAAttnProcessor2_0,
    #LoRAXFormersAttnProcessor,
'''

GIF to Frames Conversion

The script includes a function to convert a GIF into individual frames. This is useful for preparing input data for the animation pipeline.

from PIL import Image, ImageSequence
import os

def gif_to_frames(gif_path, output_folder):
    # Open the GIF file
    gif = Image.open(gif_path)
    
    # Ensure the output folder exists
    if not os.path.exists(output_folder):
        os.makedirs(output_folder)
    
    # Iterate through each frame of the GIF
    for i, frame in enumerate(ImageSequence.Iterator(gif)):
        # Copy the frame
        frame_copy = frame.copy()
        
        # Save the frame to the specified folder
        frame_path = os.path.join(output_folder, f"frame_{i:04d}.png")
        frame_copy.save(frame_path)
    
    print(f"Successfully extracted {i + 1} frames to {output_folder}")

# Example call
gif_to_frames("girl-pose.gif", "girl_pose_frames")

Use this girl pose as pose source video (gif)

Running the Script

To run the script, follow these steps:

1. Add the Script Path to System Path:

   import sys
   sys.path.insert(0, "diffusers-sdxl-controlnet/examples/community/")
   from animatediff_controlnet_sdxl import *
   from controlnet_aux.processor import Processor
   

2. Load Necessary Libraries and Models:

   import torch
   from diffusers.models import MotionAdapter
   from diffusers import DDIMScheduler
   from diffusers.utils import export_to_gif
   from diffusers import AutoPipelineForText2Image, ControlNetModel
   from diffusers.utils import load_image
   from PIL import Image
   

3. Load the MotionAdapter Model:

   adapter = MotionAdapter.from_pretrained(
       "a-r-r-o-w/animatediff-motion-adapter-sdxl-beta", 
       torch_dtype=torch.float16
   )
   

4. Configure the Scheduler and ControlNet:

   model_id = "svjack/GenshinImpact_XL_Base"
   scheduler = DDIMScheduler.from_pretrained(
       model_id,
       subfolder="scheduler",
       clip_sample=False,
       timestep_spacing="linspace",
       beta_schedule="linear",
       steps_offset=1,
   )
   
   controlnet = ControlNetModel.from_pretrained(
       "thibaud/controlnet-openpose-sdxl-1.0",
       torch_dtype=torch.float16,
   ).to("cuda")
   

5. Load the AnimateDiffSDXLControlnetPipeline:

   pipe = AnimateDiffSDXLControlnetPipeline.from_pretrained(
       model_id,
       controlnet=controlnet,
       motion_adapter=adapter,
       scheduler=scheduler,
       torch_dtype=torch.float16,
   ).to("cuda")
   

6. Enable Memory Saving Features:

   pipe.enable_vae_slicing()
   pipe.enable_vae_tiling()
   

7. Load Conditioning Frames:

   import os
   folder_path = "girl_pose_frames/"
   frames = os.listdir(folder_path)
   frames = list(filter(lambda x: x.endswith(".png"), frames))
   frames.sort()
   conditioning_frames = list(map(lambda x: Image.open(os.path.join(folder_path ,x)).resize((1024, 1024)), frames))[:16]
   

8. Process Conditioning Frames:

   p2 = Processor("openpose")
   cn2 = [p2(frame) for frame in conditioning_frames]
   

9. Define Prompts:

   prompt = '''
   solo,Xiangling\(genshin impact\),1girl,
   full body professional photograph of a stunning detailed, sharp focus, dramatic 
   cinematic lighting, octane render  unreal engine (film grain, blurry background
   '''
   prompt = "solo,Xiangling\(genshin impact\),1girl,full body professional photograph of a stunning detailed"
   negative_prompt = "bad quality, worst quality, jpeg artifacts, ugly"
   

10. Generate Output: (Use Genshin Impact character Xiangling)

    prompt = '''
    solo,Xiangling\(genshin impact\),1girl,
    full body professional photograph of a stunning detailed, sharp focus, dramatic 
    cinematic lighting, octane render  unreal engine (film grain, blurry background
    '''
    prompt = "solo,Xiangling\(genshin impact\),1girl,full body professional photograph of a stunning detailed"
    
    #prompt = "solo,Xiangling\(genshin impact\),1girl"
    negative_prompt = "bad quality, worst quality, jpeg artifacts, ugly"
    
    generator = torch.Generator(device="cpu").manual_seed(0)
    output = pipe(
        prompt=prompt,
        negative_prompt=negative_prompt,
        num_inference_steps=50,
        guidance_scale=20,
        controlnet_conditioning_scale = 1.0,
        width=512,
        height=768,
        num_frames=16,
        conditioning_frames=cn2,
        generator = generator
    )
    

11. Export Frames to GIF:

    frames = output.frames[0]
    export_to_gif(frames, "xiangling_animation.gif")
    

12. Display the Result:

    from IPython import display
    display.Image("xiangling_animation.gif")
    

Target gif

Use Anime Upscale in https://github.com/svjack/APISR

Conclusion

This script demonstrates how to use the diffusers-sdxl-controlnet library to generate animated images with ControlNet and SDXL models. By following the steps outlined above, you can create and visualize your own animated sequences.