lllyasviel

/

ControlNet-v1-1

like 3.61k

Hi,i'm trying to finetune a multiControlNet using inpaint and canny. i find that, when the images' type is PIL.Image, the model could generate successfully, however , when i try to transform the image to torch.Tensor(the shape is B C H W), the output would be wrong.

Here is my code

controlnet1 = ControlNetModel.from_pretrained("inpaint") ##inpaint model
controlnet2 = ControlNetModel.from_pretrained("canny") ##canny model 
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "botp/stable-diffusion-v1-5",
    controlnet=[controlnet1,controlnet2],
    local_files_only=True,safety_checker=None
)

def trans(img):
    return torch.tensor(img)


outputs = pipe(prompt=prompt,
               image=[trans(rec1),trans(canny)],
               control_image=trans(control_img),
               controlnet_conditioning_scale=[1.0, 0.8],
               mask_image=trans(mask_img),
               eta=1,
               num_inference_step=20,generator=generator,negative_prompt=negative_prompt).images

the generated image is

i dont know the reason. Does anyone else have the same situation?