hakurei/waifu-diffusion · Error on macbook with pipe.to('mps')

import torch
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained('hakurei/waifu-diffusion', torch_dtype=torch.float32)
pipe = pipe.to('mps')

pipe.enable_attention_slicing()

prompt = "test"

_ = pipe(prompt, num_inference_steps=1)

image = pipe(prompt, guidance_scale=6).images[0]
    
image.save("test.png")

This throws the error

...
line 11, in <module>
    _ = pipe(prompt, num_inference_steps=1)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/waifu/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/waifu/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 362, in __call__
    timesteps_tensor = self.scheduler.timesteps.to(self.device)
TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.

It works if I use runwayml/stable-diffusion-v1-5 instead of waifu diffusion.
This issue might be related, https://github.com/pytorch/pytorch/issues/78168, but it has been fixed. This issue only happens when I use waifu-diffusion, not stable-diffusion