Unable to train Dreambooth on Mac M1

[muhammadsr]

I have probably been stuck at this for a week. Could not find a solution. I have installed Stable Diffusion webui just fine and it's generating images. However, I am unable to train Dreambooth.

I get this error:
Exception training model: 'Expected query, key, and value to have the same dtype, but got query.dtype: c10::Half key.dtype: float and value.dtype: float instead.'.

I am running this using:

arch -x86_64 ./webui-user.sh && ./webui.sh

my webui-user.sh:

#!/bin/bash
####################################################################
#                          macOS defaults                          #
# Please modify webui-user.sh to change these instead of this file #
####################################################################

if [[ -x "$(command -v python3.10)" ]]
then
    python_cmd="python3.10"
fi

export install_dir="$HOME"
export COMMANDLINE_ARGS="--skip-torch-cuda-test --upcast-sampling --no-half-vae --use-cpu interrogate"
export TORCH_COMMAND="pip install torch==2.0.1 torchvision==0.15.2"
export PYTORCH_ENABLE_MPS_FALLBACK=1

####################################################################

Dependencies:

Initializing Dreambooth
Dreambooth revision: e9f8e4ee3548d76dffef03447783e6f0f1ebd707
Checking xformers...
Checking Dreambooth requirements...
[Dreambooth] accelerate v0.21.0 is already installed.
[Dreambooth] bitsandbytes v0.41.2 is already installed.
[Dreambooth] dadaptation v3.1 is already installed.
[Dreambooth] diffusers v0.23.1 is already installed.
[Dreambooth] discord-webhook v1.3.0 is already installed.
[Dreambooth] fastapi vNone is already installed.
[Dreambooth] gitpython v3.1.40 is not installed.
Successfully installed gitpython-3.1.40
[Dreambooth] pytorch_optimizer v2.12.0 is already installed.
[Dreambooth] Pillow vNone is already installed.
[Dreambooth] tqdm vNone is already installed.
[Dreambooth] tomesd v0.1.2 is already installed.
[Dreambooth] transformers v4.30.2 is already installed.
[Dreambooth] tensorboard v2.13.0 is already installed.
[!] xformers NOT installed.
[+] torch version 2.0.1 installed.
[+] torchvision version 0.15.2 installed.
[+] accelerate version 0.21.0 installed.
[+] diffusers version 0.23.1 installed.
[+] transformers version 4.30.2 installed.
[+] bitsandbytes version 0.41.2.post2 installed.

Any idea how to fix this on Mac M1?. I have tried to train dreambooth on 5 different machines now including linux and docker and I always run into issues. Dreambooth is very hard to get working :(