Evaluation prompt

[hellos0032]

Thank you very much for your great work!

As I understand, in a typical personalization task, the model is fine-tuned first, and then images are generated using a test prompt to compute evaluation scores.
Could you please share which test prompts (evaluation prompts) were used for fusion in K-LoRA, and also explain in detail how the evaluation scores were computed?
(I am currently using Stable Diffusion.)

Thank you!

[ouyangziheng]

Thank you for your interest. You may refer to the following issues:

#3

#5

[hellos0032]

Thank you for your reply !

I’d like to ask about the process of training the style LoRA.

• Some styles have multiple reference images, while others only have a single image. How did you handle this difference during training?
• The provided prompt seems to be “a cat in szn style.” Shouldn’t the prompt vary depending on each style image?
• More generally, what kind of prompts should be used in training style LoRA process?

Thanks in advance for your clarification!

export MODEL_NAME="stabilityai/stable-diffusion-xl-base-1.0"

# for style
export OUTPUT_DIR="lora-sdxl-waterpainting"
export INSTANCE_DIR="waterpainting"
export PROMPT="a cat of in szn style"
export VALID_PROMPT="a man in szn style"

accelerate launch train_dreambooth_lora_sdxl.py \
  --pretrained_model_name_or_path=$MODEL_NAME  \
  --instance_data_dir=$INSTANCE_DIR \
  --output_dir=$OUTPUT_DIR \
  --instance_prompt="${PROMPT}" \
  --rank=8 \
  --resolution=1024 \
  --train_batch_size=1 \
  --learning_rate=5e-5 \
  --report_to="wandb" \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --max_train_steps=1000 \
  --validation_prompt="${VALID_PROMPT}" \
  --validation_epochs=50 \
  --seed="0" \
  --mixed_precision="no" \
  --enable_xformers_memory_efficient_attention \
  --gradient_checkpointing \
  --use_8bit_adam \```

[ouyangziheng]

Sorry for the late reply. For style training, I recommend using prompts like ‘a product (building)(man)’ to weaken the original objects in the style. This helps the model focus more on learning the style concept itself.