Clarification on Teacher Checkpoint and Dataset Details for Student Model Training

[dargma]

Hi,

I'm trying to reproduce the Sana-Sprint experiment and have two questions regarding the configuration and dataset used for training the student model:

1. Teacher Checkpoint for Student Initialization:
   As I understand from the Sana-Sprint paper, the student model should be initialized using a fine-tuned teacher model. However, in the configuration file (SanaSprint_1600M_1024px_allqknorm_bf16_scm_ladd.yaml), the parameter is set as follows:

   model.load_from: hf://Efficient-Large-Model/Sana_Sprint_1.6B_1024px/checkpoints/Sana_Sprint_1.6B_1024px.pth

   Should this checkpoint actually be the teacher checkpoint (for example, something like Sana_Sprint_1.6B_1024px_teacher.pth), which is then used to initialize the student model? Please confirm if the current setting is correct.

2. Dataset Details for Student Training:
   There isn’t enough information available about the dataset used for training the student model. Could you provide details on:

   • The number of image-caption pairs used.
   • The sources of the images (e.g., LAION, SA-1B, internal databases, etc.).
   • How the captions were generated (e.g., using models like GPT-4 or other methods).

Any clarification or additional documentation on these points would be greatly appreciated.

Thank you!

[lawrence-cj]

We load the teacher model here:

Sana/configs/sana_sprint_config/1024ms/SanaSprint_1600M_1024px_allqknorm_bf16_scm_ladd.yaml

Line 22 in daa661f

teacher_model: hf://Efficient-Large-Model/Sana_Sprint_1.6B_1024px_teacher/checkpoints/Sana_Sprint_1.6B_1024px_teacher.pth

I assume 1M dataset would be enough, we have our internal dataset for the fine-tuning and you can get the caption through GPT-4 or other methods, it's ok with the distillation training.

[dargma]

Thank you for your response.
To clarify, I have an additional question.

You mentioned "We load the teacher model here: ..." in your answer. Does that mean that teacher_model: ../Sana_Sprint_1.6B_1024px_teacher.pth is loaded not only for the teacher model but also for initializing the student model?

[lawrence-cj]

If you want to train from a teacher model. You should load both teacher and student model with the provided teacher model. If you want to fine-tune, you can load teacher model with the teacher checkpoint and load the student model with sana-sprint checkpoint which is the distilled one.