How do you run Validation Tests with Musubi-Tuner?

[niceguy4]

Reference: https://github.com/spacepxl/demystifying-sd-finetuning

It is my understanding that its important to run validation testing against a separate dataset that is not included in the original training dataset to see how the model is progressing towards generalizing. I believe this helps to identify when the model starts to overfit.

How are ya'll running validation tests with Musubi-Tuner? I understand that I can just generate videos/images for each epoch but is there a more scientific approach that can be automated that follows standard validation testing methods?

[Oruli]

I also came looking for an answer to this, seems Musubi doesn't support eval sets

[niceguy4]

Well, I'm a shit programmer so forewarned but I used Claude to write a validation test for Wan2.2. I've forked this project because I have no idea how to submit a git commit to a project that is not mine or suggest changes to the main project. Maybe someone smarter then me can port the code over and submit a update request for the validation testing. I'm sure there are issues with the code but here it is if you are interested: https://github.com/niceguy4/musubi-tuner-wan-validation/