Training scripts released in JoyVASA

[xuyangcao]

Hi, we released training scripts in JoyVASA, which is highly inspired by VASA-1, see here for reference: https://github.com/jdh-algo/JoyVASA

[johndpope]

hey @xuyangcao - thanks again for bringing this to my attention - i went out on a limb to wire up emo + noise + data loader etc - to no success.

i started pulling apart diffposetalk - and then i revisit this - do you think vasa perform this same transition training?

seems like it goes to a lot of trouble to put the model through - shortening the window - making it guess the subsequent frames - doesn't the previous motion context + audio assist the model to pick up from where it left off?
(that was my understanding) they also did the windowing with just 50 frames and then the 25 stride pattern making the model see more transitions

https://github.com/jdh-algo/JoyVASA/blob/main/src/dataset/talkinghead_dataset_hungry.py#L173

[johndpope]

@xuyangcao
i am getting closer - https://github.com/johndpope/VASA-1-hack

https://wandb.ai/snoozie/vasa-overfitting/runs/qqprlbs6?nw=nwusersnoozie

when the expressions match - then the frame generation is spot on -

i may need to rejig things still - my model is 46 million parameters - and vasa-1 is 29 million.
im training on a 5090 - need more compute.

JP

[johndpope]

i unplug theta projection to focus on lipsync -
i use these s1 training videos
https://zenodo.org/records/3625687

i swap in videotalk perceiver audio projection -
https://wandb.ai/snoozie/vasa-overfitting/runs/ybx6ohwo?nw=nwusersnoozie

losses are coming down.