what about the training results

[taichuai]

awesome work, can you show some cases about your inference result

[johndpope]

thanks for noticing.

this model should achieve on par results to the emo volumetric avatar model for inference. (itʼs using it as backbone)
https://github.com/neeek2303/EMOPortraits
this codebase (mit license) includes a fork / nemo (apache) -
i added a pipeline.py class that can do driving video.

https://drive.google.com/file/d/1PYrUrowldG4roVaJKJb3eEzoLC6IFr8R/view?usp=sharing

when i run this - even on my 5090 - i was only scratching 20 frames / second.
so to match vasa -i either need to do some batch processing / cuda torch script - or rip out some computations in that code to get higher throughput.

the vasa_model is still cooking - i just replaced the audio to align with joyvasa a few hours ago - and retraining on small set.
i will run it through (updated) inference in the morning. vi.py - if that fails - i will need to pull apart attention mechanism - (like the expression here) https://wandb.ai/snoozie/vasa-overfitting/runs/c8bc4sj4?nw=nwusersnoozie

Thereʼs been so many critical errors come out in the wash over last week but thanks to Grok (adding padding to channel ). this caused me so much grief - without the padding - all the training seemed to be working - but when i was running inference - the peoples heads were popping crazy - it sent me down a rabbit hole.
https://drive.google.com/file/d/1A726Yq3V9dTPqrYyRUtl6hhD-PcZSZWq/view?usp=sharing
(this was from January)

from emo

[johndpope]

i threw out most of the vasa_model / trainer code today - and replaced it with some vanilla implementation from grok. seems ok. i did a quick training just then for about 5 mins
https://github.com/user-attachments/assets/492e733b-a41d-4a77-b7d8-8d71f14c8b7b
i will train all night and post results in morning.

UPDATE - i haven't pushed code yet - but im close to making this all work.
i overlooked the warping.

UPDATE - i pushed code to train on the warping -
https://wandb.ai/snoozie/vasa-overfitting/runs/khyzmhm0?nw=nwusersnoozie

i add this - https://github.com/johndpope/nemo/blob/main/pipeline_face_attr.py
to basically create video from warps - this was succesful today and is now what vasa is aiming to achieve in transformer

for context - i was (incorrectly) just predicting the (rigid warping) theta / translation / rotation - but after days of training - the head movements looked still / flat - i thought the expression embedding was taking care of this -

vasa-output-overfit_config-7.mp4

After re-reading the paper - we need rigid and non rigid warps.

i add some h5 caching to get all the warps / expressions / emotions / eye gaze angles to speed up training.
still a work in progress...

[johndpope]

getting closer -
this is critical clean warp extraction from nemo -
4fb7904

ive isolated the warps successfully - with caches to h5 file - so given a video + image - can reapply the warps - this will help in fps in vasa - i just gotta wire up the dataset to extract the warps from video -

TODO - go through the training videos - > extract the warps - store them - cache h5 file
wire up the diffusion motion transformer to predict warps -
connect the decoder.

[johndpope]

https://wandb.ai/snoozie/vasa-overfitting/runs/9d0di1su

[johndpope]

https://wandb.ai/snoozie/vasa-overfitting/runs/bsl6966l?nw=nwusersnoozie
I throw in a cross attention into decoder layers for audio -
finally results are coming good.

the overfitting is target 2 layer depth - so it trains quickly.
i turn off a bunch of losses - just to isolate things.
i am overfitting to one video - will expand tomorrow.

some progress - had to rejig model.
https://wandb.ai/snoozie/vasa-overfitting/runs/qqprlbs6/panel/b0gp2dkme?nw=nwusersnoozie

[johndpope]

after a day of training on 5090 - it was crap - i added a cheat sheet db for expressions -

#61
it solves the expression floating in space - like this

step 0 !!!!

wont need 4x a6000 training if i can sample a huge dataset of expressions.

vasa-output-overfit_config-videovideoeI2V8Bd5X9s-scene6_scene1.mp4

[johndpope]

https://drive.google.com/drive/u/0/folders/1kmERS5AGQbGPbvb2t6OIIH3h7PMtidbC

https://drive.google.com/file/d/1DQ0CcH93m5rrcC2VCMXs9_PyhPzrZBUR/view?usp=drive_link
it's a little bit better - but now looking at lip sync losses - any ideas - im all ears. i guess the synchformer may help....

im overfitting to a training video - but it's not aligning. ...

UPDATE -
my latest commit implements the syncformer loss correctly -
the gradients blew up mid training - but i just need a good video training set.