WhisperX alignment misaligns lyrics

[rzru]

That's a known issue I'm still working to understand.

Sometimes, especially when the vocals stem contains back-vocals and/or chorus, the alignment slides off the rails, sometimes it thinks the word is longer than it actually is, and in general, it is pretty non-deterministic.

I would appreciate any support from the experienced with the alignment models devs to figure out this issue

[judilsteve]

I'm no ML engineer, but it seems that "crisper whisper" is at-or-near SOTA for word-level timestamps:

https://github.com/nyrahealth/CrisperWhisper/blob/main/transcribe.py

I think youtube has started using it recently because I've noticed the subtitle timing has markedly improved in quality.

The CrisperWhisper repo doesn't mention forced alignment (to use with pre-downloaded "canonical" lyrics) but I wonder if it could be as simple as plugging the nyrahealth/CrisperWhisper model name into this:

https://huggingface.co/spaces/Anioji/WhisperX/blob/9c2d684d1a778e559b6b13bb90112712dbc20568/whisperx/alignment.py

[rzru]

@judilsteve amazing stuff, I will play around some time later!

[0xC0ncord]

I noticed this with one song I tried importing as well, "Heaven or Las Vegas" by Cocteau Twins. The model seems to do a pretty good job in some parts, but in most other areas it is way off, probably because the vocals are sung with a very heavy accent that not even most English speakers can understand.

[rzru]

@0xC0ncord yes mate, that's indeed a big problem. I would try the thing @judilsteve have proposed and check if it does a better job

[judilsteve]

The CrisperWhisper repo doesn't mention forced alignment (to use with pre-downloaded "canonical" lyrics) but I wonder if it could be as simple as plugging the nyrahealth/CrisperWhisper model name into this:
https://huggingface.co/spaces/Anioji/WhisperX/blob/9c2d684d1a778e559b6b13bb90112712dbc20568/whisperx/alignment.py

Unfortunately not so. Further inspection shows WhisperX doesn't actually use whisper for forced alignment at all, the whisper model family simply doesn't support it. The code in alignment.py uses Wav2Vec2 models for forced alignment.