[Feature request] Adaptation to the Whisper's JSON output

[sensboston]

Hello, is it possible to adapt your project to the Whisper's JSON output? I'm working on karaoke program for Windows, and need all words in the lyrics to be timestamped.
I'll be glad to issue a PR for this feature but unfortunately I'm not proficient in Python programming (mostly use a C# & C++).

[EtienneAb3d]

@sensboston

I will have a look at it as soon as possible.
To be sure of what you expect, can you provide me with an example (JSON+TXT)?

WhisperTimeSync is not written in Python but in Java.
;-)

[sensboston]

Here we go: samples.zip
There are two directories: English (Smokie, "Living Next Door to Alice" and Russian (Bit-quartet Secret, "Alice") with JSONs and original lyrics (my daughter's name is Alice 😉 ). Whisper's English output is kinda affordable but Russian is a complete mess.

[EtienneAb3d]

@sensboston
Hmmm... The problem I see with this JSON format is that each word has a mandatory description including its timestamp. It will be very hard to decide what to do with not-matching words.
🤔

[sensboston]

Yeah, it's an issue, agree. But I haven't looked to your (or Java code you've ported) implementation (yet). Theoretically it's possible, even without involving AI - for example by using "soundex" algorithm. I thought about this but initially wanna check if someone already done this.

[EtienneAb3d]

@sensboston
The problem is not to match word by word, this is what WhisperTimeSync is doing.
The problem is to know what to do with unmatching words in this specific JSON description.
I may adapt an algo I already have for similar cases, but this is quite a work.
Do you have a budget for this?

[sensboston]

No budget at all, I do development just for fun, will publish open source here when it done.

P.S. If you want, I'll add you to this private repo (but you need a Windows PC to test at least).

[sensboston]

Any progress? Or you have no idea how to implement this? Please le me know - I don't wanna to waste a time.

[EtienneAb3d]

@sensboston
I understood you were working on the subject on your side.
On my side, without a budget, I have to find/allocate on my free time.