Evaluate Transcription Models

[jspada200]

Vexa has been our primary driver for transcription. Under the hood it is using whisper live with the tiny or medium model. We need to evaluate different transcription models for accuracy and speed. There is some test data being generated in another ticket however as long as the test audio clip is consistent, it should be ok for this evaluation.

Acceptance Criteria:

• Reasearch different models for accuracy and speed
• A spreadsheet with the model, the size, some metric for accuracy and speed.

[camerontarget14]

@jspada200 going to take a pass at this one too! Thinking a big ol' afternoon of benchmarking. Can I be assigned here? (:

[camerontarget14]

Does it make sense to fork: https://github.com/huggingface/open_asr_leaderboard and run tests there? I was thinking just the ones from the "long form" tab on this list: https://huggingface.co/spaces/hf-audio/open_asr_leaderboard

[loorthu]

I captured the transcripts using whisper medium model (running on nvidia RTX A6000 card) and seeing many transcription errors. I used the shared review sample audio by @camerontarget14 for this. I asked Gemini to evaluate the results and attaching it too.

gemini_evaluation.md
vexa_transcript.txt
original_transcript.txt

[jspada200]

Thanks for taking a look guys! I wonder, have we tried dumping the audio to a file from Vexa? The bot captures chunks of audio then sends to whisper. I wonder what the audio quaility is like on those chunks.... Just a thought.