Transcribe Task Translates Without Language Code

[huggingfacename]

There is unexpected behavior when attempting to transcribe non-english text.

The following code:

import whisper_s2t

model = whisper_s2t.load_model(model_identifier="large-v3", backend='TensorRT-LLM')
files = ['nonenglish-testfile.wav']
lang_codes = None
tasks = ['transcribe']

out = model.transcribe_with_vad(files,
                                lang_codes=lang_codes,
                                tasks=tasks,
                                batch_size=24)

Will return English text even if the spoken language is not English. If I pass in a language code, then the model will return that language, but this is manual and will also fail for speech that has multiple languages.

Openai and huggingface implementations of whisper have built in detection and do not have this problem.

Was something lost during the model conversion to tensorRT?

[PedroMPagani]

Hey, did you figure this out?

[ustclan]

Because it has a default language, if lang_codes=None, it will default to using 'en' for ASR. Of course, this is not ideal. In fact, an automatic language detection feature could be added, which is quite simple to implement. Refer to the implementation of WhisperX: sample a segment of the audio, input the prompt <|startoftranscript|>, and let the model output the first token, which is the language code. Here, the softmax output can also be used as a confidence score. If the confidence is too low, default to using 'en' for ASR.