await in model.transcribe

[JohnConnor123]

Can I use await model.transcribe(...)? I want to send hundreds of messages without kafka queue, and current gpu load at 1 request is low and as far as I understood, the time to wait for request before sending to model.transcribe() grows with number of requests.

[MahmoudAshraf97]

transcribe function is synchronous so you cannot await it as is, fortunately there are workarounds, the easiest of them is to use asyncio.to_thread but it's not the most effecient, for best effeciency, you will need to rewrite transcribe function, especially the generate_with_fallback part to make it asyncrounus, CT2 supports async inference
So you'll modify this:

faster-whisper/faster_whisper/transcribe.py

Lines 1404 to 1417 in 1383fd4

	result = self.model.generate(
	encoder_output,
	[prompt],
	length_penalty=options.length_penalty,
	repetition_penalty=options.repetition_penalty,
	no_repeat_ngram_size=options.no_repeat_ngram_size,
	max_length=max_length,
	return_scores=True,
	return_no_speech_prob=True,
	suppress_blank=options.suppress_blank,
	suppress_tokens=options.suppress_tokens,
	max_initial_timestamp_index=max_initial_timestamp_index,
	**kwargs,
	)[0]

to follow this pattern (This is psuedocode but I implemented it so it's guaranteed to work)

future = self.model.generate(**kwargs, async=True)
while not future.done():
    await asyncio.sleep(0.001)
result = future.result()

this way you will have truly async inference and high gpu utilization
https://opennmt.net/CTranslate2/python/ctranslate2.models.Whisper.html#ctranslate2.models.Whisper.generate

or you can use a framework that is designed for production use cases such as TRT-LLM

[sh1man999]

@MahmoudAshraf97 Why not add this to the current package ?

[suyip01]

@MahmoudAshraf97 Can you integrate this into the code? I'm not very familiar with the development of functional modules.

[MahmoudAshraf97]

I don't have the capacity to work on that ATM, but I'm willing to accept user contributions

[sh1man999]

I don't have the capacity to work on that ATM, but I'm willing to accept user contributions

can you tell me how to make asynchronous BatchedInferencePipeline ? where to start

[sh1man999]

Right?

futures: List[WhisperGenerationResultAsync] = self.model.model.generate(
            encoder_output,
            prompts,
            beam_size=options.beam_size,
            patience=options.patience,
            length_penalty=options.length_penalty,
            max_length=max_length,
            suppress_blank=options.suppress_blank,
            suppress_tokens=options.suppress_tokens,
            return_scores=True,
            return_no_speech_prob=True,
            sampling_temperature=options.temperatures[0],
            repetition_penalty=options.repetition_penalty,
            no_repeat_ngram_size=options.no_repeat_ngram_size,
            asynchronous=True
        )
        async def await_result(future):
            while not future.done():
                await asyncio.sleep(0.001)
            return future.result()

        results = await asyncio.gather(*[await_result(future) for future in futures])

[MahmoudAshraf97]

Correct

[sh1man999]

Correct

I'm willing to create a pull request
https://github.com/sh1man999/faster-whisper-async

[sh1man999]

approv pls
#1291

[sh1man999]

transcribe function is synchronous so you cannot await it as is, fortunately there are workarounds, the easiest of them is to use asyncio.to_thread but it's not the most effecient, for best effeciency, you will need to rewrite transcribe function, especially the generate_with_fallback part to make it asyncrounus, CT2 supports async inference So you'll modify this:

faster-whisper/faster_whisper/transcribe.py

Lines 1404 to 1417 in 1383fd4

result = self.model.generate(
encoder_output,
[prompt],
length_penalty=options.length_penalty,
repetition_penalty=options.repetition_penalty,
no_repeat_ngram_size=options.no_repeat_ngram_size,
max_length=max_length,
return_scores=True,
return_no_speech_prob=True,
suppress_blank=options.suppress_blank,
suppress_tokens=options.suppress_tokens,
max_initial_timestamp_index=max_initial_timestamp_index,
**kwargs,
)[0]

to follow this pattern (This is psuedocode but I implemented it so it's guaranteed to work)
future = self.model.generate(**kwargs, async=True)
while not future.done():
await asyncio.sleep(0.001)
result = future.result()
this way you will have truly async inference and high gpu utilization https://opennmt.net/CTranslate2/python/ctranslate2.models.Whisper.html#ctranslate2.models.Whisper.generate

or you can use a framework that is designed for production use cases such as TRT-LLM

can you tell me how to make vad asynchronous ?

[lukaLLM]

Hey I am not sure if this got finished or not. I am working with faster whisper a bit and could try to make it async as I need it for project anyway. Is it still required, would be nice to finally contribute :) ?

[sh1man999]

Hey I am not sure if this got finished or not. I am working with faster whisper a bit and could try to make it async as I need it for project anyway. Is it still required, would be nice to finally contribute :) ?

Yes, he is needed