Performance & Slowness

[thewh1teagle]

I tried to run whisper-rs-cli on 16khz mono wav file with length of 30s (default cli params)
It took 1m and 9.408s.

Spec: amd ryzen 5 4500u, windows 11

1. It looks pretty slow to me, is there a way to improve it?
2. What's the best way to stream data into the input instead of holding everything in ram at a time?
   I tried to split the samples and call state.full_n_segments multiple times and it was, slower, took 1m and 47.975s

[tazz4843]

Best question is are you compiling in release mode?

Failing that, it'd be nice to see your code.

There's no way to stream audio due to fundamental limitations in the whisper model itself, which feeds audio in 30-second chunks, so you must load the entire file into memory and process that.

[thewh1teagle]

Very good question, I didn't tried yet in release mode, missed it
https://github.com/thewh1teagle/ruscribe

if whisper feeds audio in 30 second chunks,
we can stream 30 seconds chunk each time? no? (at least that instead of load the entire file)

[tazz4843]

For feeding in 30 second chunks, I found the following issues on upstream's repo:

ggml-org/whisper.cpp#137
ggml-org/whisper.cpp#426
ggml-org/whisper.cpp#1428

This would need upstream support unfortunately before we have any way to implement it. Right now we expose the entirety of whisper.cpp's API.

[thewh1teagle]

I'll see them.
Wondering if it will slow down the process.
I think it's not usable to use it when you load everything to memory first. I have to load it in chunks of 10-30 seconds from fs,
But I hope that it won't affect performance as everytime I fill the model with new sample it will do some heavy stuff,
any toughts?

[thewh1teagle]

Checked in release mode
The wav file with length of 30 seconds took
0m and 42.074s, better!

[tazz4843]

It's still rather interesting that it takes so long. Perhaps you could use something like cargo flamegraph to see where the issue lies.

https://github.com/flamegraph-rs/flamegraph

[thewh1teagle]

1. Can we hide somehow the output of whisper.cpp completly? (it always show loading model .... )?
2. I tried to use progress callback for using with indicatif
   When I set it inside the closure of on_progress with move || {} I get access violation error (core dump)

[tazz4843]

See #112 for some logging utilities that will help avoid those output lines

[thewh1teagle]

@tazz4843

[thewh1teagle]

Also with tracing feature

[tazz4843]

Should be fixed, give it another shot.

[thewh1teagle]

Going to try.
Do you have idea about this? #114

[thewh1teagle]

I made this open source project vibe
I'm planning to show the text transcribed in realtime and progress bar.

vibe.mp4