Question to review my understanding of the operation process for RealTimeSTT text function

[sangheonEN]

The code for [Text Function] below refers to the previous code audio_recorder.py and creates a callback function, creates a thread, and applies in/out parameters. Please refer to the attached audio_recorder.py for my code. (It may be a little different from the latest version because I referred to your old version, but the voice streaming and stt transcription operation codes will probably be similar.) The text function is called in the main code of Thomas_audio_control_src.py.
realtimestt.zip

text, thomas_event_state = recorder.text(utils.main_process, start_time, communicator, similarity_cal, params.similarity_config)

In other words, I implemented a structure that uses the text function to perform voice streaming and STT transcription.

However, since I do not have professional knowledge about voice streaming, it is too difficult to customize the process. So I'm writing this because I want to review if I understand the process of voice streaming and STT transcription correctly. Please read the article and if there is any part where the analysis is wrong or you can advise me to understand it better, I would appreciate it. I want to gain more professional knowledge about voice streaming.

I think it would be appreciated if you could tell me the sequence. How does audio come in through the microphone, and how does data processing occur in which thread, and how does voice stream and STT transcription occur?

• Running environment: WINDOW 11, python 3.11.5 (venv), nvidia gpu 4060 labtop -

[Analyzed content]

1. Multi Processing: Process and thread creation when creating AudioToTextRecorder object

• sub_processor: _audio_data_worker
• sub_processor : _transcription_worker
• main processor: recording_thread(_recording_worker), VAD(silero_speech)_thread, multi process parameters (self.audio_queue, self.start_recording_event, self.stop_recording_event, self.shutdown_event, self.interrupt_stop_event, self.was_interrupted, self.main_transcription_ready_event, self.parent_transcription_pipe, child_transcription_pipe)

2. audio_data_worker: Audio data stream open and
   -Open pyaudio’s interface stream

• audio_queue.put(audio_data)
• Mic Reconnection → Exception handling

3. _recording_worker

1. data=audio_queue.get()
2. handle_buffer_overflow
3. use_wake_words trigger
4. start_recording_on_voice_activity → Call start() function when True
5. Check if VAD ACTIVATION has been performed.
6. self.frames.extend(list(self.audio_buffer))

• The voice data stored in self_audio_buffer is the audio data that exists just before/after recording. It feels like giving a margin for flexible voice data processing.

7. Save all voice data. self.frames.append(data)

4. _transcription_worker

1. model = faster_whisper.WhisperModel
2. main_transcription_ready_event.set()
3. audio, language = child_transcription_pipe.recv()
4. segments = model.transcribe
5. transcription = " ".join(seg.text for seg in segments)
6. child_transcription_pipe.send(('success', transcription))

5. multi process parameters

• start_recording_event: When set in the start() function, it breaks from the wait_audio() function and moves to the next line of code.
• stop_recording_event: When set in stop(), shutdown(), it breaks from the wait_audio() function and moves to the next line of code. - use_microphone : flag to use local mic
• audio_queue :
• Save audio data : data = stream.read(buffer_size) is put in _audio_data_worker() and stored.
• Transmit audio data : get from recording_worker and transmit.
• shutdown_event : If set in the shutdown() function, all processes are terminated
• interrupt_stop_event : If KeyboardInterrupt occurs, both audio_data_worker and transcription_worker interrupt_stop_event.set() to stop recording.
• was_interrupted : set() when transcription ends and output an empty string
• main_transcription_ready_event : Go into _transcription_worker, load the faster-whisper model, and set() immediately
• parent_transcription_pipe : Send the recorded audio data and language information from transcribe() to child_transcription_pipe. And receive the text inference result from child_transcription_pipe. - child_transcription_pipe : _transcription_worker() recv audio data and language parameters from parent_transcription_pipe. Send text result inferred by stt model to parent_transcription_pipe.

6. text function

1. wait_audio

• Check if recording is in progress
• Save final audio data.
• self.audio = audio_array.astype(np.float32) / INT16_MAX_ABS_VALUE
• reduce noise
• reduce db

2. transcribe

• self.parent_transcription_pipe.send((self.audio, self.language)) → Send() final audio data (self.audio) to transcription_worker using parent_transcription_pipe.
• status, result = self.parent_transcription_pipe.recv() → Recv() text inferred by transcription_worker. - Perform _preprocess_output
• return self._preprocess_output(result)

[KoljaB]

Audio enters in two ways:

1. From the microphone: The _audio_data_worker process reads data from your mic and writes audio chunks into the audio_queue.
2. From external sources: The feed_audio method lets you add audio from another source and also places chunks into the audio_queue.

Once in the queue, the _recording_worker thread processes these chunks. It handles wake word detection, VAD analysis, and more. This thread is the core of RealtimeSTT. The flow is:
_audio_data_worker or feed_audio → audio_queue → _recording_worker.

Then, the _transcription_worker process handles the final transcription while the _realtime_worker thread manages everything related to live transcription.

self.audio_buffer compensates lags in wake word detection. If you say "Jarvis turn the lights on" and the system detects "Jarvis" too late, parts of the command might be lost. The audio buffer (together with pre_recording_buffer_duration parameter) make sure no words get missed in the transcription.

[sangheonEN]

Wow, that's really cool. Is chaining up webrtcvad and silerovad a great technique for speech recognition?

[KoljaB]

It's not absolutely crucial for the library but makes VAD fast and reliable. Webrtcvad is lightweight and very sensitive, it often triggers false positives. Silero is more resource-intensive. With chaining them you use webrtcvad as a prefilter and only activate Silero when voice activity is already detected. So Silero provides a second layer of validation. If you do Silero only you put extra load on your GPU, and if you do Webrtcvad only you deal with more false alarms. Combining both gives a solid balance.

[sangheonEN]

Oh right! And isn't the code for loading wav files and getting the transcription inference text results included separately in that library?

[KoljaB]

Unsure if I got that question right. The faster_whisper library includes the code to load wav files and output transcription text results, is that what you were asking?

[sangheonEN]

Yes, that's right. Thank you, I find the src at (https://github.com/SYSTRAN/faster-whisper) from usage