[Tracking] Asynchronous API extensions for VAD and KWS

[reuben]

For features based on Voice Activity Detection and Keyword Spotting, we should ideally implement a callback based API. These are latency sensitive features where applications have no way of knowing what's a good time or interval to query the status of VAD/KWS without guessing. A callback based API lets us notify the application instead.

One simple example is automatic end of stream detection. We can implement this with Voice Activity Detection quite easily by looking at the probability of blank labels over time in the decoder loop (see e.g. arxiv:1611.09405). Exposing this functionality in our current API, on the other hand, is not so simple. Our API is entirely synchronous and information flows are always started by the application, not the library, so there's no simple API extension that lets us propagate the end of stream information to applications.

Implementing an asynchronous callback based API is non-trivial. We need to be extra careful around reentrancy, for example. Maybe manage a worker thread (or threads) ourselves to make it less error prone. (This would also simplify problems of stalling the main thread and dropping recorded samples in naive integrations of DeepSpeech as single-threaded systems, at the cost of removing low level control from applications.)

Binding an asynchronous API to higher level languages is even more complicated, as each language has its own preferred syntax/ergonomics/event loop details these days, and SWIG will not do much to avoid having to write code by hand for each language binding.

The good news is that these changes should all be doable as extensions of our current API, without breaking backwards compatibility. This means we can ship these mid 1.x cycle without too much hassle. This is a tracking issue for the overall work of figuring out how to add asynchronous APIs.

[StuartIanNaylor]

@reuben prob a very dumb coment but can you not expect dynamic Cue-Points Chunk that VAD & KWS should create?
You can to a file but don't know if you can dynamically to a open stream.

[reuben]

@StuartIanNaylor I don't understand. What even is a chunk in this context?

[StuartIanNaylor]

All the audio files we commonly work with have a header that I think Riff is a common container as compression is just on top of that.
I was just thinking like some proprietory camera/av stuff they use a cue naming scheme and the cue chunk of the header.
That there is no connection or requirement to VAD or KWS just that VAD KWS leaves a cue of activity.

As said when it comes to PCM streams it sort of doesn't work as there isn't a chunk or header as there is no file.

I dunno to be honest as trying to image how it will work purely from reading but for many instances Deepspeech is single thread because of the GPU or accelerator they use.

So you have a multithread KWS VAD queue feeding a single thread instance and what you wrote sounded like it could be very complex.

Linux is just files so to keep it simple have a cue file for each input as often these are going to be queued?
I was just thinking VAD and KWS prob will be working with realtime streams but deepspeech will be a naive single thread due to the hardware used with it?

I have no idea and bow to you wonderful work and knowledge but just thought an accompanying cue file might be needed as KWS/VAD are likely not to be in sync with what is being processed?
My complete lack of knowledge and fear of a asynchronous callback based API's ducked under the cover of a simple file base.

[reuben]

I think you're misunderstanding how the KWS/VAD functionality would work here. There is no multi-threading. The threading implications are the same as with current DeepSpeech: users must make sure the inference calls don't block any other critical threads like a UI thread or the microphone recording thread.