Use system voice input

[devycarol]

This branch has no code yet, but I'm opening this PR to synthesize all the duplicate voice input issues and create an outline for features to implement. I plan to at least start to work on the code for this in this branch.

Closes #764, closes #988, closes #2728 by implementing seamless voice recognition into the HeliBoard keyboard via the system speech recognition service. The voice key will use the system speech recognizer service to transcribe speech without closing the board. No more janky, laggy IME switching unless the user insists. This will almost certainly require the RECORD_AUDIO permission.

Closes #192, closes #1234, closes #2699 with the rationale that we make it a non-goal for HeliBoard to have its own self-specific voice recognition code. No accommodations need to be made for any specific recognizer app, the app just needs to implement the speech recognition service. Someone could even make a "voice recognizer service for HeliBoard" in that mold which would also be compatible with other apps.

Behaviors

Add a settings page for voice input. In it add an option for voice key behavior. The default option will be in-board voice recognition with the system speech service. You can toggle to use a separate voice IME instead. In either case, you will be offered to choose a specific voice service or stick with the system default. Someone suggested also having an "always ask" option, I think a better alternative is to have the long press of the voice key be a selection dialog for voice recognizer/keyboard similar to the space long press. The preference may need to be ignored if there is a recognizer service but no voice IME or vice versa.

Voice recognition will need three states: off, listening, and waiting. The third is needed because it takes some time to load the recognizer, particularly when initializing it. Another voice typing option could be to always initialize the recognizer on keyboard load. Could cause unwanted resource usage but would alleviate startup delay when pressing the voice key.

You can apparently use intent parameters and such to request that the recognizer "prefer on-device recognition". This could be another default-on voice typing option.

We must not EVER, ever, ever, ever, ever stop listening for voice input just because the user hasn't spoken in the last 0.021 seconds. Every single damned time the Google voice IME stops listening to me right as I start to speak again because "wow!! hands free!!1!" makes me want to scream. We must only stop listening when the user tells us to.

Possible future work

The VTT output should honor shift state. To do this, we would reject stupid capitalization coming from the Google or any other voice input service—don't accept capitalized words from the speech recognizer unless it is a proper noun. Seems simple enough, if (word is title case) and (word.toLowerCase(locale) is in dictionaryOf(locale)), force the lowercase word. You should be able to change the shift state between lowercase, shifted, and all caps while you are talking. This raises the question of where and how to start changing the words from the recognizer—shift the next word said by the user somehow using timestamps, or shift the next word to be output by the recognizer which may lag behind?

In general, you should be able to interleave voice input with tap/gesture typing. This presents the same challenges as handling shift state.

There should be a 'spelling mode' for when the user knows something they want to type isn't in the voice dictionary. So the dictionary would be clamped to just letter and special character names. A, Q, 9, #, (, space, etc. This would probably need a distinct recognizer with a more limited and precise dictionary, though you could have a hacky implementation of this using "rewrite rules". Bee -> b, be -> b, pound -> #, hash -> #, etc.

[devycarol]

I haven't finished my prototype for this yet but I'm pretty sure "alternate interpretations" are done for entire phrases? If true, that may present some challenges for providing suggestions. Could offer suggestions just based on similarly spelled words, or offer multi-word suggestions. Idk I'll keep working on the prototype.

[devycarol]

I now have a working prototype for this. The RECORD_AUDIO permission has been added, there's no way around this. I haven't implemented the actual request flow yet, so you need to grant the permission manually in the app info page to test the prototype. I've tested with the "Speech Recognition and Synthesis from Google" app installed on my phone. Note that if "Android System Intelligence" is installed on your phone, you may have to switch the "Speech recognition" app in the system settings to another—it's a recognition service that's completely unimplemented. ???? maybe that's just the case for my LineageOS ROM?

Right now the parsed text is output to Log.d. It's probably trivial to move it to actual keyboard output, I just know that something explodes any time I touch the InputConnection :P.

Necessary additional work is documented in the TODO LOLs. I want to avoid swearing on the internet, so I instead invite you to skim through the comment explaining "EXTRA_SPEECH_INPUT_COMPLETE_SILENCE_LENGTH_MILLIS" and join me in silently seething over Google's horrible design choices :).

It looks like the Google speech recognition service offers no facility for correction suggestion ideas in the results bundle despite the interface having lots of optional results to that end, so to have specialized correction suggestions I'd need to work with a service that implements the full suite. It's also possible that there's some intent parameter I just haven't found that's necessary to get more than one hypothesis. Either way, I'd say it's a future problem. In the meantime, we have the generic correction system.

Currently there's no visual indication that voice recognition is in progress. This should be added, and there are a lot of fun options for this. We can even use the onRmsChanged() callback to have some sort of volume wobbler graphic.

Lastly: I have very little experience with Jetpack Compose, so I don't know how to make a new settings page without breaking things. Could you create a blank template for a "Voice Typing" screen? My idea is to put it between text correction and secondary layouts.

[devycarol]

hehehehe REALLY excited about the synergy between this and the D-pad layout :)))

[devycarol]

Just tested the Sayboard service. Frustratingly, its "partial results" are spammy—intermittently adding to the utterance while still reporting the entire thing each time. Its "full results" are in the onResults() bundle. It also does not honor the request to not stop listening 0.021s after the user stops speaking.

Google on the other hand includes nothing whatsoever in the onResults() and the partial results are where all the text comes from, piece by piece. It's also more sluggish than Sayboard which is unfortunate.

So we need to write app-specific code. Yaey.

[woj-tek]

Isn't this somewhat related to #1547 and overlaping with main...notune:HeliBoard:voice-input-recognizer-app ?

[devycarol]

That branch didn't appear to overlap with what I'm doing here unless I missed something. This branch uses the system service to recognize speech with no pop-up windows or separate input method editors.