Voice interaction

[plehegar]

What should we do with voice?

Relevant CG(s):

• Speech API
• Voice Interaction CG

MS input (November 2016)

[iherman]

I am really not expert on this, but speech synthesis in general is a major topic for publishers and, potentially, ebook readers. Audio books is a major big market; in some areas it evolves faster than any other format of ebooks. Obviously, having only one 'source' for a publication that can be used as an audio as well as a 'traditional' book is a major plus.

I do not know how the various reader implement this and whether a standard speech API is of interest. Worth asking...

[dontcallmedom]

Some useful data points on the Speech API:

• Speech Synthesis is pretty widely deployed
• Speech Recognition not so much but has ongoing implementation work in other browsers

A more in-depth technical dive into the API and its implementation state: https://youtu.be/gJ0gYki6F9k?t=862

The work has intersections with other topics:

• VR, AR/MR (speech recognition is a useful way to interact with a immersive systems)
• WebRTC (e.g. for automatic video captioning)
• AI (lots of progress in speech recognition / synthesis made possible with machine learning, lots of AI interactions based on voice interactions)
• IOT with voice as a way to give input to connected objects

[svgeesus]

I notice the frozen final report https://dvcs.w3.org/hg/speech-api/raw-file/tip/speechapi.html
and also an unofficial version with errata https://dvcs.w3.org/hg/speech-api/raw-file/tip/webspeechapi.html
which brings up the maintenance problem for work from CGs.

Certainly Speech API has way more traction than the old Aural CSS ever had.

[travisleithead]

The TAG started an issue on this topic (w3ctag/design-reviews#214), but intends to close it as the better home for future design and strategy planning seems to be in this repo. (Said another way: there's not currently implementer interest in driving a design forward, and no one is looking to the TAG's review for feedback. We have some thoughts on what a good design would be--see below--but hope putting the information here will be useful at such a time as implementer interest is rekindled.)

Design wishlist for an improved Web Speech API (Synthesis and Recognition):

Synthesis

• The original spec predates Web Audio. Web Audio already offers a lot of the controls that the speech-synthesis API duplicates--but with much more control fidelity. A speech Synthesis should likely integrate nicely into the Web Audio graph to allow voice customization using the entire Web Audio pipeline.
• Proper buffering/notification/tracking of what requested text has been spoken. Similar to how WebVTT specifies timing for text tracks, getting the timing tracking for speech-synthesis is critical to any scenario that need input/output using speech (so they can know the precise time to start listening after sysnthesis finishes speaking).
• Voice selection. This may not be possible to standardize, but being able to reliably enumerate and select a voice to use from the set of available voices (some voices may need to be acquired asynchronously as an extensible browser package), will be an improvement to the current API.

Recognition

• Some recognition processing can happen in the cloud, some can be done locally. UA's may be able to dynamically switch between the two. The API should accommodate an abstraction that does not depend on either local or remote processing--but may be able to take requirements in the form of latency requests that can dictate how the implementation chooses to service the request (perhaps that the cost of quality).
• Speech-to-text seems like the default output, however, there may be an opportunity for speech to "command" or generate proxy actions on behalf of the speaker such as pointer-clicks, keyboard presses, or higher-level actions like those associated with the ARIA model. From the developer standpoint, the API could (at a minimum) provide a way to process command strings as input, and provide signals and context for when those commands are vocalized in speech recognition.
• Users of the API will come from different world regions with different languages and dialects. Speech-language detection will be critical to enable recognition to be successful. The API should be careful as things like detected language can be a new source of user-fingerprinting (literally "voiceprinting") which the spec author should be mindful about exposing--ensuring there are good developer use-cases for such data.

[wseltzer]

thanks @travisleithead
Indications of implementer interest are welcome here.

[koalie]

Related to #134

[draggett]

There is good browser support for Web Audio and WebRTC, making it feasible to implement speech recognition and synthesis server side for voice interaction with web pages. What's missing is a standard markup language and scripting API that can be used for both text and voice based chatbots, see Chatbots#134

[slightlyoff]

From today's TAG F2F to add to @travisleithead's comments:

• In general, these features aren't well layered. Integration of synthesis with web audio is important, but so is recognition from other platform audio sources. Thus far, the recognition API doesn't allow a stream of audio from <audio>, <video>, RTC media track, or Web Audio to serve as input. That suggests poor layering.
• Would like to see the API surface area modernized. The recognizer API, e.g., could benefit from promises, iterators, and perhaps Streams
• The recognition.start() method predates both promises and the Permissions API. It would be helpful to model this that way.
• We're missing extensibility for recognition and plabyck. Like Payment Request Handler, we should be able (perhaps) to register hot-wording and recognition backends at the user's behest as well as pluggable default synthesis.

[nigelmegitt]

as well as pluggable default synthesis.

A point made (by me) and supported (by others) in the web speech breakout at TPAC was that a standardised "web voice" resource format would be helpful, along the same lines as a web font, so that a site wanting to use a specific voice, e.g. for branding purposes, could provide it for use in the context of presenting a particular page.

[bkardell]

It's not clear to me if there was ever any follow up here - currently this is a CG thing, and there is no more CG afaik... Is there an interest in transitioning it to an existing WG, or a new one? Have there been some efforts to revitalize work, even in the sense of just making an official spec that reflects inter-operable implementation realities as a start?

[dontcallmedom]

I very briefly discussed this with @cwilso end of June, who indicated willingness to give it another try if there was enough interest; maybe a TPAC breakout would help figuring out next steps?