Discussion thread for an `<usermedia>` element

[andypaicu]

I've added an explainer which contains a proposal for a <usermedia> element. I would like to start a discussion thread based on this explainer. Let me know what you think of it @mikewest @martinthomson @jan-ivar.

[jan-ivar]

Thanks @andypaicu I like this direction, and I think you've identified the right 3 elements. Lots of detail in usermedia_element.md so pardon the brevity on each topic (I realize each one is a potential conversation starter):

1. I'd prefer discrete <camera>, <microphone>, and <usermedia> elements since each one is inherently different in some way: default visuals, valid inputs (constraints), and behavior. E.g. we don't want a <usermedia> combined toggle.

2. I like the idea of <camera> and <microphone> being user-driven toggles with strong guarantees. So I think they need to control track.muted (i.e. UA mute), not track.enabled (which JS can alter). This also solves event firing.

3. I like the stream attribute as a means to give JS the result from gUM. But as written, it sounds like if I create two <camera> elements, and the user picks the same source in both, then the two toggles only toggle their half of the tracks, giving the user two independent toggle states to manage. This seems problematic. I'd instead tie element control to the chosen source. This should map well with 2 above as well (mute is a source property).

4. Regarding constraints, less is more. I suggest element.setInitialConstraints(constraints) to distinguish them from their live counterparts in element.stream.getVideoTracks()[0].getConstraints() which are modifiable through element.stream.getVideoTracks()[0].applyConstraints(constraints). I fear anything more elaborate would draw attention and confuse web developers into thinking the element ones are live.

5. "Exclamation mark” badges to convey error states seems like a nice idea but secondary to the core model. Do websites need help here, or can they handle error states themselves? (Meet appears to treat lack of permission as an error state, adding some confusion to the screenshots)

6. Selection dropdown — Looking good. I'm all for it!

[andypaicu]

Thank you for thoughtful feedback, I agree that there will likely be a lot of long discussions that we'll probably need to have on these various aspects. My thoughts below:

1. I see the elements as a lot more alike than different: ultimately the behavior of a <usermedia> element is that it calls getUserMedia with the specified constraints and then mutes/unmutes the resulting stream. If we were to separate it into 3 elements the only real tangible difference I can see is that we don't want a combined toggle (though I'm not actually sure that there's any benefit to forbidding it by design - I can't see the harm in it if it was allowed). Things like icon/text are not really differences at the semantic level and user agents don't need the separation to implement the different icon/text. And the constraints are a JSON dictionary regardless.

2. I assume that if that was the case it would mean that sites can't do things like the "are you trying to speak" detection? Sites ultimately control if the element is in the page or not. So if our toggle element means degraded functionality, sites will likely just replace it with their own buttons after the stream has been started.

3. Would this mean that muting a source in one tab would result in it being muted in another tab as well (since the source itself is muted)? Or are MediaStreamTrack sources scoped to the browsing context? If it's the former, I really worry about it being unexpected behavior for users and potentially quite compromising in various situations (imagine a user that joins two meetings at the same time for whatever reason, and them unmuting to speak in one meeting but accidentally speaking in both meetings). If it's the later, I have no complaints except for what was raised at point 2.

4. I'm afraid I don't really understand the context of this feedback and what it's referring to. Could you elaborate?

5. The issue is that the base <permission> element (and therefore the <usermedia> element(s)) do not allow other HTML elements to overlap them (the element becomes invalid). Because of this sites wouldn't be able to simply add badges to the element as they do today to their existing UI. We either need to support badges in some way, or rethink the security restriction that doesn't allow overlap. Either way this is probably solved at the <permission> element level, but if the goal is to allow sites to replace their mute/unmute toggles it is a dependency we will need to address.

6. Just for clarification since I'm paranoid: the selection drop-down is not really part of the proposal, just existing UI. It's there to showcase a scenario in which sites will still sometimes need to call gUM on their own and then associate the obtained MediaStream with a <usermedia> element.

[simon-friedberger]

The enabled property can also be set by developers to programmatically mute/unmute all associated tracks.

Doesn't this defeat the purpose of having muting be done by the browser to give the user more control?

[andypaicu]

See points 2. and 3. above about why I don't think it would be a good idea to do this, at least for this initial version:

Considering the element's existence on the page depends on sites keeping it in the page, doing something like this will just push developers switch to their regular UI elements for muting/unmuting since they can no longer provide the same experience if the source itself is muted. Plus I still am unsure if muting the source can be done independently for each browser context, if it's a global thing I really don't think it would be a good idea as muting/unmuting in one tab would affect a separate tab.

I don't think muting the source can be done yet without also adding measures which force developers to use the element as the mute/unmute button after the stream started, which would likely need changes to the JS API itself to make it more difficult to use in some way without having the dedicated elements. I think for the future this seems like a good direction to move towards but for this initial version I don't think it's achievable.

[simon-friedberger]

they can no longer provide the same experience if the source itself is muted

What exactly is this user experience? There may be cases where admins can unmute but I'm not sure it's worth supporting that. It doesn't really work for mute buttons on headsets and webcam privacy sliders anyway.

doing something like this will just push developers switch to their regular UI elements

We don't necessarily have to defend against this. It would still make the new element better and it's much more likely that we might change only the old API instead of the old and the new option. Also, given that this is supposed to provide benefits over the legacy API developers should still be incentivized to use it.

if it's a global thing I really don't think it would be a good idea as muting/unmuting in one tab would affect a separate tab.

I'm not sure how often this is relevant in practice but it's a good point. It should probably apply per-tab even if the permissions are granted per origin.

[andypaicu]

they can no longer provide the same experience if the source itself is muted

What exactly is this user experience? There may be cases where admins can unmute but I'm not sure it's worth supporting that. It doesn't really work for mute buttons on headsets and webcam privacy sliders anyway.

I'm referring to the "your microphone is muted" detection feature which videoconferencing sites use to highlight that you might be speaking while mute. I assume if we mute the source, this feature would no longer work (though if I'm honest I'm not sure about the details of how it works in the first place).

[jan-ivar]

1. I see the elements as a lot more alike than different...

How would web developers or users benefit from a combined element with a type= property?

2. I assume that if that was the case it would mean that sites can't do things like the "are you trying to speak" detection?

Good point, though see w3c/mediacapture-extensions#164 for a potential long-term solution to that.

...Sites ultimately control if the element is in the page or not. So if our toggle element means degraded functionality, sites will likely just replace it with their own buttons after the stream has been started.

I concede this is a good point. We may need to settle for something simpler, at least in the short-term.

3. Would this mean that muting a source in one tab would result in it being muted in another tab as well (since the source itself is muted)? Or are MediaStreamTrack sources scoped to the browsing context?

A "source" in spec terms is understood to be an independently stoppable abstraction per global object. No cross-channel implied, as that would be bad obviously.

The spec is flexible, offering UAs a way to mute individual documents for each device exposed. But no browser uses all of that flexibility. So far, Safari has used this to implemented a "Pause" feature in the URL bar (per top-level document, all devices).

FWIW we did create a way for websites to undo track.muted in w3c/mediacapture-extensions#39 (comment) through the mediaSession API (though notably not per-source). But it satisfied Safari's needs at least. If we end up going the muted route, we'd probably need to pay attention to that.

If we stick to modifying track.enabled (or maybe just firing click?) we may face different (possibly simpler) problems.

4. constraints

Constraints are inputs to gUM that survive as mutable track properties. If constraints become modifiable properties of the element, what are web developer expectations from modifying them after the element has called gUM?

[andypaicu]

Apologies for the delay I have been OOO quite a while in the last month (both vacation and sick leave).

1. I think the benefit is that we don't have 3 different elements which behave identically, I think practically speaking the difference will be minimal between the 2 options (1 element vs 3 elements), so I'm just approaching this from a purely design perspective (and I think we can agree that from a purely design perspective if the elements behave identically there should only be 1 element instead of 3). The differences outlined initially (different text/icon) are mostly superficial, and I don't see a reason why there shouldn't be a combined toggle. However it seems that WebRTC WG folks have some interest in this so perhaps it will be more constructive if that discussion happens between other people which know more than me.

2-3. In that case I think it's fair to mute the source itself, assuming the missing functionality will be covered via different APIs.

4. Good point I did not consider the mutable nature of the constraints. I can see 2 options: either the constraints are immediately applied when changed, or they are only applied in the initial getUserMedia call. I'm leaning towards the later option as the the more flexible alternative as it allows developers to apply the constraints via JS if they wish to.

[jan-ivar]

Deep-dive on 1: It seems § 3.7. Give each HTML element a single purpose applies: "Rather than defining elements that can operate in different modes, having a separate element for each mode is preferable. This simplifies usage for authors."

HTML tends to give distinct things distinct elements like <audio> and <video>, not polymorphic <media type="video"> changing semantics at runtime. This helps authors, css, tooling, and a11y.

I think a combined button is likely to have different semantics and not be a toggle. I see no sign of immediate demand for a combined toggle perusing existing websites. But should that arise later, I don't see a problem with adding a third element with clearly scoped behavior.

[mharbach]

@jan-ivar -- We would love to continue this discussion at TPAC. @mikewest and myself will be there. Can you maybe help us to get this on the agenda for the WebRTC meeting? Or should we propose a breakout? Wdyt? 🙏

[jan-ivar]

@mharbach happy to put this on the WebRTC agenda. Do you have a preference for Tuesday or Thursday?

[mharbach]

I think Tuesday is preferable for me as the Thu slot clashes with a few other meetings I might attend. 🙏

[tommygaessler]

Hi @andypaicu, and team, first off, PEPC has improved the UX on our web platform and decreased errors around getUserMedia significantly. Thank you for the collaboration on this.

We do want to offer the "are you trying to speak" detection, as well as a user previewing their video, for example, picking their virtual background before publishing the video stream to the other participants.

Hope you all enjoy TPAC!

[mneubert-1]

@andypaicu shared the <usermedia> proposal with my team. Sharing some summarized comments from the discussion we had:

In our case, we believe it could be useful for handling permission requests, especially if the element is used temporarily for the duration of the prompt. However, we have concerns about using a persistent <usermedia> element as the primary means of controlling mute state within a web application, particularly for more complex scenarios:

Separation of Concerns:
Many applications separate the management of MediaStream state for different purposes. For example, we might want to allow a local preview of the camera even when the user is "muted" in a call to allow users to test out their equipment or video effects. This element seems to couple the UI control directly to the track's state, potentially conflicting with this separation.

Mute Mechanism:
Some applications manage enabled for reasons other than just user-initiated mute. Tying the element to the enabled property of MediaStreamTrack could interfere with application logic.
Using the muted property might be more aligned, but it raises questions about the scope of this mute action. Is it device-level? If so, how does the user unmute if the <usermedia> element isn't always visible? Is there a risk of getting into an unrecoverable state?

Application Complexity:
Applications often have multiple ways to control mute state (e.g., UI buttons, keyboard shortcuts). Streams are also frequently cloned and transformed. A single UI element bound to the original track state might not integrate well with these more complex architectures.

Abstraction Layers:
We view mute control as a higher-level abstraction than raw device access. Mute state typically applies to a track, which can be independent of the underlying source. Introducing mute control at the device access layer could add complexity.

[mehagar]

I generally agree with the comments from #62 (comment).

• Just like Meet, our app (Zoom) can show a preview of the user's video even if the user is muted in the call.
• We support the "are you trying to speak" feature, which seems would not be supported according to this comment: Discussion thread for an <usermedia> element #62 (comment) .
• We customize our camera/microphone icons and error UI to look like our native clients, so we wouldn't benefit from the built-in UI provided by these new elements.

My read is that this new version of the API is intended to give more users control over how their camera and microphone is used on a web page. However, as developers, coupling the mute state of the source with the mute state of the application seems to limit the existing functionality and UI of our application, which is not acceptable.

As is, I would prefer the way the existing permission element works over this new proposed version. I would also like to know if this new element supports the same use cases as the existing element. Can we just use it to obtain permissions, without dealing with the returned stream?