API for planes and meshes

[blairmacintyre]

In #6 we seem to have settled on the idea of allowing developers to request the kinds of data that actually want, be it planes of various kinds, meshes or other things in the future (tables, the ground, etc ... anything platforms might be able to provide). To do this, we need to update the proposal to as follows. Initially, focusing on planes and meshes should be sufficient to test across a good number of platforms. (We might also be able to test faces in the WebXR Viewer).

We need:

• a way of requesting (and checking the success of) the specific types. For example, I might ask for meshes, and if those are unavailable, ask for planes (and then convert them to meshes myself). This may be done by adding features to the main webxr api.
• a way of configuring each of these (already there with xrSession.updateWorldTrackingState). (Suggestion: change this to xrSession.updateWorldSensingState, as we will probably move this to the "Core AR Module" when that happens, and use it for other sorts of sensing configuration as well)
• a way to get the data (already there with frame.worldInformation

[cabanier]

Unlike planes, meshes are very big and change constantly. I think we can create a module that just deals with meshing.
The simplest approach would be to add an optional parameter to requestSession so an author can request additional permissions. The UA can then ask the user if it's ok to start an immersive session with world and hand meshing. For instance:
navigator.xr.requestSession("immersive-ar", { hand-meshing: "high-quality", world-meshing: "medium-quality"]) .then((session) => { ... });

I'm a bit unsure what the best approach for the delivery of these meshes is.
We could do it with a custom event so one could write session.addEventListener("meshdata", function(mesh){..)); but that would require more bookkeeping on the UA side.
An alternate approach would be to add optional parameters to XRFrameRequestCallback.

For world mesh data, the UA would pass chunks of vertices, each of them representing a mesh with a unique id. Subsequent calls could remove meshes, add new ones or update existing ones.
For hand mesh data, the UA would pass a new set of vertices.
@blairmacintyre, what do you think?

[blairmacintyre]

@cabanier why not just build on the current proposal?

As described, there would be a way to request meshes similar to planes in updateTrackingState. The reason I liked that part of the proposal is that it could be called at any point, and changed as the session goes on: the kind of data we get should probably be independent of the session creation. Session options should probably be limited to just "I want world geometry or not" for many apps, since we should encourage a pattern of (a) having apps do their best with what the UA can provide, and (b) have UAs provide some common fallback for world geometry that all apps can fall back to.

I would suggest that "mesh" is that fallback. Since planes, faces, objects, etc can all be represented as meshes, even if not ideal. So, I could do something, even if I'd prefer planes. Perhaps some apps would fail, but I would hope that frameworks will emerge that will polyfill things (e.g., a UA that provides only meshes might work with an app+framework that polyfills a plane finder as some WASM blob running in a worker)

Given all that, meshes would be delivered each frame in the worldInformation field as proposed.

This is exactly what I did in the sample code / API I posted in #6

I based the structure of the mesh data on the current Windows MR data returned by Hololens; it would be interesting to see if that structure works for ML, or if there is something missing.

[cabanier]

@cabanier why not just build on the current proposal?

Are you referring to this proposal?
I think it's not that different from mine.

Having the request for meshes (and potentially planes) be part of requestSession will make it so the browser can ask for the permissions at the time that the immersive session starts. Your updateWorldSensingState call happens while the session is established so you can no longer ask for permission.

The reason I liked that part of the proposal is that it could be called at any point, and changed as the session goes on

It makes sense that you can turn meshing on and off but that could be done with some boolean on the session. I suspect that most authors know what features and feature quality they want so there's no need to change it at runtime.

I would suggest that "mesh" is that fallback. Since planes, faces, objects, etc can all be represented as meshes, even if not ideal. So, I could do something, even if I'd prefer planes.

I think the device itself will always know better what a plane is. Authors should not have to check on what device they are running and pick an algorithm based on that.
Plane detection is a one-off API call (which will likely be promise based) and different from a mesh (which will continually update)

I based the structure of the mesh data on the current Windows MR data returned by Hololens; it would be interesting to see if that structure works for ML, or if there is something missing.

Yes, this structure could work for us.

[blairmacintyre]

I think we agree, @cabanier

Implicit (I should have made it explicit) in my current thinking is that we would have to ask for spatial geometry in requestSession, for permission management. Thanks for highlighting that.

I would argue that we should ask for a general world-geometry permission there, and then use updateWorldSensing to provide the details. This way, apps can change what they want over time. We can't turn in on or off in the session if we want to do it within the same session.

Regarding devices: I didn't say authors would check what device they are on and then choose. I said they would attempt to get the kind of information they want, and fall back if it's not available. There is no guarantee any given device will provide all the variations of data that we expose. For example, a given device might only provide meshes, and leave it up to the app to find planes -- I believe that's what most HMDs do now, right? Hololens doesn't provide planes, just meshes. Does ML currently provide planes? From the OS level, or provide a utility library?

[cabanier]

I would argue that we should ask for a general world-geometry permission there, and then use updateWorldSensing to provide the details.

At least for us, hand sensing uses a complete different set of APIs and sensors. It's also less exposure from privacy standpoint I suspect so it makes sense to make it a separate permission check.

Regarding devices: I didn't say authors would check what device they are on and then choose. I said they would attempt to get the kind of information they want, and fall back if it's not available. There is no guarantee any given device will provide all the variations of data that we expose. For example, a given device might only provide meshes, and leave it up to the app to find planes
-- I believe that's what most HMDs do now, right? Hololens doesn't provide planes, just meshes. Does ML currently provide planes? From the OS level, or provide a utility library?

I'm unsure what most devices do. ML has a rich set of APIs for plane detection and it doesn't return meshes (which is why I disagreed with your previous statement that planes are meshes too).
So, I think that for plane detection we need another structure.
For us, a "plane" is a rectangle of a certain width and height in 3d space.

Since we're so close in agreement on how meshing would work, maybe I can draft a more complete IDL that we can discuss. Does that sound reasonable @blairmacintyre ?

[blairmacintyre]

(Aside: note that your definition of plane is different from the ARKit/ARCore one)

Sure, go ahead and draft something.

[cabanier]

New classes to request or ask for support of world and local meshing:

enum XRMeshingOptions {
  "off",
  "low-quality",
  "medium-quality",
  "high-quality
};

dictionary XRSessionOptions {
  XRMeshingOptions worldMeshQuality = "off",
  XRMeshingOptions nearMeshQuality = "off"
};

Then supports/requestSession are extended:

Promise<void> supportsSession(XRSessionMode mode, XRSessionOptions? options);
Promise<XRSession> requestSession(XRSessionMode mode, XRSessionOptions? options);

Structures to hold the mesh data:

dictionary XRMeshBlock {
  required Float32Array vertices;
  required Uint16Array indices;
  Float32Array? normals;
};

[
    SecureContext,
    Exposed=Window
] interface XRMesh {
  readonly maplike<DOMString, XRMeshBlock>;
};

dictionary XRMetaData {
  XRMesh? worldMesh;
  FrozenArray<XRMeshBlock>? nearMesh;
};

XRMetaData can contain an optional set of world or near meshes.
XRMesh contains a unique id for each mesh.

Mesh data is then supplied to the page with an updated XRFrameRequestCallback

callback XRFrameRequestCallback = void (DOMHighResTimeStamp time, XRFrame frame, optional XRMetaData metaData);

• If local mesh data is provided, it will replace the existing local mesh. A vertex count of 0 means no local mesh.

• If the same unique id is passed for a world mesh, it will replace the existing mesh.

• If a world mesh with a unique id already exists and the new vertex count is zero, the mesh should be deleted.

[blairmacintyre]

I'd like @thetuvix to comment here, since the Hololens APIs are also dealing with this.

[bialpio]

Couple of questions and minor comments:

• Do we plan on allowing the application to reconfigure the mesh sensing after a session was already created, or can configuration only happen at session creation?
• It might be better to use DOMPointReadOnlys for vertex data and for normals - this is what main WebXR spec seems to be using to represent points / vectors.
• Does the order of the XRMeshBlocks matter in XRMetaData.nearMesh? If not, can we add an intermediate type XRMeshBlockSet? It could be a readonly setlike<XRMeshBlock> that we’d store in XRMetaData under nearMesh key.
• I’d propose that we add the XRMetaData into xrFrame.worldInformation object - this way, we’ll keep all world-sensing data in one place.
• Keys in a dictionary are optional by default, so the question marks are not needed for XRMeshBlock.normals, XRMetaData.worldMesh & XRMetaData.nearMesh. I don’t fully understand all other implications of using dictionaries here, I’ve seen them being used mostly for initialization purposes in main WebXR spec.

I’m not sure if I understand the protocol for mesh updates across frames, can you elaborate a bit more on that? Sample code would be great!

[cabanier]

• Do we plan on allowing the application to reconfigure the mesh sensing after a session was already created, or can configuration only happen at session creation?

In my current proposal, there is no way to reconfigure the mesh quality. I think it makes sense to provide a way to turn the meshing on and off.
Meshing is an expensive operation so this could help performance if an author thinks that the mesh is "good enough".

I have not seen any applications that ask for different qualities at runtime.

• It might be better to use DOMPointReadOnlys for vertex data and for normals - this is what main WebXR spec seems to be using to represent points / vectors.

Even though that would make it easier to inspect the data in Javascript, it would be a very expensive operation to pass that data to WebGL because hundreds of meshes with thousand's of vertices need to be translated (and each vertex point would be a javascript object).

• Does the order of the XRMeshBlocks matter in XRMetaData.nearMesh? If not, can we add an intermediate type XRMeshBlockSet? It could be a readonly setlike<XRMeshBlock> that we’d store in XRMetaData under nearMesh key.

That sounds fine! What would be the advantage of doing it that way?

• I’d propose that we add the XRMetaData into xrFrame.worldInformation object - this way, we’ll keep all world-sensing data in one place.

By passing it into the call, it's more clear that this is new additional information. Would xrFrame.worldInformation contain the entire mesh, or only the updated ones (and how would you detect that?)

• Keys in a dictionary are optional by default, so the question marks are not needed for XRMeshBlock.normals, XRMetaData.worldMesh & XRMetaData.nearMesh. I don’t fully understand all other implications of using dictionaries here, I’ve seen them being used mostly for initialization purposes in main WebXR spec.

ok.

I’m not sure if I understand the protocol for mesh updates across frames, can you elaborate a bit more on that? Sample code would be great!

Sure! Let me dig my test code up and I'll post it here

[bialpio]

• It might be better to use DOMPointReadOnlys for vertex data and for normals - this is what main WebXR spec seems to be using to represent points / vectors.

Even though that would make it easier to inspect the data in Javascript, it would be a very expensive operation to pass that data to WebGL because hundreds of meshes with thousand's of vertices need to be translated (and each vertex point would be a javascript object).

@cabanier, @blairmacintyre, @thetuvix - what are the primary use cases for meshes? A couple that come to my mind are: rendering / occlusion, physics simulation (ex. bouncing virtual objects off of real ones), object placement (including reticle display), AI (controlling virtual agents based on the environment they’re in). For some of them, GL-friendly format is definitely the way to go, but others might be better off with something more JS-friendly.

• Does the order of the XRMeshBlocks matter in XRMetaData.nearMesh? If not, can we add an intermediate type XRMeshBlockSet? It could be a readonly setlike<XRMeshBlock> that we’d store in XRMetaData under nearMesh key.

That sounds fine! What would be the advantage of doing it that way?

If the XRMeshBlock object can be reused across frames, it’d be slightly simpler for the application to check that a particular mesh block was already rendered & potentially reuse the GL buffers (& it won’t imply that the order matters):

let nearMeshBlocks = …; // Near mesh blocks set from previous frame.

// Below happens somewhere in the RAF callback.
metaData.nearMesh.forEach(mesh => {
  if(nearMeshBlocks.has(mesh)) {
    // Current mesh still contains that particular mesh block.
    // No need to create GL buffers, we can reuse the ones created in previous frame.
  } else {
    // We haven’t seen that block, create GL buffers,
    // store them in a mesh block for rendering & reuse.
    someMeshBlock.context = { /* … all the data needed to render the mesh block … */ };
  }
});

// Some time later, render the mesh block.
...
// Before exiting RAF callback, store the current near
// mesh blocks for the next frame to use.
nearMeshBlocks = metaData.nearMesh;

Can we revisit this after you post some example code to demonstrate how the updates are communicated to the app? That’s the piece that I feel I’m still missing.

• I’d propose that we add the XRMetaData into xrFrame.worldInformation object - this way, we’ll keep all world-sensing data in one place.

By passing it into the call, it's more clear that this is new additional information. Would xrFrame.worldInformation contain the entire mesh, or only the updated ones (and how would you detect that?)

I’d suggest that xrFrame.worldInformation should contain exactly the same thing that you’d pass in to the request animation frame callback (IIUC, it’s an entire mesh now, w/ some information about mesh block removals). My comment was due to 2 reasons. First is that I think it’ll send a stronger signal that this mesh data is valid (potentially) only in this particular xrFrame if we make the XRMetaData a member of XRFrame. Second reason is that we’ll probably keep adding more and more objects that are potentially going to be changing on a frame by frame basis (planes, anchors, lighting estimation come to my mind) - depending on how we decide we’ll pass those to the application, it might turn out that the request animation frame callback suddenly has way more parameters than we’d like. I think it might be valuable to group all of those in some XRFrame member - that was one of the reasons for the xrFrame.worldInformation object.

It just occurred to me: we should probably make the mesh contain a XRSpace - otherwise, we’d hand out vertex coordinates w/o specifying what are they relative to.

[cabanier]

Here is my three.js sample code:

 let metadata = renderer.vr.consumeMetadata(); // returns metadata of the frame

 if (metadata !== null) {
     let vrWorldMesh = metadata.worldMesh;
     if (vrWorldMesh) {

        console.log("got meshdata");
        vrWorldMesh.forEach((block, uuid) => {
            if (uuid in meshes) {
                var three_mesh = meshes[block.uuid];
                group.remove(meshes[block.uuid]);
            }
            if (block.vertices.length == 0) {
                return;
            }

            var geometry = new THREE.BufferGeometry();
            geometry.addAttribute('position', new THREE.Float32BufferAttribute(block.vertices, 3));
            geometry.setIndex(Array.prototype.slice.call(block.indices));

            meshes[uuid] = new THREE.Mesh(geometry, worldmaterial);
            meshes[uuid].castShadow = true;
            meshes[uuid].receiveShadow = false;
            meshes[uuid].doubleSided = true;
            group.add(meshes[uuid]);
        });
    }

    let vrNearMesh = metadata.nearMesh;

    if (vrNearMesh !== null) {
        console.log("got nearmeshdata");
        if (nearMesh !== undefined) {
            group.remove(nearmesh);
            nearMesh = undefined;
        }

        vrNearMesh.forEach(block => {
            if (block.vertices.length == 0) {
                return;
            }
            var geometry = new THREE.BufferGeometry();
            geometry.addAttribute('position', new THREE.Float32BufferAttribute(block.vertices, 3));
            geometry.setIndex(Array.prototype.slice.call(block.indices));

            // this code should be updated to handle more than 1 near mesh
            nearmesh = new THREE.Mesh(geometry, nearmaterial);
            nearmesh.castShadow = true;
            nearmesh.receiveShadow = true;
            nearmesh.doubleSided = true;
            group.add(nearmesh);
        });
    }
}

This is called in the render method

[cabanier]

• It might be better to use DOMPointReadOnlys for vertex data and for normals - this is what main WebXR spec seems to be using to represent points / vectors.

Even though that would make it easier to inspect the data in Javascript, it would be a very expensive operation to pass that data to WebGL because hundreds of meshes with thousand's of vertices need to be translated (and each vertex point would be a javascript object).

@cabanier, @blairmacintyre, @thetuvix - what are the primary use cases for meshes? A couple that come to my mind are: rendering / occlusion, physics simulation (ex. bouncing virtual objects off of real ones), object placement (including reticle display), AI (controlling virtual agents based on the environment they’re in). For some of them, GL-friendly format is definitely the way to go, but others might be better off with something more JS-friendly.

The biggest use case is occlusion.
I personally like to use the near mesh to show my hands as a wireframe because it looks so cool :-)

For detecting objects and interacting with the environment, I suspect that simpler and stable constructs are the better way to go. Generally, you just want to push a mesh directly into a GL call.

Of course, if the platform can't support object detection, an author would have to pull apart the raw data.
I'm unaware if there are such platforms though...

• Does the order of the XRMeshBlocks matter in XRMetaData.nearMesh? If not, can we add an intermediate type XRMeshBlockSet? It could be a readonly setlike<XRMeshBlock> that we’d store in XRMetaData under nearMesh key.

That sounds fine! What would be the advantage of doing it that way?

If the XRMeshBlock object can be reused across frames, it’d be slightly simpler for the application to check that a particular mesh block was already rendered & potentially reuse the GL buffers (& it won’t imply that the order matters):

let nearMeshBlocks = …; // Near mesh blocks set from previous frame.

// Below happens somewhere in the RAF callback.
metaData.nearMesh.forEach(mesh => {
  if(nearMeshBlocks.has(mesh)) {
    // Current mesh still contains that particular mesh block.
    // No need to create GL buffers, we can reuse the ones created in previous frame.
  } else {
    // We haven’t seen that block, create GL buffers,
    // store them in a mesh block for rendering & reuse.
    someMeshBlock.context = { /* … all the data needed to render the mesh block … */ };
  }
});

// Some time later, render the mesh block.
...
// Before exiting RAF callback, store the current near
// mesh blocks for the next frame to use.
nearMeshBlocks = metaData.nearMesh;

Can we revisit this after you post some example code to demonstrate how the updates are communicated to the app? That’s the piece that I feel I’m still missing.

Yes. I pasted my sample code above and that is what I'm doing: only update the meshes that are new and leave the others as-is.

• I’d propose that we add the XRMetaData into xrFrame.worldInformation object - this way, we’ll keep all world-sensing data in one place.

By passing it into the call, it's more clear that this is new additional information. Would xrFrame.worldInformation contain the entire mesh, or only the updated ones (and how would you detect that?)

I’d suggest that xrFrame.worldInformation should contain exactly the same thing that you’d pass in to the request animation frame callback (IIUC, it’s an entire mesh now, w/ some information about mesh block removals).

Only deltas are passed into the callback but I guess appending them to xrFrame works.

My comment was due to 2 reasons. First is that I think it’ll send a stronger signal that this mesh data is valid (potentially) only in this particular xrFrame if we make the XRMetaData a member of XRFrame. Second reason is that we’ll probably keep adding more and more objects that are potentially going to be changing on a frame by frame basis (planes, anchors, lighting estimation come to my mind) - depending on how we decide we’ll pass those to the application, it might turn out that the request animation frame callback suddenly has way more parameters than we’d like. I think it might be valuable to group all of those in some XRFrame member - that was one of the reasons for the xrFrame.worldInformation object.

Yes. I agree that xrFrame could become a container for more information.

It just occurred to me: we should probably make the mesh contain a XRSpace - otherwise, we’d hand out vertex coordinates w/o specifying what are they relative to.

Yes, I punted on that for now. The mesh should have an associated XRSpace. For now I assume it's in the same reference space as the one passed to getViewerPose

[cabanier]

@blairmacintyre @bialpio I created a proposal on my personal github: https://cabanier.github.io/real-world-geometry/webxrmeshing-1.html
Can you take a look and let me know what you think?

[bialpio]

Thanks, the example code helps a lot!

For detecting objects and interacting with the environment, I suspect that simpler and stable constructs are the better way to go. Generally, you just want to push a mesh directly into a GL call.

In that case, maybe we can design this API in a way that would directly return WebGL buffers and optionally provide a JS-friendly access only if needed by the app? This way we could maybe skip moving significant amounts of data?

Can we revisit this after you post some example code to demonstrate how the updates are communicated to the app? That’s the piece that I feel I’m still missing.

Yes. I pasted my sample code above and that is what I'm doing: only update the meshes that are new and leave the others as-is.

Based on the examples, it seems that in your proposal you’re not reusing the mesh block objects across frames, so it might not change that much to use a setlike<XRMeshBlock>. OTOH, the XRMesh doesn’t have to be a maplike<DOMString, XRMeshBlock> & your sample code would still work (it relies on the fact that XRMeshBlock has an uuid) - I think we could make XRMetaData simply contain 2 keys, nearMesh and worldMesh that would both be XRMeshes, and XRMesh would be a selike<XRMeshBlock>. All this is just me spit-balling the possibilities, take them or leave them. :)

Only deltas are passed into the callback but I guess appending them to xrFrame works.

Now I think I see it - the mesh block “updates” also are passed in, but there is only one case of a mesh block update and that is a mesh block removal (signaled by passing 0 vertices in an already-existing block).

As for deltas: is it possible to get the full state of the currently detected mesh if the app missed some of the updates? Is the “app missed an update” something that we should worry about? I’d generally try to make an API that still allows the app to get back in sync with current state, but maybe that’s not the goal here.

I created a proposal on my personal github: https://cabanier.github.io/real-world-geometry/webxrmeshing-1.html

Thanks! I didn’t have a chance to take a deeper look at it and the PR yet, but the early feedback that I can give is that usually we start with an explainer that describes use cases but does not include a specific Web IDL proposals since it’s more difficult to reach consensus on a specific API proposal if we don’t first agree on what we’re trying to solve. :)

Just so you know, we’re not currently planning on spending too much time on mesh detection APIs - if you’re interested in driving the discussions around the API, I can give you permissions to the repo so you won’t have to wait for me to approve the PRs & plane detection and meshes could still live side by side in this repo.