Exposing PhysX, OpenVR, and other libraries to the web

[shawwn]

As mixed reality matures, it is increasingly evident that physics will play a big role in how developers create influential experiences. Whether it's basic collision detection, IK, or breakable walls, physics is one of the mainstays of MR.

I think the best physics engine will tend to win, and right now that appears to be PhysX. It was a bit hard to determine, but PhysX is in fact free to use and to incorporate into products, even though you have to jump through a couple hoops to get the source code. You also have to notify them whenever you're incorporating it into a commercial product, but you're still able to use it for free. I think the only requirement is including their branding somewhere in your product. I'm fine with that.

That said, I want to create mixed reality games powered by PhysX. There are a few ways to do this:

1. Incorporate PhysX directly into exokit, and expose it directly to the web.

2. Define an XRPhysics spec, and implement it in terms of PhysX / other physics engines.

3. Allow websites to tell Exokit "Download this node module and expose it to my website."

I think option #3 is the most attractive. It's impossible in general to prevent people from doing this, except by establishing a lot of inertia and becoming yet another gatekeeper. And it's the most flexible approach when security isn't a concern.

It seems like a losing battle to define a specification for every aspect of mixed reality -- the specs are usually lowest common denominator functionality, and exokit proves it's no longer necessary to tolerate such things.

We plan to incorporate PhysX into our fork of exokit in some fashion. If I were to write it as a node binding (similar to the magicleap binding), and exokit were to implement strategy #3, then all forks would be able to use the Physx binding rather than creating a balkanized race-you-to-acquire-users situation per fork.

There are other situations I've wanted this strategy too, e.g. to get some more advanced functionality out of OpenVR than what WebVR/WebXR is willing to provide. If a website could indicate "download and require this node module, then expose it to me," then that would solve the problems in a way that benefits everyone.

History shows that it's a bad bet to try to prevent developers from doing the most flexible thing -- see Apple's early attempts to ban dynamic languages from the app store, for example. Preventing a site from saying "Require this module" seems in a similar vein.

As a closing point, exokit currently exposes dgram to user sites. But this is a symptom of a more general situation: the dgram library is sufficiently useful that people want to incorporate it into their sites. Ditto for PhysX, OpenVR, Magic Leap, and many others.

[avaer]

I think you hit the nail on the head. Extensions are a general pain point Exokit's had as well with adding e.g. XRMultiplayer.

But there's a lot to unpack here.

Prior art

There is already a "Browser Exensions" spec for cross-browser extensions but it seems to be solving a different problem (that of reigning in UI/UA decorators i.e. what people call "extensions").

This is kind of different than that; it's basically allowing sites to decide to download extra code to the browser, kind of like a <script type=binary src="https://physx.org/physx.bin"/> that exposes a function which takes addExtension(window), with necessary auto-caching.

I guess the main problem here isn't implementing this-- that's very easy, at least for Exokit (measured in days).

Potential problems

The main problems are security and ABI management. Security and process isolation are their own cans of worms.

Node has gotten better in terms of a standard ABI in recent months but it's still something that needs consideration.

A lesser problem is speccing it out to meet all cases with consideration for e.g. event loops, contexts.

Alternate approaches

One alternative I think that's missed above is just using WASM for such things. That doesn't require a new spec, or security thought, or implementation, or coordination with anyone. That's all supported in both Node and Exokit already.

So I'd try just building PhysX with emscripten or something first. Creating JS bindings is of course going to be required anyway.

But, the WASM approach doesn't add a permanent extension to the browser; it would be sites-specific like any other script. It could still be equally cached, but the runtime feel and DX would be a little bit different. 🤔

[shawwn]

I’ve been giving security some thought.

The central problem isn’t that it’s insecure — it’s that some people are malicious, and some have nothing to gain by being malicious. Theoretically npm installing some code could act maliciously, but it almost never happens. Why is it different just because it’s the web? What if we could make it the same, somehow?

One pitfall with wasm is that it’s mostly unknown right now. For example, if I want to write a site that uses the fs module, how would I do that if I can only use wasm? I’m not saying you can’t, just that I have no idea.

Supporting npm install would also be a form of caching + updating, and in general it might solve a lot of problems. Would it make sense to just prompt the user whether they’re ok with a site installing certain packages? This needs more thought to be a realistic option though.

I’m not sure a module needs to be mounted or unmounted. dgram is exposed as dgram, and that seems fine. What’s the benefit of the added complexity?

[HeadClot]

Hey @shawwn - I am working on a project using Exokit and I really could use PhysX 3.4 and the origin shift feature found in PhysX. Ideally It would be nice to have this function exposed to Java Script.

Ideally It would be great to have it exposed as a function that I could call in code to shift the origin of world as needed.