Wrist attachment and standard controller mount points

[dantman]

No rush, but I think a VRTK_WristAttachment or a left and right wrist alias object in the SDK Manager could be a useful feature. Watches are a useful interface that we may see more of in VR; they can tell in game time / level timers / or work as a menu interface. A few games have done this so far just with a watch on a hand model. But it would be good to have a point we can simply attach something to to have it follow the spot your wrist should be in.

In general it would probably be nice to have a standard set of attachment transform points available for controllers (eg: a grab transform for the relative location your hands grab around, to lock a pistol grip or sword handle to; a barrel transform for the location a gun barrel would naturally point and you'd expect a generic gun to shoot towards; a point transform for the direction a laser pointer would feel natural with the controller, for Vive wands that's usually upwards from the wand while on Touch you normally point with an index finger) since they vary by controller.

However I think it's worth special casing the wrist transform and giving it a dedicated script. It's more of a fixed attachment point rather than a point relative to grabbing. Also having a script dedicated to the wrist leaves us room to support any changes to how the wrist location is tracked. We've already seen tracked gloves that will be coming out. So it's possible that in the future the wrist and controller could be part of separate tracked objects. Also the gloves shown off seem to track how the wrist bends, so the location of the wrist relative to the controller is variable.

[thestonefox]

The SDK Manager is really for identifying core aspects of the VR setup.

A wrist attachment point is not part of a core set up. Also it's easy enough to do by just attaching something as a child to the controller script alias (the same way the tooltips are attached to the controller alias).

As for the standard set of attachments, this is done via the "Interactable Object" snap handles and cannot be standardised because each object would potentially have a different snap point. No 2 pistols of different makes are held in the same position.

[dantman]

The SDK Manager might not be the best place, I am thinking of something higher level. Though parenting the wrist to the controller may not be the best place either.

---

I understand you can attach something wrist relative with a transform. However there are two common things / reasons I think it would be a good VRTK feature.

Firstly, it would be nice for that transform to be standardized and easy to drop in, instead of every game with a watch/wristband/etc... having to re-discover or copy and paste that transform.

Secondly, as I mentioned the wrist as a per-game hardcoded controller-relative transform is not very future-proof. There are a number of future situations where that won't work correctly.

• If the controller is a glove and the controller's origin is relative to the wrist, then the transform is [0,0,0] and a Wand/Touch relative transform will be incorrect.
• If the controller is a glove and the controller's origin is relative to the hand, then the wrist will not be a fixed transform but vary based on the angle from the forearm sensor.
• If any new controller (Valve's knuckles controller or a new accessory) includes a forearm sensor, then the wrist location definitely won't be a fixed transform.
• If a glove + controller combination is used (ie: a left hand wearing a glove accessory holds on to a controller) then the wrists end up relative to a completely different tracked object than the left/right controllers.

We already know that glove accessories will be coming out soon. We can't write code to handle them yet. But we can at least prepare by making sure that when they come out handling the wrist is a simple case of "update VRTK and tweak an attribute or two on the wrist attachment script to suit your game" instead of "refactor your game so wrist locations aren't hardcoded, parented to the wrong object, etc...".

---

On the topic of standard attachment points, I wasn't referring to the attachment points on grabbable objects. I'm referring to vectors/transforms for standard spots on the controller; which may be useful for scripts or making game elements follow. Not all of the points are grab related. And they aren't the same across platforms.

I've drawn up some diagrams to try and illustrate what I am thinking of.

• (yellow) is a grab/grip transform, ie: the center of the position on the controller your hand grabs around aligned with the axis that a straight cylindrical object would point if you were holding it.
• (cyan) is the point transform/vector, ie: the point and direction from which pointers are emitted. This differs between platforms; on the Vive wands the pointer normally emits from the end of the wands and points along the same axis as you grip (yellow), on the Touch controllers the pointer normally emits aligned to the forward vector from where your index finger should be.
• (red) is a forward vector, ie: the direction your fingers point when you stretch them out and if you were to hold a generic sci-fi gun you'd expect it to fire.
• As you can see in in the final photo the axis you grip with (yellow) is not perpendicular to the forward vector (red) of your hand, so getting one from the other is not a simple 90˚ rotation.

Looking at the OpenVR source, it appears as if some of these transforms are even standard coordinate system components available in SteamVR. However I only see VRTK making use of the tip coordinate system.
https://github.com/ValveSoftware/openvr/blob/master/headers/openvr.h#L2987-L2994
https://github.com/ValveSoftware/openvr/blob/master/headers/openvr.h#L2850-L2854

[bddckr]

I think it would be awesome for VRTK to support all these "Grab attach points on the controller". Given that the Oculus SDK probably doesn't provide something like that by default we'd have to introduce our own abstract attach points.

[dantman]

@bddckr Yeah, probably. On the positive side though, Oculus via the SDK only has one type of controller and doesn't support 3rd party controllers because every new tracked object has to be designed and approved by Oculus; so we can just hardcode the transforms in the VRTK/SDK/OculusVR code.

Even better, SteamVR may have already done some of the work. If OpenVR outputs those coordinate system components when you plug a Rift+Touch into it then all we have to do is double check the controller origin is the same in OculusVR and SteamVR, then copy and paste the transform values into the OculusVR code.

[cameronoltmann]

This strikes me as something that would be best in two parts. A generic part that belongs in the toolkit and lets users set up arbitrary mount points. And an example scene or two that make use of these to implement things like watches, swords, and wands.

I'd be wary of assigning any standard mount points for things like gun barrels and watches as part of the core toolkit. Even the most genetic seeming thing often turns out to be very app specific. Look at the forums for a lot of the shooting games and you'll see people can't agree on the optimal angle between the barrel and the handset, for example. That's why I think the best bet is to include examples rather than building them into core.

[dantman]

@cameronoltmann Being able to setup arbitrary mount points to connect things to would be a nice extra. The key thing though is that the mount points you create shouldn't be limited to being relative to the origin of the controller, they should be relative to known locations on the controller (handgrip, tip, base, forward vector, button components, ...) which may differ by controller type.

I still think the wrist use case needs more special handling than just a controller relative mount point. Especially given the whole wrists on a separate tracked object issue. However, I can compromise on wrists using mount points for now if:

• It's very easy for me to insert a handgrip relative mount point on a controller, do the same on another tracked object, and it only takes a simple line of code to make an object attached to one mount point switch to another.
• It's possible to create scripted mount points, ie: mount points where the relative transform is not fixed but can be updated by a script. So that if something comes along, like forearm tracking, it's very easy to write a script that will simply drop in on a mount point to make it follow the new tracking data.

Then the mount points are flexible enough that a "Wrist" point can be attached to a mount point; and in the future if any new controller features come out it's easy to just attach that wrist to a mount point on another controller or make the mount point itself dynamic.

This level of flexibility will probably be useful for some other type of feature a game might want to implement; though I can't immediately think of an idea off the top of my head.

[cameronoltmann]

The example scenes would include things like tip/grip/wrist for different controllers. The core toolkit would just need the raw functionality to enable those scenes, which is to say, the ability to define mount points with arbitrary offsets from the controller tip. With that functionality, it's trivial to create mount points that match any position or move in any way you want. And since most of those ways will be application specific, I don't think they belong in the core toolkit, but in the examples.

I also think you're unnecessarily complicating the wrist issue. If the wrist is a separate tracked object, it's really got nothing to do with the main controller, and thus nothing to do with the core toolkit. Additionally, since there are no standards beyond headset plus two controllers at this point, there's probably not much point in trying to architect the toolkit for things that don't exist.

[dantman]

@cameronoltmann tip/handgrip/base are not simple offsets, you can't implement them that way; they are different for each type of controller. If you create a transform for a 'tip' based on the location of the tip on a Vive wand, then the tip will not be at the tip when you use Oculus Touch or Valve's knuckles controllers. And OpenVR understands this concept, defining separate coordinate systems for controllers (see https://github.com/ValveSoftware/openvr/blob/master/headers/openvr.h#L2991).

[cameronoltmann]

They are simple offsets. They just happen to be different for each controller type. And as new controller types are released, there will continue to be new sets of offsets that will make sense to have. And if you have example scenes for the different controller types, people will have good reference material to use when adapting to a new controller.

The reason I don't think it makes sense to bake these points directly into the toolkit, but rather to have example scenes, can be summed up by this question:

Which of the "standard" offset vectors is appropriate on all controller types to line a bow up with?

On the Vive, this is generally along the shaft of the controller. On the Oculus Touch controllers, none of the standard vectors is appropriate. And new controllers will introduce new configurations we haven't thought of yet. So baking in a predefined set of points will only serve to artificially limit a medium that is still made of wide open space and room for exploration. It would be a mistake, in my opinion.

[dantman]

@cameronoltmann Please, please, look at the OpenVR code I am linking to. tip/handgrip/base are not offsets you hardcode, in SteamVR they are offsets you ask OpenVR for the location of. When the controller is a Vive wand it'll give you one value, when the controller is a Touch it'll give you a different value, and when the knuckles controllers are released you'll get another value without needing to update your game. (On that topic, games should not need to be updated every time a new 3rd party controller with the standard control-set comes out, that is the antithesis of how OpenVR is supposed to work and results in a piles of old games that'll never get updated not working correctly with new controllers).

---

On the topic of bows there are two answers to that:

• A) If your game is designed with tools where all interaction is done with an "interaction sphere" at the tip, then the mount point you attach the bow and bow string to is relative to the tip. (From the looks of the use of "tip" in the SDK code, this is VRTK's current behaviour), otherwise
• B.1) The mount point you attach the bow to will be relative to handgrip, so that the location you hold a bow with will be placed relative to where your hand is physically grabbing.
• B.2) Instead of the handgrip where your full hand is, the mount point the bow string attaches to can be relative to the location of the trigger, then the bow string is held near your index finger which you normally grab a bowstring with.

And the answer to "What 'standard' offset vectors would VRTK let you create mount points relative to?" would be "Every one at least one supported SDK provides". If a transform is not available through SteamVR for a controller, it'll fall back to the origin or perhaps the closest known point. If the Oculus SDK doesn't provide an offset, we'll hardcode the one SteamVR outputs for the Touch controllers (though from the look of the OculusVR folder code, most of these points may already be provided by the SDK).

The offsets currently supplied by SteamVR are:

• origin: Not entirely defined where this is
• tip: An unambiguous tip, used for laser pointing
• base: For controllers with a defined base to it
• handgrip: A neutral ambidextrous hand pose, on a plane relative to the neutrally posed thumb and index finger.
• status: OpenVR calls this a "1:1 aspect ratio status area, with canonical [0,1] uv mapping", I'm not entirely certain what it actually maps to, but I'm guessing that this might be the where SteamVR displays things like the battery level indicator in the dashboard.
• all controller components: All the controller components SteamVR knows about (trigger, grip buttons, touchpad/joystick, application button, system button, etc) also have a defined component from which you can ask OpenVR where their location is.

[cameronoltmann]

Yeah, I see the set of predefined points. I see those as sort of a lowest common denominator.

Can you define mount criteria (position and orientation) that will work for all controllers for bow orientation? Sure, it makes sense for the body of the bow to go in the middle of your hand. But which way does it face?

[dantman]

@cameronoltmann The coordinate information given by OpenVR should be a matrix, which the Unity plugin should be turning into a transform. So each coordinate system/component position OpenVR exposes also comes with an orientation.

I can't examine the SteamVR's output right now, so exactly what those orientations are is just theory at the moment:

• tip: Presumably since the tip is intended for laser pointing the transform will be oriented so that whatever axis OpenVR considers 'forward' is pointing in the direction a laser/ray should be cast.
• base: Presumably since this is for controllers with some sort of base the orientation would be something aligned with the flat plane of the base, so you know how it naturally rests on objects.
• status: Since this seems intended for a canvas to attach to, this is probably oriented the way a canvas should be oriented
• handgrip: It's hard to say which axis the handgrip will be oriented to, however given the statement "plane relative to the neutrally posed thumb and index finger" the orientation is probably relative to the forward direction of your hand rather than the axis you grip on (look at my hand diagram to see the difference).

For a bow, since OpenVR states that handgrip is relative to the plane where the index and thumb are placed: One option would be for the game developer to create a transformed mount point that is offset from the index/thumb so it instead sits in the center of your hand and set the origin/snap on their bows in the center point of the handle and face them forward.

Alternatively if you wanted to work more with cylindrical objects like sword grips. You could instead define a mount point relative to the handgrip but rotate it a bit so it roughly fits your grip axis instead. Then you could define the snap point on your objects so they are oriented relative to whatever semi-cylindrical axis you grab the object with.

[cameronoltmann]

My point is that you'll need to offset the built in points by a different amount on different controllers, regardless of what you use as a base. Hence the utility of having more than the built in points as examples. Because different controllers result in different grip poses, which result in different mounting practices being appropriate for some actions. The will be no one-size-fits-all solution.

[dantman]

@cameronoltmann If you had to vary your offsets on different controllers that would defeat the point of OpenVR having these points in the first place.

The handgrip is defined by OpenVR as being a "neutral hand pose, ambidextrous, between the index and thumb". Each controller driver defines where this is on the controller. And then OpenVR tells us where this is.

So it doesn't matter what controller you use. Whether you use the Vive wands, Oculus Touch, Valve's knuckles controller, or a 3rd party controller that implements a SteamVR driver; the handgrip component will always tell you where the plane the index finger+thumb sits on and how it is oriented.

From there the transform from your index+thumb plane to the center of your hand is the same, it doesn't matter what controller you're holding.

Different examples for mounting strategies would be good, but those mount points you create are still the same transform whatever controller you use.