Skip to content

Commit a760863

Browse files
authored
Add support for hand tracking microgestures (#17255)
This pull request adds the support for hand tracking microgestures, a Meta OpenXR extension - compatible with Quest 2/Pro/3/3S - that adds five boolean inputs per hand (thumb tap + D-pad–like swipes). `WebXROculusHandController` has been created starting from `WebXRGenericHandController` as base and mapping the input profile with the`oculus-hand` profile: https://github.com/immersive-web/webxr-input-profiles/pull/274/files. ___ ### Behaviour As today, all the input controller profiles are retrieved from the WebXR controller [repository](https://immersive-web.github.io/webxr-input-profiles/packages/viewer/dist), unfortunately, the `oculus-hand` controller is missing. I made the PR immersive-web/webxr-input-profiles#278 to add it, but i don't know if/when it will be approved. **As-is** 1. Starting a WebXR session with a compatible Quest device, `xrController.inputSource.profiles` is correctly populated with `oculus-hand` profile: ```ts ['oculus-hand', 'generic-hand', 'generic-hand-select', 'generic-trigger'] ``` 3. All input controller profiles are retrieved from: https://immersive-web.github.io/webxr-input-profiles/packages/viewer/dist/profiles/profilesList.json (but `oclus-hand` is missing). 4. Get `generic-hand` because it is the first available profile: https://immersive-web.github.io/webxr-input-profiles/packages/viewer/dist/profiles/generic-hand/profile.json 5. Map `generic-hand`. **To-be (after pull request is merged)** 1. The user create an XR experience disabling the online controller repository (see "Considerations" below): ```ts scene.createDefaultXRExperienceAsync({ inputOptions: { disableOnlineControllerRepository: true } }) ``` 2. Starting a WebXR session with a compatible Quest device, `xrController.inputSource.profiles` is correctly populated with `oculus-hand` profile: ```ts ['oculus-hand', 'generic-hand', 'generic-hand-select', 'generic-trigger'] ``` 3. The `oculus-hand` profile is retrieved from Babylon.js itself (local). ___ ### Note Forcing the input profile: ```ts scene.createDefaultXRExperienceAsync({ inputOptions: { forceInputProfile: 'oculus-hand' } }) ``` is not a valid option because in any case the `generic-hand` profile will be retrieved. ___ ### Test Tested with Oculus Quest 3. Playground (valid after PR build): https://playground.babylonjs.com/#F41V6N#2275. ___ ### Considerations When the `oculus-hand` profile will be added to online repository, it shouldn't introduce regressions because the `oculus-hand` it will be retrieved and consumed automatically. The `oculus-hand` profile is 1:1 mapping with `generic-hand` profile with the adding of specific swipe components (`menu` - only left hand, `swipe-left`, `swipe-right`, `swipe-forward`, `swipe-backward`, `tap-thumb`). ___ ### Discussion Forum link: https://forum.babylonjs.com/t/hand-tracking-microgestures/60860.
1 parent 1cba3e7 commit a760863

3 files changed

Lines changed: 260 additions & 0 deletions

File tree

packages/dev/core/src/XR/motionController/index.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,5 +5,6 @@ export * from "./webXRGenericMotionController";
55
export * from "./webXRMicrosoftMixedRealityController";
66
export * from "./webXRMotionControllerManager";
77
export * from "./webXROculusTouchMotionController";
8+
export * from "./webXROculusHandController";
89
export * from "./webXRHTCViveMotionController";
910
export * from "./webXRProfiledMotionController";

packages/dev/core/src/XR/motionController/webXRMotionControllerManager.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,7 @@ export class WebXRMotionControllerManager {
8282
this.RegisterFallbacksForProfileId("samsung-odyssey", ["generic-touchpad"]);
8383
this.RegisterFallbacksForProfileId("valve-index", ["generic-trigger-squeeze-touchpad-thumbstick"]);
8484
this.RegisterFallbacksForProfileId("generic-hand-select", ["generic-trigger"]);
85+
this.RegisterFallbacksForProfileId("oculus-hand", ["generic-trigger"]);
8586
}
8687

8788
/**
Lines changed: 258 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,258 @@
1+
/* eslint-disable @typescript-eslint/no-unused-vars */
2+
import type { IMotionControllerLayoutMap, IMinimalMotionControllerObject, MotionControllerHandedness } from "./webXRAbstractMotionController";
3+
import { WebXRAbstractMotionController } from "./webXRAbstractMotionController";
4+
import type { Scene } from "../../scene";
5+
import type { AbstractMesh } from "../../Meshes/abstractMesh";
6+
import { WebXRMotionControllerManager } from "./webXRMotionControllerManager";
7+
8+
/**
9+
* Oculus hand controller class that supports microgestures
10+
*/
11+
export class WebXROculusHandController extends WebXRAbstractMotionController {
12+
public profileId = "oculus-hand";
13+
14+
/**
15+
* Create a new hand controller object, without loading a controller model
16+
* @param scene the scene to use to create this controller
17+
* @param gamepadObject the corresponding gamepad object
18+
* @param handedness the handedness of the controller
19+
*/
20+
constructor(scene: Scene, gamepadObject: IMinimalMotionControllerObject, handedness: MotionControllerHandedness) {
21+
// Don't load the controller model - for now, hands have no real model.
22+
super(scene, OculusHandProfile[handedness], gamepadObject, handedness, true);
23+
}
24+
25+
protected _getFilenameAndPath(): { filename: string; path: string } {
26+
return {
27+
filename: "generic.babylon",
28+
path: "https://controllers.babylonjs.com/generic/",
29+
};
30+
}
31+
32+
protected _getModelLoadingConstraints(): boolean {
33+
return true;
34+
}
35+
36+
protected _processLoadedModel(_meshes: AbstractMesh[]): void {
37+
// no-op
38+
}
39+
40+
protected _setRootMesh(meshes: AbstractMesh[]): void {
41+
// no-op
42+
}
43+
44+
protected _updateModel(): void {
45+
// no-op
46+
}
47+
}
48+
49+
// register the profiles
50+
WebXRMotionControllerManager.RegisterController("oculus-hand", (xrInput: XRInputSource, scene: Scene) => {
51+
return new WebXROculusHandController(scene, <any>xrInput.gamepad, xrInput.handedness);
52+
});
53+
54+
// https://github.com/immersive-web/webxr-input-profiles/blob/main/packages/registry/profiles/oculus/oculus-hand.json
55+
const OculusHandProfile: IMotionControllerLayoutMap = {
56+
left: {
57+
selectComponentId: "xr-standard-trigger",
58+
components: {
59+
// eslint-disable-next-line @typescript-eslint/naming-convention
60+
"xr-standard-trigger": {
61+
type: "trigger",
62+
gamepadIndices: {
63+
button: 0,
64+
},
65+
rootNodeName: "xr-standard-trigger",
66+
visualResponses: {},
67+
},
68+
menu: {
69+
type: "button",
70+
gamepadIndices: {
71+
button: 4,
72+
},
73+
rootNodeName: "menu",
74+
visualResponses: {},
75+
},
76+
// eslint-disable-next-line @typescript-eslint/naming-convention
77+
"swipe-left": {
78+
type: "button",
79+
gamepadIndices: {
80+
button: 5,
81+
},
82+
rootNodeName: "swipe-left",
83+
visualResponses: {},
84+
},
85+
// eslint-disable-next-line @typescript-eslint/naming-convention
86+
"swipe-right": {
87+
type: "button",
88+
gamepadIndices: {
89+
button: 6,
90+
},
91+
rootNodeName: "swipe-right",
92+
visualResponses: {},
93+
},
94+
// eslint-disable-next-line @typescript-eslint/naming-convention
95+
"swipe-forward": {
96+
type: "button",
97+
gamepadIndices: {
98+
button: 7,
99+
},
100+
rootNodeName: "swipe-forward",
101+
visualResponses: {},
102+
},
103+
// eslint-disable-next-line @typescript-eslint/naming-convention
104+
"swipe-backward": {
105+
type: "button",
106+
gamepadIndices: {
107+
button: 8,
108+
},
109+
rootNodeName: "swipe-backward",
110+
visualResponses: {},
111+
},
112+
// eslint-disable-next-line @typescript-eslint/naming-convention
113+
"tap-thumb": {
114+
type: "button",
115+
gamepadIndices: {
116+
button: 9,
117+
},
118+
rootNodeName: "tap-thumb",
119+
visualResponses: {},
120+
},
121+
},
122+
gamepadMapping: "xr-standard",
123+
rootNodeName: "oculus-hand-left",
124+
assetPath: "left.glb",
125+
},
126+
right: {
127+
selectComponentId: "xr-standard-trigger",
128+
components: {
129+
// eslint-disable-next-line @typescript-eslint/naming-convention
130+
"xr-standard-trigger": {
131+
type: "trigger",
132+
gamepadIndices: {
133+
button: 0,
134+
},
135+
rootNodeName: "xr-standard-trigger",
136+
visualResponses: {},
137+
},
138+
// eslint-disable-next-line @typescript-eslint/naming-convention
139+
"swipe-left": {
140+
type: "button",
141+
gamepadIndices: {
142+
button: 5,
143+
},
144+
rootNodeName: "swipe-left",
145+
visualResponses: {},
146+
},
147+
// eslint-disable-next-line @typescript-eslint/naming-convention
148+
"swipe-right": {
149+
type: "button",
150+
gamepadIndices: {
151+
button: 6,
152+
},
153+
rootNodeName: "swipe-right",
154+
visualResponses: {},
155+
},
156+
// eslint-disable-next-line @typescript-eslint/naming-convention
157+
"swipe-forward": {
158+
type: "button",
159+
gamepadIndices: {
160+
button: 7,
161+
},
162+
rootNodeName: "swipe-forward",
163+
visualResponses: {},
164+
},
165+
// eslint-disable-next-line @typescript-eslint/naming-convention
166+
"swipe-backward": {
167+
type: "button",
168+
gamepadIndices: {
169+
button: 8,
170+
},
171+
rootNodeName: "swipe-backward",
172+
visualResponses: {},
173+
},
174+
// eslint-disable-next-line @typescript-eslint/naming-convention
175+
"tap-thumb": {
176+
type: "button",
177+
gamepadIndices: {
178+
button: 9,
179+
},
180+
rootNodeName: "tap-thumb",
181+
visualResponses: {},
182+
},
183+
},
184+
gamepadMapping: "xr-standard",
185+
rootNodeName: "oculus-hand-right",
186+
assetPath: "right.glb",
187+
},
188+
none: {
189+
selectComponentId: "xr-standard-trigger",
190+
components: {
191+
// eslint-disable-next-line @typescript-eslint/naming-convention
192+
"xr-standard-trigger": {
193+
type: "trigger",
194+
gamepadIndices: {
195+
button: 0,
196+
},
197+
rootNodeName: "xr-standard-trigger",
198+
visualResponses: {},
199+
},
200+
menu: {
201+
type: "button",
202+
gamepadIndices: {
203+
button: 4,
204+
},
205+
rootNodeName: "menu",
206+
visualResponses: {},
207+
},
208+
// eslint-disable-next-line @typescript-eslint/naming-convention
209+
"swipe-left": {
210+
type: "button",
211+
gamepadIndices: {
212+
button: 5,
213+
},
214+
rootNodeName: "swipe-left",
215+
visualResponses: {},
216+
},
217+
// eslint-disable-next-line @typescript-eslint/naming-convention
218+
"swipe-right": {
219+
type: "button",
220+
gamepadIndices: {
221+
button: 6,
222+
},
223+
rootNodeName: "swipe-right",
224+
visualResponses: {},
225+
},
226+
// eslint-disable-next-line @typescript-eslint/naming-convention
227+
"swipe-forward": {
228+
type: "button",
229+
gamepadIndices: {
230+
button: 7,
231+
},
232+
rootNodeName: "swipe-forward",
233+
visualResponses: {},
234+
},
235+
// eslint-disable-next-line @typescript-eslint/naming-convention
236+
"swipe-backward": {
237+
type: "button",
238+
gamepadIndices: {
239+
button: 8,
240+
},
241+
rootNodeName: "swipe-backward",
242+
visualResponses: {},
243+
},
244+
// eslint-disable-next-line @typescript-eslint/naming-convention
245+
"tap-thumb": {
246+
type: "button",
247+
gamepadIndices: {
248+
button: 9,
249+
},
250+
rootNodeName: "tap-thumb",
251+
visualResponses: {},
252+
},
253+
},
254+
gamepadMapping: "xr-standard",
255+
rootNodeName: "oculus-hand-none",
256+
assetPath: "none.glb",
257+
},
258+
};

0 commit comments

Comments
 (0)