Detached controllers implementation #2818

yurec0977 · 2025-05-05T12:29:13Z

We have a VR project — a machine gun simulator — where we use a real physical model of a machine gun. Our goal is to track where the user is aiming with the real machine gun and transfer that data into the virtual world. At the same time, we also need to track the user’s hands so that the player can see their virtual hands and accurately grab the machine gun handles.

To achieve this in Unity3D, we had to treat the VR controllers as trackers, and then extract tracking data from them in Unity accordingly.

It is clear pull request instead previous one #2794 which has some master merge issues

This gives possibility to track real world object and hands in the same time. As example to track aiming of stationary placed real world gun by attached controller to it. And in the same time still have hands skeletones in game.

Use controllers as any fake tracker on server side

zmerp · 2025-05-06T02:07:05Z

You can still fix the other PR, you just need to rebase your branch. Just run git rebase master making sure you have your branch checked out and you fetched the latest changed from upstream. If you want to continue on this PR please copy the info from the top post of the previous PR into this one.

zmerp · 2025-05-06T02:08:01Z

You still need to fix the formatting and some lints so please follow up on that

yurec0977 · 2025-05-06T07:59:32Z

Is there any way to see what exactly is wrong in cpp files? For rust scripts console gives me nice instructions for problems? but for cpp files just fail without instructions(((

P.S. I've found some formatting issues in my changes, but seems there still a lot of issues not related to my changes : Some mappings in Path.cpp looks not pass format check.
Creating some body providers are not ok.

If you want I could fix some of that problem, but not sure if it is good idea to do this fixes in not PR related places

zmerp · 2025-05-07T11:28:18Z

Please run cargo xtask format. This is our custom command to automatically format the whole codebase, both Rust and c++. Sorry I haven't mentioned it before; we should say this in the PR template

yurec0977 · 2025-05-07T11:44:12Z

So strange I've done cargo xtask format. And localy cargo xtask check-format is success. But here it failed

zmerp · 2025-05-07T11:54:00Z

xtask format uses clang-format internally. Maybe it's not finding the .clang-format file. are you running the command from the repository root?

yurec0977 · 2025-05-07T12:11:06Z

Yes I used it from the root. And file which failed here was modified by format command. And as I mentioned localy "check-format" success after format command

zmerp · 2025-05-08T01:29:15Z

Look in the source code, run the clang format command yourself checking that it picks up the config file

yurec0977 · 2025-05-08T08:58:11Z

Could you please double-check locally what's going wrong? I'm having trouble understanding the issue on my side.

Running cargo xtask format formats the files as expected, so it seems like the config file is being picked up correctly.
Also, cargo xtask check-format passes successfully after formatting — which makes me think everything is working as intended.

zmerp · 2025-05-08T09:25:20Z

Passes the check for me after I ran cargo xtask format. Your system is configured wrong somehow

zmerp · 2025-05-08T09:32:55Z

@yurec0977 sorry to tell you this so late but I remembered I already had a detached controllers branch, "poll-detached". I can work on integrating that into this PR

zmerp · 2025-05-09T04:31:30Z

@yurec0977 Could you help checking that this PR still works after my changes?

yurec0977 · 2025-05-09T09:01:52Z

Of course, I'll check it today.

alvr/session/src/settings.rs

The-personified-devil · 2025-05-09T09:35:05Z

alvr/server_openvr/src/lib.rs

+                                if let Some(motion) = &ffi_left_detached_controller_motion {
+                                    motion
+                                } else {
+                                    ptr::null()
+                                },
+                                if let Some(motion) = &ffi_right_detached_controller_motion {
+                                    motion
+                                } else {
+                                    ptr::null()


This seems like it can just be a .unwrap_or()

I am not Rust or c++ expert. Just take as example set for controllerMotion and handSkeleton in hand data.

alvr/session/src/settings.rs

The-personified-devil · 2025-05-09T09:38:37Z

alvr/session/src/lib.rs

+    pub use_left_controller_as_fake_tracker: bool,
+    pub use_right_controller_as_fake_tracker: bool,
+    pub left_controller_as_fake_tracker_binding: i32,
+    pub right_controller_as_fake_tracker_binding: i32,


4 fields for this seems like a lot, maybe the bools can be merged as essentially being an off enum option, tho idk if that's actually a good idea

Sounds nice. Removed bool variables

The-personified-devil · 2025-05-09T09:41:24Z

alvr/server_openvr/cpp/alvr_server/Paths.cpp

+
+uint64_t GetFakeTrackerBindingID(int settingsBinding) {
+    switch (settingsBinding) {
+    case 0:
+        return BODY_CHEST_ID;
+    case 1:
+        return BODY_HIPS_ID;
+    case 2:
+        return BODY_LEFT_ELBOW_ID;
+    case 3:
+        return BODY_RIGHT_ELBOW_ID;
+    case 4:
+        return BODY_LEFT_KNEE_ID;
+    case 5:
+        return BODY_LEFT_FOOT_ID;
+    case 6:
+        return BODY_RIGHT_KNEE_ID;
+    case 7:
+        return BODY_RIGHT_FOOT_ID;
+
+    default:
+        return (uint64_t)0;
+    }
+}


This should really be handled by setting the actual enum values in rust and c++ and then just using a cast

As I mentioned earlier I am not Rust or C++ expert, so have no idea, how to do it. Could you please tell me if there is any example in project?

So I made the mistake of forgetting that those aren't actually enums but rather ids so what I suggested was indeed bogus, sorry for the confusion. But what I'd prefer is to just do the conversion on the rust side since the ids are also accessible there and directly put the id into the openvr config.

@zmerp since these are hashes is that fine? The openvr config is always regenerated so it shouldn't be an issue but idk.

Tho that would also require unmerging the bools and ids, but that doesn't matter, the openvr config struct is already such a mess that it doesn't really matter.

If we have hash IDs then there shouldn't be the need to convert to ordered numbers. @yurec0977 send back the ID through openvr_config if necessary, this function is not needed

Seems that config doesn't support uint64_t. When I rework it to use hash IDs and try to get it from config, build streamer failed.
m_rightControllerAsFakeTrackerBinding
= config.get("right_controller_as_fake_tracker_binding").get<uint64_t>();

@zmerp Could you help me with it? Meybe there is some example how to do it?

alvr/server_openvr/cpp/alvr_server/alvr_server.cpp

yurec0977 · 2025-05-09T10:03:18Z

@yurec0977 Could you help checking that this PR still works after my changes?

@zmerp It works as expected.

alvr/session/src/settings.rs

The-personified-devil · 2025-05-11T16:01:16Z

alvr/server_openvr/cpp/alvr_server/Paths.cpp

+
+uint64_t GetFakeTrackerBindingID(int settingsBinding) {
+    switch (settingsBinding) {
+    case 0:
+        return BODY_CHEST_ID;
+    case 1:
+        return BODY_HIPS_ID;
+    case 2:
+        return BODY_LEFT_ELBOW_ID;
+    case 3:
+        return BODY_RIGHT_ELBOW_ID;
+    case 4:
+        return BODY_LEFT_KNEE_ID;
+    case 5:
+        return BODY_LEFT_FOOT_ID;
+    case 6:
+        return BODY_RIGHT_KNEE_ID;
+    case 7:
+        return BODY_RIGHT_FOOT_ID;
+
+    default:
+        return (uint64_t)0;
+    }
+}


So I made the mistake of forgetting that those aren't actually enums but rather ids so what I suggested was indeed bogus, sorry for the confusion. But what I'd prefer is to just do the conversion on the rust side since the ids are also accessible there and directly put the id into the openvr config.

@zmerp since these are hashes is that fine? The openvr config is always regenerated so it shouldn't be an issue but idk.

Tho that would also require unmerging the bools and ids, but that doesn't matter, the openvr config struct is already such a mess that it doesn't really matter.

alvr/server_openvr/cpp/alvr_server/alvr_server.cpp

Yurii Seredovych added 2 commits May 5, 2025 15:15

Implemented possibility to use controller as fake tracker

dc06376

This gives possibility to track real world object and hands in the same time. As example to track aiming of stationary placed real world gun by attached controller to it. And in the same time still have hands skeletones in game.

implementation for detached controllers from client side in proper way

e80717b

Use controllers as any fake tracker on server side

yurec0977 mentioned this pull request May 5, 2025

WIP: Use VR controllers as trackers while preserving accurate hand tracking #2794

Draft

Fixed formatting issues

e98c1f2

Removed unused import

b674427

Formatting

604981f

zmerp force-pushed the Detached-controllers-implementation branch from d0b307f to 604981f Compare May 8, 2025 09:24

Refactoring from poll-detached branch

955bcca

zmerp force-pushed the Detached-controllers-implementation branch from 8351431 to 955bcca Compare May 9, 2025 04:34

The-personified-devil requested changes May 9, 2025

View reviewed changes

After review fixes

8928e68

The-personified-devil reviewed May 11, 2025

View reviewed changes

small tuning

8785ac7

Uh oh!

Detached controllers implementation #2818

Are you sure you want to change the base?

Detached controllers implementation #2818

Uh oh!

Conversation

yurec0977 commented May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zmerp commented May 6, 2025

Uh oh!

zmerp commented May 6, 2025

Uh oh!

yurec0977 commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zmerp commented May 7, 2025

Uh oh!

yurec0977 commented May 7, 2025

Uh oh!

zmerp commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yurec0977 commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zmerp commented May 8, 2025

Uh oh!

yurec0977 commented May 8, 2025

Uh oh!

zmerp commented May 8, 2025

Uh oh!

zmerp commented May 8, 2025

Uh oh!

zmerp commented May 9, 2025

Uh oh!

yurec0977 commented May 9, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yurec0977 commented May 9, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

yurec0977 commented May 5, 2025 •

edited

Loading

yurec0977 commented May 6, 2025 •

edited

Loading

zmerp commented May 7, 2025 •

edited

Loading

yurec0977 commented May 7, 2025 •

edited

Loading