Add iOS virtual controller support

[kisg]

Adds iOS virtual controller support to Godot.

The virtual controller may be configured in the project settings.

Apple reference: https://developer.apple.com/documentation/gamecontroller/gcvirtualcontroller?language=objc

Developed by Migeran, sponsored by Xibbon Inc.

[kisg]

@bruvzg Thank you for your review. We have another idea that could work better:

• We would add a new VirtualController abstract class (under core/input/), and the Input singleton would have a single get_virtual_controller() method that can be used to retrieve it. If it returns null, then it is not supported.
• During startup the platform code can set a platform-specific implementation (IOSVirtualController on iOS for now), that will implement the same API.

What do you think?

[bruvzg]

What do you think?

That's probably OK as well.

[kisg]

We implemented the proposed changes, please take a look.

[bruvzg]

Tested on iPad, seems to be working fine.

[akien-mga]

Since there's now an OS abstraction layer for virtual controllers, it would be good to assess what's the plan for other platforms. Is iOS the only one where this is supported at OS level, or can this be supported on other devices too (Android, Web on mobile, touchscreen laptop)?

[bruvzg]

Is iOS the only one where this is supported at OS level, or can this be supported on other devices too (Android, Web on mobile, touchscreen laptop)?

I do not think there are direct equivalents on other platforms, but we probably can add some generic software implementation of touch controller overlay.

[hpvb]

For a more useful API to implement on other platforms I'd prefer we follow the API described by the Apple documentation, with a way to enable elements by enum, and allow users to use all of the sticks, buttons, and triggers defined by the API.

I think then the only additional thing we might want is a way to query the virtual joystick implementation for what elements it support. Or we could just limit it to whatever Apple has, but that might change in the future, so I think a way to query the capabilities would be useful.

We can then later also add support for setting the position of elements, but that could be done as an addition to this PR later. I feel that changing the hardcoded button/stick name methods later would be harder to do in a compatible fashion.

I think then a way for the user to retrieve an array of elements, and a way to enable/disable elements by the enum would probably make this API way more future proof, and I think would be a pretty small change?

[akien-mga]

I agree with @hpvb's feedback, and generally I'd really like us to implement this in a cross-platform way. As @bruvzg pointed out only iOS seems to provide this natively, so it's fine that it's the only native implementation for the class. But we should be able to implement an engine API for the other platforms.

That's not necessarily something to be done in this PR or by this contributor. But before committing to an API we'd want to preserve compatibility with, we should do a bit of research on what would be the API we want to expose for such a cross-platform feature taking inspiration from Apple's native one. There's also a bunch of plugins users made over the year for virtual controllers for mobile input, which would be good to review for further inspiration.

So that means I was a bit too quick to set this on the 4.4 milestone last week. We're entering feature freeze and the beta phase for 4.4 today/tomorrow, and given the above I'd prefer we keep this for 4.5 and workshop the API a bit more (we can do that in the meantime so this gets merged early on for the 4.5 dev cycle).

I put this on the 4.5 merge queue so we keep track of it.

[dsnopek]

To me this kind of feels like something that should be done as a GDExtension.

That would allow you to either (1) only support iOS, and folks only download the extension if they are doing iOS, or (2) be more "experimental" with the generic implementation for other (non-iOS) platforms.

Particularly for nr 2: given that on-screen gamepads can be done in so many different ways, and they are sometimes tuned to the specific art style or gameplay of the app, it seems like it'd be tricky to agree on a generic (non-iOS) implementation for upstream Godot that everyone would be happy with.

[hpvb]

Particularly for nr 2: given that on-screen gamepads can be done in so many different ways, and they are sometimes tuned to the specific art style or gameplay of the app, it seems like it'd be tricky to agree on a generic (non-iOS) implementation for upstream Godot that everyone would be happy with.

I think a "generic" implementation that largely mimicks Apple's would be useful. If only for testing/writing cross-platform (mobile) games. I think it would make the lives of developers a lot easier to have something "in the box" so to speak that works the same same across all mobile platforms.

Any developer that wants to do their own already has those tools, apparently Apple felt it was important enough to add to their platform, and I can see why! I think there's a really good case to have this as an engine feature, and one that will "just work" for all touch platforms. It'd be nice if an indie with a relatively small team/budget could just port their desktop game to ios/android/mobile web by just rejigging the ux a bit and enabling a virtual gamepad.

[Nintorch]

I believe this PR might be superseded by #116716

[Repiteo]

Superseded by #116716; thanks for your contribution!

[kisg]

Superseded by #116716; thanks for your contribution!

I am not sure this is completely the case. While the other PR adds a built-in virtual controller, to me it seems that it is not the same as the built-in iOS virtual controller. We will review that PR in more detail to see if it solves all the use cases we have, but there might be valid use cases supporting the built-in iOS virtual controller.

[migueldeicaza]

I left some comments on #116716 because I think that the PR submitted and the decision to opt for that one did not address the topics raised here, but also, I find it to be a step backwards on iOS.