Native visionOS platform support

[rsanchezsaez]

Dear Godot community,

I'm on Apple's visionOS engineering team, and we would like to contribute Vision Pro support to the Godot engine. This is the first PR that lays the foundation for that.

First, I'd like to mention that we're really excited to be working with the Godot community on adding visionOS support. We've attempted to follow Godot's coding standards and a high-quality bar for our contributions. We hope that our contributions align with Godot's goals. Lastly, even though we have tried to split the changes into smaller self-contained PRs, we acknowledge that some of these PRs can be of considerable size.

We're very happy to iterate on our PRs after receiving feedback and suggestions from the community.

High Level Overview

The immediate goals of our contributions are:

1. To support current Godot games running natively on a planar window on visionOS.
2. To support creating Immersive experiences by using a new Godot's visionOS VR Plugin.

To achieve that, and in order to make reviewing easier, we have structured our contributions in three incremental PRs.

1. Add the native visionOS platform. Uses iOS as the starting point. Reuses as much code as possible between the iOS and Vision Pro platforms. (This PR).
2. Add ability to compile and link Swift files within Godot, and replace main.mm on the visionOS platform by the SwiftUI app lifecycle. This is a requirement to be able to launch Immersive scenes on visionOS.
3. Introduce Vision Pro VR plugin for Immersive support.

Even though we have a working version including points 2 and 3, those PRs are not up yet. Our current plan is to open them sequentially, after each of the previous PRs merge.

Technical Discussion

This PR implements a new native visionOS platform.

It's very close to the iOS platform in terms of implementation. In order to reuse as much code as possible, it introduces a new drivers/apple_embedded folder, to host code shared between the iOS and visionOS (but not macOS) platforms. We took inspiration from the new drivers/apple folder, which hosts code that applies to all Apple platforms.

The platform-specific logic (including app instantiation, client code, display server, os support, and export plugin) was refactored, and now the bulk of the implementation is on drivers/apple_embedded. The platforms provide small subclasses that specialize the concrete aspects that are different between platforms. We did this refactor with care, trying not to alter the existing iOS functionality.

The visionOS platform doesn't have OpenGL support, as it's not supported by visionOS.

In order to make reviewing easier, we have tried to split this PR into individual self-contained commits when that made sense, and we have added detailed descriptions to most of them about what's contained in each commit. It's easier to review commit by commit, to see how the changes were incrementally implemented.

Documentation Considerations

Now, the export plugin for iOS and visionOS share the majority of the code and most of their options (with the exception of launch storyboard support, which is iOS only; and specific platform icon support). Because of this, we have renamed EditorExportPlatformIOS.xml to EditorExportPlatformAppleEmbedded.xml, and moved it to drivers/apple_embedded.

We'd like to ask the community if this is appropriate from the docs tooling perspective, and we're requesting further guidance on how to modify this file or the docs tooling to provide specific documentation for each of the platforms.

Testing

We have been testing this PR mainly with the Platformer demo project. We have verified the project continues to work on iOS, and it now runs natively on visionOS. We have tested both the Mobile and Forward+ renderers with the Metal rendering driver on both platforms.

Open Questions

In all our tests, we exported an Xcode project using the corresponding export plugin and template, and we then ran this project directly to an iOS or visionOS device.

We'd like to ask the community to provide guidance, or to help testing the following functionality:

• We have not tested the ability of the iOS/visionOS export templates to embed and link plugins at export time. We tried to preserve this functionality on the visionOS platform, and we assume it will work in the same way it works on iOS. If anybody can attach a project using this functionality, we can help test it.
• We have not been able to make direct Archive/IPA export, nor One-Click-Deploy to work, not even in master. There are some differences between our developer account and what external developers use, so it's a bit difficult to pinpoint the problem. We'd appreciate it if anybody that has One-Click-Deploy currently working is able to test on both platforms.
• Likewise, we have not tested deploying directly to an iOS device using ios_deploy (used with Xcode versions prior to 14.0). If somebody has a working setup using this, we'd appreciate it if you could test this functionality still works. Alternatively, if ios_deploy support is no longer desired due to its age, we're happy to remove those codepaths for code simplicity.

Missing Functionality

• The DPI metrics on visionOS are hardcoded for now. They can change at runtime depending on how close the window is to the viewer. We'll address this in the PR adding SwiftUI, as the metrics come from SwiftUI APIs.
• We have not implemented building a visionOS Icon Asset Catalog into the exporter. If somebody from the community can step in and implement it, that would be awesome. Otherwise, we may submit this as a later PRs. For now, you can work around this by manually creating your visionOS icon after exporting the Xcode project. Here you can read how visionOS icons work. And here's an example icon showing the Asset Catalog structure: visionOS-icon.xcassets.zip
• The SVG logo for the visionOS platform is just text, it'd be good if somebody could come up with a nice graphic.

---

We're happy to answer any questions or address any concerns. We're looking forward to collaborating with all of you.

@BastiaanOlij @clayjohn @coppolaemilio @stuartcarnie Feel free to mention anybody who would be interested in this change.

[RPicster]

Sorry if I am missing information here, but I couldn't find a link to a proposal for this PR.

How is this maintained after merging it into core? The hardware to test it is extremely expensive and I don't know how available it is to core contributors - or if someone in the core team even has interest to maintain it.

The typical questions of a proposal should be answered e.g. why core and not an extension?

Looking from the outside (maybe the XR team just knows it all) it looks like a very big PR with a lot of new code and changes for a very niche use case that is hard to maintain 🤔
It would be great to make it more transparent and offer a perspective.

[saji8k]

@rsanchezsaez Will this Vision Pro support for Godot have eye-tracked foveated rendering in immersive experiences?

This wasn't possible before through Metal, but it seems an unreported change was made and Unity does now have some sort of eye-tracked foveated rendering support for fully immersive Metal Unity games. This can be seen in the new update to Demeo:
https://www.uploadvr.com/demeo-graphics-upgrade-foveated-rendering-on-apple-vision-pro/

[BastiaanOlij]

@RPicster I think this is the kind of work where a proposal is overkill and doesn't provide any value that isn't already provided by the description of the PR. The other points you raise are very valid however.

On the question why this isn't a plugin, I think that is self evident due to this being a separate OS. This is not going to function without changes in the core. We've had a few other people diving into Apple Vision Pro support and they've all ran into this problem.

The need for this to be added to Godot is hard to quantify. This is the sort of "build it and they will come" scenario but also, this is an emerging market where Apple has a focus mostly outside of games. The problem with that is that success stories are far less visible. There are no youtubers streaming using the latest application to millions of viewers.
So it is all hard to quantify. But from my perspective, the positive reception today on the news going public and the requests coming my way in the past, indicate there is definitely a need here and an opportunity for Godot to grow.

The maintenance question is a real one, one I don't have an answer to today, and one that I think we need to discuss in a larger group.

On the one hand, the approach taken here minimises the overhead of maintaining a new platform. Most of the implementation between iOS and VisionOS is shared, and there is a lot of tooling on the compiler side that will prevent breaking one platform as a result to changes to another. I haven't done a deep dive either but early impressions are that this is a very clean way of getting to this goal.

On the other hand, it is another platform to maintain and with the majority of Godot developers within the Apple eco system focussing on iOS, there is a realistic potential that enhancements and fixes performed for iOS, either break or are not applied to VisionOS. We can not expect any of the developers who work on iOS enhancements and fixes to test their work on VisionOS.
This means this burden falls on the Godot Foundation and that will need funding. It is a topic for further discussion for sure.

That all said, personally I am super excited by this work and believe that there is a lot of potential for us here.

[BastiaanOlij]

@rsanchezsaez Will this Vision Pro support for Godot have eye-tracked foveated rendering in immersive experiences?

This wasn't possible before through Metal, but it seems an unreported change was made and Unity does now have some sort of eye-tracked foveated rendering support for fully immersive Metal Unity games. This can be seen in the new update to Demeo: https://www.uploadvr.com/demeo-graphics-upgrade-foveated-rendering-on-apple-vision-pro/

This PR lays the foundation for having Godot run on AVP and focusses on the platform implementation and running normal Godot apps within the AVP environment.

As indicated in the introduction, supporting fully immersive XR will be submitted as a separate PR (/PRs) once this PR is stable, else we're just going to keep chasing our tail :)
I'm not in a position to confirm what will be in that implementation, I'm pretty sure foveated rendering will be corner stone to that work, and if not, we can push for it in once those PRs are in play :)

[stuartcarnie]

@RPicster

or if someone in the core team even has interest to maintain it.

As the contributor and current maintainer of the Metal backend, I can confirm I have an interest in this work. I've also receive requests from independent developers attempting to add visionOS support to Godot, so I will presume they too will be excited about this PR.

@jammus @Clancey are two people I know who have been actively working on it.

Some other projects using Godot and visionOS that I expect will benefit:

• https://godot.vision
• https://github.com/kevinw/GodotVision
• https://github.com/ManjitBedi/Hello-Godot-visionOS

[dsnopek]

First of all, thanks for starting this work! It would be really great to have VisionOS support, and getting code contributions directly from Apple engineers is very cool :-)

(Sorry for jumping in on the backwards compatibility thing right away, I just happened to notice it when initially skimming the diff)

@RPicster:

The typical questions of a proposal should be answered e.g. why core and not an extension?

This is a valid question, and one that I'm also wondering about. Adding more platforms ("platforms" here meaning directories under platform/ in Godot's code base) means more CI builds, more export templates, more steps to create releases, etc.

If it were possible have a single unified "ios" platform (again, in the Godot code base sense) that could support all iOS-derived platforms (here "platforms" meaning actual OS's, like iPadOS, tvOS, VisionOS), that would be much preferable to new Godot platforms for each.

I understand that there needs to be changes to main app life-cycle specifically for VisionOS, but if that could be done by building on top of the libgodot PR and just wrapping Godot in a new main loop, then perhaps it could be implemented as an extension? There would probably still need to be changes to the unified "ios" Godot platform that I'm imagining here for VisionOS, but hopefully they'd be minimal.

However, I don't have a very deep understanding of the platforms in question to know if that would really work

[RPicster]

Thanks for so many answers, insights and perspectives regarding my concerns! ❤

[migueldeicaza]

This is a valid question, and one that I'm also wondering about. Adding more platforms ("platforms" here meaning directories under platform/ in Godot's code base) means more CI builds, more export templates, more steps to create releases, etc.

Even if you were to use a module, the iOS binaries are not compatible across those three platforms (among other things they link different libraries, have different api surfaces, they use a different, it spiritually similar SDK). So you would still need export templates and CI to support the additional platforms, reducing the savings.

And then the downside would be imposing additional hoops for the ongoing development of these additional platforms.

[Clancey]

When you try and bundle, it will fail. The library paths are incorrect.

[spease]

The maintenance question is a real one, one I don't have an answer to today, and one that I think we need to discuss in a larger group.

I've got a Vision Pro and did a couple paying projects with it, coded for the Freespace open source game engine when I was in high school and early college, and am currently figuring out what my next gig will be. I haven't contributed to Godot before, but I might be able to help out if there's some well-scoped work.

The largest contribution I made to FS2Open was the Lua scripting system: https://github.com/scp-fs2open/fs2open.github.com/blob/85e22a48654da1f68ae754e276503ea414c78d84/code/parse/lua.cpp

[rsanchezsaez]

Thanks everybody for your comments!

---

@RPicster I acknowledge your concerns. In addition to what Bastiaan said, I want to point out that, strictly speaking, there's not as much new code in this PR as it seems.

The bulk of the change is a refactor of the iOS platform into drivers/apple_embedded, with mostly class renames. (Which admittedly makes it difficult to review.) If you look at the /platform/ios and platform/visionos contents, that's basically the new code (in addition to the build system changes).

---

@saji8k When using Metal to render directly on visionOS through Compositor Services, only static foveation is supported. We're planning to support this in Godot as well (on the Mobile renderer). Here are additional technical details about how this is done on visionOS: https://developer.apple.com/wwdc23/10089

---

@stuartcarnie Thanks you for your interest, and for your great work on the Metal driver!

To avoid confusion, I want to clarify that Godot Vision's approach is a bit orthogonal to ours. They are dynamically translating Godot entities into RealityKit entities, and using the RealityKit backend for rendering. Whereas, we are looking to contribute direct native Metal rendering on visionOS.

---

@dsnopek Yeah, we considered subsuming visionOS support into the iOS platform, but found several important downsides.

Like @migueldeicaza mentioned, the precompiled Godot framework in the export template for such a platform would need to have all individual platform binaries to choose from, so at the end of the day the build system for such a platform would do what's done individually on iOS/visionOS here. This would impose a burden on users interested in iOS but not in visionOS.

And then, the client-specific code on iOS would have a lot of if iOS { ... } else if visionOS { ... }. We felt that subclassing an abstract platform was the best solution here for code simplicity and maintainability.

Regardless, the libgodot contribution sounds really interesting. It seems like a worthwhile improvement for the engine's architecture. It would be good to know what the planned timeline for that PR is, because it has significant merge conflicts with the changes in this PR. Thanks!

[rsanchezsaez]

@migueldeicaza I'm also curious, how does #90510 compare to https://github.com/migueldeicaza/libgodot ?

[rsanchezsaez]

@bruvzg Thanks for the review! I addressed your comments.

[akien-mga]

I'll do a quick review pass ASAP, and then we should be good to merge as far as the code changes are concerned.

Also, commits should be squashed to either a single commit or a few individual commits (if you prefer to keep it separated in some meaningful way) to keep history searchable and clean.

This point will still be outstanding, as we prefer PRs to be carefully crafted to have one or a handful meaningful commits, but not the full history of development, upstream merges and review fixes (which can be relevant during an ongoing PR review, but not to the merged Git history and instead introduce noise and potential issues when bisecting).

We could also exceptionally use the "Squash and Merge" feature if we think it's worth keeping the review history in this PR, but have it all squashed to a single commit. But I suspect there might be value in keeping some more granularity here, such as a first commit for the iOS -> apple embedded refactor, then a commit to add visionOS, and maybe a few more for adjacent engine changes.

[rsanchezsaez]

Force pushed with the two commits suggested by @akien-mga. I have made sure checking out the first commit builds the iOS platform successfully.

[akien-mga]

I reviewed the code mostly with an eye for: code style and maintainability, architecture, buildsystem, thirdparty, documentation, compatibility.

Overall it looks great to me!

I pointed out a few things which should IMO be addressed before merging (marked as "[BUG]" or "[BUG?]"), and a bunch more things which are mostly code style / quality / maintainability improvements, which could be done in a follow-up, or here, as you prefer.

[akien-mga]

And before merging I'll have to test run the iOS build through our current build scripts to confirm it still works fine: https://github.com/godotengine/godot-build-scripts

As for adding official builds for visionOS, that would be great but will require quite some work I expect, ideally to make it possible to build with osxcross like we do for macOS and iOS.

A few years ago @naithar did work on adding tvOS support, which was never finalized/merged, but shows that it was likely possible:

• [3.x] [tvOS] Basic tvOS Support #45829
• tvOS build support build-containers#71
• tvOS build support godot-build-scripts#21

I can't promise that I'll have time myself to work on adding visionOS builds to our pipeline. I'll try to have a quick look but if it requires several days of work, it's unlikely I'll be able to allocate that time.

[akien-mga]

And before merging I'll have to test run the iOS build through our current build scripts to confirm it still works fine: https://github.com/godotengine/godot-build-scripts

I tested and confirmed that iOS export templates can still be built properly with osxcross/cctools-port derived toolchains (w/ iOS SDK 18.2).

Here's a test build from the current PR if someone wants to test and validate that there's no evident regression (including the C# build):

• Standard, macOS editor: https://github.com/akien-mga/godot/releases/download/_tmp/Godot_v4.5-dev-gh105628_macos.universal.zip
• Standard, iOS templates: https://github.com/akien-mga/godot/releases/download/_tmp/Godot_v4.5-dev-gh105628_export_templates.tpz
• Mono, macOS editor: https://github.com/akien-mga/godot/releases/download/_tmp/Godot_v4.5-dev-gh105628_mono_macos.universal.zip
• Mono, iOS templates: https://github.com/akien-mga/godot/releases/download/_tmp/Godot_v4.5-dev-gh105628_mono_export_templates.tpz

[rsanchezsaez]

@akien-mga Thanks for your really in-depth review! I have addressed or replied to all of your comments. Could you take another look?

[rsanchezsaez]

@AThousandShips Thanks for the review! I have addressed all of your comments.

[shiena]

Just a quick question: will the visionOS template build be added in a separate pull request after this one is merged?

[akien-mga]

Looks great to me.

Amazing work @rsanchezsaez and everyone involved!

[akien-mga]

Just a quick question: will the visionOS template build be added in a separate pull request after this one is merged?

Do you mean on the GitHub CI?

This could be added in a follow-up yes, it should be fairly easy.

[akien-mga]

Thanks! And congrats for your (very impressive) first merged Godot contribution 🎉

[AThousandShips]

Great work!