Implement texture mip-level streaming - #113429
Conversation
59cceac to
9a6ea56
Compare
|
I did some initial testing on Mac and android and as of now it seem´s to work pretty well in the bistro demo. Though it seems that on the new textures ui upon setting a custom value for max / min and then save it to proj sethings, it doesn't apply in game , only in the editor, even if they appear modified in the project sethings( I checked it and while they are modified to use the values I set up in the ui , upon running in game, it doesn´t work). It only works when you set the min/ max res via the project settings directly or when you reimport the texture. it works the same way when you set your proj sethings directly, the ui textures does not update according to the setup sethings. Also if I may ask which are the areas that need´s the most testing rn ? since even in my own project I don´t see any issues + they load correctly in game(first it loads the lowest res and then the one I set up earlier, though I don´t see any pop in?). Also it does seems like the streaming buffer doesn´t use kb or mb just says a whole lot of numbers. Edit : For feedback i would add a sething that reimports all compressed or 3d material texture as streamed ones. Moreover are lightmaps textures supported rn? |
|
Seems to work really well. In the profiler, could we have the Streaming Texture Mem Used with the same units as the other? 
|
|
I haven't explored the PR in detail, overall it seems the techniques employed are good, but I'll have to see it in action in a big project to make sure it performs well, as I'm definitely concerned about there being no throttling between the threads dedicated to streaming and the main threads when doing the swapping. That should be evident once I look at how the threads interact between each other. Is it worth introducing a new type of resource for it? I figure in the long run most users would just want this to be a toggle in CompressedTexture2D along with the streaming configuration so they don't have to re-import all of their assets. It should be fine to default to it being enabled normally, artists can usually just go and disable it for assets that are critical to retain full quality. That'd also allow for much quicker testing on bigger projects. |
Sounds like a bug...i'll investigate.
The general processes work. I think I'm most concerned if this matches what others needed or not. So definitely want bigger projects and people actually making games to test it out.
Definite just an over sight. I had my own overlay in test projects which did convert it and forgot to add the case for the editor graph. |
Definitely an area I want more testing as well. I've a test project I'll be uploading that allows to create really big vram projects and it seems to handle it fairly well there. The main limiting factor seems to be the GPU transfer throttle the texture uploads but I could be wrong. I would often see this in my graphs where i'm request x resolution but you see the overall VRAM ramp up more slowly. If we need to add better throttle there I have a plan for that.
I was on the fence about this a well but that was the direction Clay thought was best. Re-importing is a bit of pain for sure. |
|
I've pushed fixes for monitor units and the saving of settings (seperate commits for now but will rebase them later). |
This is a particular area I'd like to make sure that it's taking advantage of using RD's transfer queues, which are only used if textures are created with initial data at the moment (as in, the option to pass a Vector of bytes during texture creation). Are you aware if the PR is currently taking this path for any textures streamed in? |
Its doing the same as |
Lightmaps and 3D textures are not supported yet. I think they could be eventually but I need to look into them both more before I could really say how hard it would be. The basics of the system is to calculate a mip-level and its a per-texture callback that is used to change the textures so if you can make those systems work....it can be done. The things that might complicate it would if calculating the mip-level needs to be different or if the format for them needs to change for it. I'll add these both to the future work list though. Edit: Forgot to say, adding a UI that helps reimport is possible though. If we decide to stick with the separate import type that is something that could be added. A window with a select/deselect all plus a filter to narrow it down would probably make it pretty fast for most people. |
This sounds like an area that we should definitely improve upon then, as I imagine it's consuming a lot of the upload bandwidth just reuploading the mipmap chain over and over. But you'll probably need some deeper help in the RD department to make that efficient. |
|
I've thought about adding new methods to copy as much of the existing texture gou side and then only upload the new level if needed. This makes downsizes go inside only. Definitely an opportunity for future improvements. |
|
Is there any technical limitation why these Texture2D options aren't available in Texture2D Streamed?
I already know that the other options in Texture2D can't be in Texture2D Streamed due to limitations. |
|
Bit of a question - How much will this reduce VRAM usage? Do you plan on having an example project? |
It depends on how is your project and the amount of unique textures you have, but generally this helps a lot with texture vram. Also as he said at the top you can set a memory budget on the Max amount of vram that the textures on your scene will consume. You need to test it and send some feedback if you can. |
I don't have a great answer. Clay suggested removing them. I kinda was under the impression they are more focused on 2D uses. For a png or jpeg there is no technical reason that couldn't be done though. For DDS textures, which are already compressed via lossy algorithms, decompressing to run processing and recompressing would further degrade the result. I think in general there are open questions in this area. |
Channel Pack: I understand is used to compress a texture in BC4 when it only presents the R channel as information, which is why I believe it's relevant for streaming textures. Channel Remap: allows for the reorganization of texture channels, which is why I also think it's relevant for streaming textures, but I believe it would make the Streamed Texture2D format significantly more complex. Fix Alpha Border: is a bit of a dilemma because it's not made clear if it's only for 2D sprites or if it can be used for textures in the 3D environment. Premult Alpha: is similar to Fix Alpha Border as it fixes the borders of textures with transparency, but the description clearly states that it is necessary when using BLEND_MODE_PREMULT_ALPHA in a BaseMaterial3D. Normal Map Invert Y: perhaps this option can cease to be a "deprecated" option and become an exclusive option for Streamed Texture2D as a small replacement for Channel Remap, as it allows converting DirectX-style Normal Maps (Y-) to OpenGL (Y+). |
|
This looks great! Haven't had time yet to dig too deep into the code, but I like what I am seeing. Here is some initial feedback: I am really not a fan of the new texture type. IMO this should just be a checkbox on the already existing texture asset. I am also not happy with the fact that the min/map/initial resolution values are specified in pixels. After all, what really matters in 3D scenes is not the actual image size, but the Texel density: Let's say I have two textures, TexA is a baked 256x256 texture for my "old_key" asset while TexB contains a 4096x4096 trim sheet I use to texture parts of my environment. Both have the same Texel size. Now I want to reduce my scenes VRAM on low end hardware and set the max texture size to 1024. While doing so will save me some VRAM I now have widely different texture resolutions across my scene: my trim sheet is quite low res while my key is still using the full resolution texture. If, however, those settings were to use a mip_level int instead of a pixel resolution as input I could avoid this problem. By e.g. setting the max resolution to e.g. "1" all my textures would now skip mip level 0, my entire game is thus now rendered with half res textures. |
Thanks!
That is actually how i've had it during development. I only split it out after I presented it at a rendering meeting. One thing is it is wanted for 3D but not 2D. Texture2D is used for both. There are presets that could default it on or we could add a system preference to default it on or off. Definitely something that I think needs agreement. The other issue not all Texture2D types are streamable. My original implementation would just show a streamable bool(plus the min/max) for VRAM compressed.
I've gone back an forth personally on this. The thing that change my opinion was just that mip level 0 doesn't mean anything either besides the highest quality. I have no idea if the texture is big or small. I don't have a strong preference for resolutions though. If the consensus is display to miplevels we can do that. I ended up with this because i had a project which had a texture that was really an atlas with lots of smaller textures packed. I was more clear to me on that project what size was needed to maintain a certain visible level. But, i was also just trying to get the core feature implemented. I'd love more feedback from people making games to know what they need. I think unity just allows you to specify how many levels the texture can drop. There are definitely diminishing returns in multiple ways. Memory gains get less plus quality becomes garbage. Switching schemes really isn't that difficult so we really just need to decide how it should be exposed to people. |
|
I do agree with ansraeer that having it separate it's a bit counter intuitive, specially for new users that don't bother checking sethings, though i think you should talk with clay about it since i think most of us woukd rather a toggle rather than a separate class. Also testing with my project which has max 1024/512 texture quality, it works pretty well and i don't have any performance issues ( i use lots of alpha scissors materials tho , + pbr stylized materials). |
|
Also I found some pop in issues with alpha materials like these. I set a very low memory budget like 4mb in order to see these more clearly. Also it reduced the texture gram in bistro a ton if people want to know ( 2 times less cram with default memory budget). alpha.mp4streaming.alpha.mp4 |
There are some limits in what you can do with a really constrained budget. For less pop-in you usually have accept a lesser quality in general. I kinda want a option that does that. But any time the textures exceed the budget decisions have to be made....and that can cause bigger transitions. Its all much less noticeable when you have a min size that still has detail. When you allow a min that looses all detail you really notice that addition or detail. Definitely an area to work on more. Hopefully with more feedback we can get a plan that works well. |
|
This seems to help with individual mipmap copies for the same texture, but different textures still have unnecessary barriers inserted inbetween. I think this is indeed because RDG is force inserting barriers due to not knowing what the previous usage of the texture is, as it doesn't have a draw tracker. Have you added a path to fully reload the texture instead of copying existing mips? I feel like we may need to go with that to avoid complications coming from this approach. EDIT: Oh, I see you did! I'll be testing this and compare. |
See https://github.com/godotengine/godot/pull/113429/changes#diff-d52171cd03d9690f66ea71a6db3309b3971ed6dc8804a82a9e0aa23b28387a2bR54 or search for STREAMED_TEXTURE_ALWAYS_RECREATE It seems to be hitting this, https://github.com/godotengine/godot/blob/master/servers/rendering/rendering_device_graph.cpp#L570, because the new texture has USAGE_NONE initially, and then this new method is checking the destination to see the it needs transitioned to mutable, so the graph adds the barrier. BUT....i'm not sure we need this destination check for the streaming case because we always have a new texture(you do need it for an existing texture but it could be optional for a new texture if it gets initialized correctly maybe). I've not had enough caffeine yet to convince myself fully. I'm also fine with the full replace path if that is what we want. They both seem to perform well enough for real life scenarios. |
|
Also, what's the reason for |
I had cases where I would see a frame spike because freeing a texture normally triggers recreating the full uniform set and its likely needed in that case. I think this was mostly when a single texture was shared in multiple place, you then had lots of meshes that needed uniform sets recreated because the tracker just marks it as dirty whenever a texture is freed. The compatible method was an attempt to just patch the existing uniform because we know we are not changing formats or other things that could done with the other path. Its the same texture, just sized differently. Not very easy where its not bindless or whatever the modifiable after creating textures are called. I think at the time it was more cost per call but it didn't include the side effects of recreating the full uniform set so it was more consistent. Its kinda frustrating area to trouble shoot since the texture freeing just queues work which you see processed on future frames when the textures are no longer in use. |
|
I'm looking at it right now, and the culprit doesn't appear to be This could probably be improved separately, removing the need for |
Well, that is what I meant, I think later the update recreates they uniform set as well, but it was months ago so I could me misremembering slightly. Just glad to have someone finally reviewing so thanks! |
|
Finally had the time to test on my macbook so i could check the attic scene from nvdia ported by @johnlogostini and in the editor as seen here is able to stream the textures of this AAA scene pretty fast 20260706_102520.mp4Meanwhile when going into play mode it's sadly far slower in comparison( note that decreasing the bilinear scaling resolution didn't make any difference in the streaming time as it was being as slow as before.Also while increasing the max ops to a value like 10k mitigate it, it's still slower than the editor loading it instantly. 20260706_103242.mp4As for vram the texture mem went from 930 to 340, so ye basically 3 times. As for framerrate , without streaming it gets to 40 fps and with streaming it drop to 16-17 , though there were no particle system in the scene from what i saw. On other side the bistro demo suffers a similar issue with streaming load times ( though fps are pretty much fine costs way fewer ms than the attic scene). |
|
Were you testing a "click-play-and-run" build from the editor, or a Release export? Not looking for a gotcha, just need full context for the numbers you're giving. |
I tested with an artifact, though i think the loadtime shouldn't be that different. |
| uint32_t mip_width = header.width >> load_mip; | ||
| uint32_t mip_height = header.height >> load_mip; |
There was a problem hiding this comment.
Need MAX(1u, x) for both of these, otherwise they can become 0.
There was a problem hiding this comment.
Bump @tdaven, please don't forget this one, need MAX(1u, x) for both
| #ifdef MODULE_TEXTURE_STREAMING_ENABLED | ||
| { | ||
| RD::Uniform u; | ||
| u.binding = 37; | ||
| u.uniform_type = RD::UNIFORM_TYPE_STORAGE_BUFFER; | ||
| RID instance_buffer = TextureStreaming::get_singleton()->feedback_buffer_get_uniform_rid(); | ||
| if (instance_buffer.is_null()) { | ||
| instance_buffer = scene_shader.default_material_feedback_buffer; | ||
| } | ||
| u.append_id(instance_buffer); | ||
| uniforms.push_back(u); | ||
| } | ||
| #endif // MODULE_TEXTURE_STREAMING_ENABLED |
There was a problem hiding this comment.
There seems to be a potential performance impact from the feedback buffer.
Since it's passed as a writable storage buffer, RDG assumes the buffer will be written to in every pass and inserts pipeline barriers between everything. This nearly doubles the amount of render graph levels in every project I tested. On a RTX 5080, I lose about 0.2 ms GPU time from this, by simplying enabling the texture streamer and not even using any streamed textures.
Not sure how this could be fixed. Even if you use the buffer only for the main pass and fall back to default_material_feedback_buffer for the rest, the latter still has a draw tracker that would insert barriers between everything.
Maybe by having the uniform only exist in the passes where the feedback buffer is used?
There was a problem hiding this comment.
I'm not sure that would work because I think the way its written today it expects a single uniform that works for all. We could just teach render graph about the use case and add a way to mark a buffer and not needing the same constraint.
The entire operation is just the buffer clear, the atomicMax and the buffer copy so the fact the render graph thinks it needs extra barriers because its writable is kinda just it not knowing about this new use case. It doesn't actually need the barriers. If that buffer could be tagged as not needing the barrier the RDG could skip adding them so hopefully non-streaming cases see a more similar result.
There was a problem hiding this comment.
I added a commit which adds a new creation flag to the buffer which the RDG then uses to skip the additional barriers. Seems to work from my testing if its not considered too much of a hack.
There was a problem hiding this comment.
Your approach is similar to what I was planning, but it has the issue of implying every usage is globally atomic, when that's not the case. Calling buffer clears repeatedly would make them overlap, for example. We want to more explicitly specify this for storage buffers instead.
I have an idea after discussing this with Darío, I'll work on implementing it this week.
There was a problem hiding this comment.
I kept poking at this. No clue if this is more what you had in mind but now it tries to just track via usage instead so it only does barriers when usage is different which should help the clears as well if you have mixed usages though i've not tested that much.
|
I've noticed that the texture preview in the resource also changes its quality when changing LODs. Is this intended? 
|
@kekuyer228, nope! I broke this when a rebase at one point. Fix queued up. Thanks! |
| textures_budget_slider->set_min(1); | ||
| textures_budget_slider->set_max(8192); // 8 GB |
There was a problem hiding this comment.
Add a visible suffix for the unit, and limit the lower bound to a reasonable value:
| textures_budget_slider->set_min(1); | |
| textures_budget_slider->set_max(8192); // 8 GB | |
| textures_budget_slider->set_min(256); // 256 MiB | |
| textures_budget_slider->set_max(8192); // 8 GB | |
| textures_budget_slider->set_suffix(" MiB"); |
There was a problem hiding this comment.
256MB was the whole VRAM size for PS3. During my (small) period of testing, the engine would start complaining about the budget being too low at roughly 10MB. So 16MB being the minimum would probably make more sense? Unless you allow entering any value manually, then any limit is fine
|
I've pushed a fix for the texture preview issue mentioned, the suffix addition and I have removed the 2 commits we agreed on during the last meeting. I've also removed the DDS file importing for now and plan to open a new PR for adding that back. The main motivation for this is to be sure the change are compatible with old projects. Its better targeted as its own PR to improve this feature than to add more to getting it merged. I'll open a draft PR for it shortly. |
|
Documentation PR godotengine/godot-docs#12268 I've got a few edits that can land in the class documentation to point to those docs after they merge as well if desired. |
blueskythlikesclouds
left a comment
blueskythlikesclouds
left a comment
There was a problem hiding this comment.
Implementation is looking pretty solid. The previous RDG performance problem I found can be fixed later, but I found an issue with early fragment tests that should be fixed before merge.
| state.min_lod_override = uint8_t(CLAMP(p_min_lod, 0, 14)); | ||
| state.max_lod_override = uint8_t(CLAMP(p_max_lod, 0, 14)); |
There was a problem hiding this comment.
Are there some inconsistencies here? User facing limit is 13 but some code clamps to 14.
There was a problem hiding this comment.
This one is actually fine sinze these map to the settings where 0 is the default., not lod 0. Should at least add a comment though so that is clearer.
There was a problem hiding this comment.
I tried to clarify this more with some constants....it might be better.
| uint32_t mip_width = header.width >> load_mip; | ||
| uint32_t mip_height = header.height >> load_mip; |
There was a problem hiding this comment.
Bump @tdaven, please don't forget this one, need MAX(1u, x) for both
| /* Varyings */ | ||
|
|
||
| #if defined(TEXTURE_STREAMING) && !defined(MODE_RENDER_DEPTH) && (defined(UV_USED) || defined(STREAMING_UV_USED)) | ||
| #if !defined(TRANSPARENT) || defined(TEXTURE_STREAMING_PREPASS_ENABLED) |
There was a problem hiding this comment.
The TRANSPARENT doesn't work correctly here without the depth prepass for alpha clip materials. It's also a wrong assumption, alpha blended objects can utilize early fragment tests just fine. It only needs to be disabled when using discard, or writing to depth.
I'd recommend getting rid of the new shader version you added entirely, since it ends up compiling more shader permutations, even when texture streaming is not used. Instead, like you already do for the mobile renderer, I would look for all defines that enable the use of discard in the scene shader, and use them in this if statement here.
This also needs to be done for user shaders. For that, you can extend usage_defines for DISCARD and DEPTH to output defines like DISCARD_USED and DEPTH_USED, then check them here.
I'd remove TEXTURE_STREAMING_PREPASS_ENABLED entirely.
Here's my test, the depth prepass option is disabled:
| Texture streaming off | Texture streaming on |
|---|---|
 |
 |
rendering via a SSBO buffer. This material feedback is merged and processed to determine the needed lod for each texture. New settings allow a system wide min/max lod selection as well as the ability to override these settings per-texture. Settings also allow constraining textures to a memory budget(when possible unless already at the minimum quality). The texture fitting algorithm tries to keep textures in memory if possible. It does a good job at using the memory budget. There is a new performance monitor to help visualize texture memory usage.
This adds a new method to the rendering server API, `texture_replace_compatible`, which is intended to be used when replacing a texture with another that is guaranteed to be compatible (same format, srgb, type etc). This allows renderers to implement a fast path for this case, which is used by the texture streaming system when replacing a texture with an updated version. This avoid expensive recreation and possible recompilation of pipelines.
20 participants
Relates to godotengine/godot-proposals#3177.
This PR implements texture mip-level streaming. This is done by loading a smaller version of a texture when its use is far away or at a angle that doesn't require as much detail. This can greatly reduce the VRAM requirements for textures.
Features:
Notes
This implementation does not use sparse or partially resident textures. This just loads and swaps textures via normal methods.
This is targeted at 3d usage. I haven't really tried to ensure 2d usage works. If this is an important use case it could be done.
A material feedback buffer is bound to the shader and the used mip-level gets written to that buffer by materials that use a texture that is configured for streaming (and streaming is enabled). From this material feedback buffer, the requested level is compared to the current loaded level and the correct level is chosen.
Textures can be constrained to a memory budget which will reduce texture resolution to maintain that budget unless the min level is insufficient to fit within the budget. The current fitting algorithm is quite good at using the full budget and keeping textures around if it has space to do so.
To use streaming, it must be enabled in settings. This will require a editor restart before it will function. Currently, settings are only read once on load. You can make runtime changes via the
TextureStreamingsingleton.After enabling streaming, textures need to be imported as streamed textures. The new texture import type "Texture2D Streamed". This new texture type has settings to override the min/max resolution allowed for that texture. The per texture properties override the system settings. The default is to use the system settings:
The editor has a new menu and options to adjust the textures while the editor is running. I've no clue if this is what people want. I implemented what I could but I'm unsure its all helpful. Feedback requested.
Streaming is implemented for the mobile and forward plus renderers. The compatibility renderer doesn't have a great way to read back data from a shader since it is opengl es 3.0 limited. In 3.1 support for SSBO were added. Someone could probably write to an FBO and read pixels back the result if it was really desired or perhaps make it an optional feature if appropriate extensions are enabled. The mobile renderer currently lacks a depth pre-pass (though i've seen rumors of others interested in doing that) so performance can be worse when using alpha scissors or shaders which use discard. This is due to forcing early z testing on. There are probably alpha related bugs or conditions which need documented better.
The
UVvariable can be written to in a fragment shader. This allows the streaming logic to get the correct miplevel for shaders that modify the UV's in the fragment shader. This might be better as a separate output but making UV writable was easy enough I just went with it.Small demo video:
Screencast.From.2025-12-01.22-27-02.mp4
In order to allow DDS files to be imported as streamed textures, I had to make some changes. DDS files to are not imported. There is a resource loader which directly loads them into a texture type based on the contents of the DDS file. I believe the change i've made is compatible but others should test. The default import type is just "Texture" as it was before. It still gives you a texture type that depends on the type of DDS file. But, you can also select a streamed version now.
The format for a
Texture2D Streamedis based on CompressedTexture2D but simplified and with a modified format that allows easier loading of specific mip-levels.Currently there is no management of a memory pool for textures. For the vulkan case, the VMA is is used and seems to do a good job. It could be we encounter fragmentation with d3d12 or even with vulkan. I've mostly developed on Linux/AMD GPU. I know it works on windows with vulkan and d3d12 and have had others use it on NVidia systems but I've not personally tested there as much. Something I think needs attention as people experiment with it. Metal has not been tested.
Projects
I've got some test projects or modified projects as demos. Will add link here eventually.
Future
Swapping textures could be more performant. Right now the entire uniform gets recreated when a texture is swapped. I believe we could add an API to update only the necessary descriptor instead of recreating it. Its been good enough for most things though.I'm sure there are bugs or things left over through out the code. Its been a long journey... This has been pet project of mine for far too long and it needs to get out to where more can test it.