More control over memory allocations (specifically images)

[BoyBaykiller]

I load Intel Sponza Base using SharpGLTF and notice that even after a couple of minutes my application has a weirdly high RAM usage of 5GB:

Almost all of that comes from dead byte[] allocated by SharpGLTF:

Explicitly calling GC.Collect() is needed:

I'd rather not have my app keep Gigabytes of unused memory arround for no reason. SharpGLTF could make memory managment more transparent. We could focus on images only for now since those alone make up 2GB on this model.

As an example, currently all external images are loaded (even if the app never references them) into this managed byte[]. But this is not specific to external images.

SharpGLTF/src/SharpGLTF.Core/Memory/MemoryImage.cs

Lines 126 to 128 in 4b28af2

	filePath = System.IO.Path.GetFullPath(filePath);
	var data = System.IO.File.ReadAllBytes(filePath);

---

I'd like to:

1. Load images into my own memory
2. Control when I load it
3. Control if I load it at all

I don't know how other libaries handle this, but this is what I found fastgltf doing which has become quite popular recently: https://github.com/spnda/fastgltf/blob/7889b48e1435da943dcbeab5af4fbf7f39f1df0b/examples/gl_viewer/gl_viewer.cpp#L516-L555
Much more control is given to the user. I don't even need to make a copy. I can tell stbi image to load the image directly from the memory or file.

[vpenades]

first of all, keep in mind that SharpGLTF is different to most libraries in that most libraries around, like fastgltf, are designed only to read glTF, not to write them. That allows these libraries to make strategic decisions that improve loading performace.

Meanwhile, SharpGLTF tries to load the whole model as a "snapshot" so you have the whole document in memory so you can edit and save it.

Certainly, that doesn't mean improvements cannot be done. In fact there's some things you can already do.

I you see the ReadSettings class that is passed to the model loader methods, it has support to hook into the image loading step so you can load the image directly and not store it, or skip image loading completely.

[BoyBaykiller]

Alright I see, feel free to close.

I've briefly looked at ReadSettings.ImageDecoder callback but it's not sufficient for my purpose.

SharpGLTF/src/SharpGLTF.Core/Schema2/Serialization.ReadSettings.cs

Lines 256 to 269 in 5153478

	foreach (var image in this._images)
	{
	// reads the image file into the current object.
	image._ResolveUri(context);
	// if we have a decoder hook, call the decoder, and free the memory.
	if (context.ImageDecoder != null)
	{
	if (!context.ImageDecoder(image))
	{
	image._DiscardContent();
	}
	}
	}

The encoded image is still loaded into memory without my control and even though it says _DiscardContent, that just sets references to null which means I still end up having to call GC.Collect(). On top of that, my image loading is parallelized and asynchronous with the render thread so integrating that into this simple callback would be an other challenge.

I just need way more control over memory in order to be "optimal" and if exposing all that is not part of the primary design thats totally understandable and fine.
I'll just accept that there will be some unnecessary copies for image loading and that right now at least in my app I have to call GC.Collect() in order to free the gigabytes of unused memory.

Let me link #267.

[vpenades]

Notice that the ImageDecoder callback returns a boolean.

So if you want to completely skip image loading you eed to override ImageDecoder callback, do nothing inside it, and return false.

_DiscardContent is intended to set the Image node in a state that is not faulty for the validators that run later.

Also notice that for satellite content loading there's yet another hook that resolves the location of the satellite file and loads the data, you probably need to hook that one too and return an empty image instead of loading the asset.

It probably feels overcomplicated, but consider it's two separate problems:

• From where and how to load the images. (there's people that want to hook into this, but want the image to be processed normally)
• What to do with the loaded image (there's people that want the image to load normally, but want to process it in some way)

It just happens you require the two of them