how to release gpu memory when use onnxruntime with fastapi

[SZ-ing]

This is probably a repetitive problem, but I still haven't found how to solve it.

I used FastAPI to build API interfaces and used onnxruntime to load models. However, due to the limited number of models and gpu memory, I hope to release this part of the occupied gpu memory after each interface call is completed. But it seems that I can never fully release it, and there will still be some models occupying gpu memory.

I want to know if there is any way to solve this problem for Python.

[skottmckay]

The infererence session will have the weights for the model in memory at a minimum. Unless you free the session that will always remain the case.

You can potentially reduce memory usage by sharing allocators between sessions. The python API has create_and_register_allocator which calls into CreateAndRegisterAllocator mentioned in 'Share allocator(s) between sessions' here. This reduces the amount of unused memory from having multiple arenas, each with their own over-allocation.

See #6411 (comment)

[panp4n]

The same question. GPU memory cannot be fully released in Python.
I tried:
ort_memory_info = ort.OrtMemoryInfo("Cpu", ort.OrtAllocatorType.ORT_ARENA_ALLOCATOR, 0, ort.OrtMemType.DEFAULT) ort.create_and_register_allocator(ort_memory_info, None)
But it doesn't work.