Session Run throws an access violation exception when I recreate the session

[xeeetu]

Describe the issue

First I create variables which I then initialise. In the main function in the while loop, when a certain condition is met, I have to change the model for the output. I try to clear the variables initialised above and assign new values to them, but after changing their values the Session.Run function throws an exception: 0xC00000000005: Access violation reading location 0x0000000000000000000000000000000000000000000000000000.
I have a question, what am I doing wrong and how to change the values in the inference variables correctly. Do the variables Env, RunOptions, MemoryInfo need to be created again each time the model is updated or should they be initialised once when the program is started? Sorry for code repetition, I tried to put initialisation in a separate function, passing variables by value, but I get the same address access error (is it possible to pass Ort variables by reference? because they contain smart pointers inside them and ownership is lost? ).

To reproduce

int main()
{
const int _netWidth = INPUT_W; //ONNX-net-input-width
const int _netHeight = INPUT_H; //ONNX-net-input-height

    int _batchSize = 1;  //if multi-batch,set this

    //ONNXRUNTIME	
    Ort::Env _OrtEnv = Ort::Env(OrtLoggingLevel::ORT_LOGGING_LEVEL_ERROR, "YOLOV5");
    Ort::SessionOptions _OrtSessionOptions = Ort::SessionOptions();
    Ort::Session* _OrtSession = nullptr;
    auto _OrtMemoryInfo(Ort::MemoryInfo::CreateCpu(OrtAllocatorType::OrtDeviceAllocator,   OrtMemType::OrtMemTypeCPUOutput));
    Ort::RunOptions _RunOptions{ nullptr };

#if ORT_API_VERSION < ORT_OLD_VISON
char* _inputName, * _output_name0;
#else
std::shared_ptr _inputName, _output_name0;
#endif

    std::vector<char*> _inputNodeNames; 
    std::vector<char*> _outputNodeNames;

    size_t _inputNodesNum = 0;  
    size_t _outputNodesNum = 0;  

    ONNXTensorElementDataType _inputNodeDataType; 
    ONNXTensorElementDataType _outputNodeDataType;
    std::vector<int64_t> _inputTensorShape; 

    std::vector<int64_t> _outputTensorShape;


    //-------------------------InitDetector----------------------------

    _OrtSessionOptions.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);

#ifdef _WIN32
std::wstring model_path(modelPath.begin(), modelPath.end());
_OrtSession = new Ort::Session(_OrtEnv, model_path.c_str(), _OrtSessionOptions);
#else
_OrtSession = new Ort::Session(_OrtEnv, modelPath.c_str(), _OrtSessionOptions);
#endif

    Ort::AllocatorWithDefaultOptions allocator;

    //init input
    _inputNodesNum = _OrtSession->GetInputCount();

#if ORT_API_VERSION < ORT_OLD_VISON
_inputName = _OrtSession->GetInputName(0, allocator);
_inputNodeNames.push_back(_inputName);
#else
_inputName = std::move(_OrtSession->GetInputNameAllocated(0, allocator));
_inputNodeNames.push_back(_inputName.get());
#endif

    //cout << _inputNodeNames[0] << endl;
    Ort::TypeInfo inputTypeInfo = _OrtSession->GetInputTypeInfo(0);
    auto input_tensor_info = inputTypeInfo.GetTensorTypeAndShapeInfo();
    _inputNodeDataType = input_tensor_info.GetElementType();
    _inputTensorShape = input_tensor_info.GetShape();

    //init output
    _outputNodesNum = _OrtSession->GetOutputCount();

#if ORT_API_VERSION < ORT_OLD_VISON
_output_name0 = _OrtSession->GetOutputName(0, allocator);
_outputNodeNames.push_back(_output_name0);
#else
_output_name0 = std::move(_OrtSession->GetOutputNameAllocated(0, allocator));
_outputNodeNames.push_back(_output_name0.get());

#endif
Ort::TypeInfo type_info_output0(nullptr);
type_info_output0 = _OrtSession->GetOutputTypeInfo(0); //output0

    auto tensor_info_output0 = type_info_output0.GetTensorTypeAndShapeInfo();
    _outputNodeDataType = tensor_info_output0.GetElementType();
    _outputTensorShape = tensor_info_output0.GetShape();

.
.
.
while(!(GetAsyncKeyState(0x70) < 0)
{
.
.
.
input_tensors.push_back(Ort::Value::CreateTensor(_OrtMemoryInfo, inputTensorValues.data(), input_tensor_length, _inputTensorShape.data(), _inputTensorShape.size()));

        output_tensors = _OrtSession->Run(_RunOptions,
            _inputNodeNames.data(),
            input_tensors.data(),
            _inputNodeNames.size(),
            _outputNodeNames.data(),
            _outputNodeNames.size()
        );

.
.
.
if(----condition----)
{
_OrtEnv.release();
_OrtSessionOptions.release();
if (*_OrtSession != nullptr)
delete _OrtSession;
_inputNodesNum = 0;
_inputName.reset();
_inputNodeNames.clear();
_inputTensorShape.clear();
_outputNodesNum = 0;
_output_name0.reset();
_outputNodeNames.clear();
_outputTensorShape.clear();

    //-------------------------InitDetector----------------------------
        _OrtSessionOptions = Ort::SessionOptions();

        _OrtSessionOptions.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED);

         modelPath = ....

#ifdef _WIN32
std::wstring model_path(modelPath.begin(), modelPath.end());
_OrtSession = new Ort::Session(_OrtEnv, model_path.c_str(), _OrtSessionOptions);
#else
_OrtSession = new Ort::Session(_OrtEnv, modelPath.c_str(), _OrtSessionOptions);
#endif

        Ort::AllocatorWithDefaultOptions allocator;

            //init input
            _inputNodesNum = _OrtSession->GetInputCount();

#if ORT_API_VERSION < ORT_OLD_VISON
_inputName = _OrtSession->GetInputName(0, allocator);
_inputNodeNames.push_back(_inputName);
#else
_inputName = std::move(_OrtSession->GetInputNameAllocated(0, allocator));
_inputNodeNames.push_back(_inputName.get());
#endif

            //cout << _inputNodeNames[0] << endl;
            Ort::TypeInfo inputTypeInfo = _OrtSession->GetInputTypeInfo(0);
            auto input_tensor_info = inputTypeInfo.GetTensorTypeAndShapeInfo();
            _inputNodeDataType = input_tensor_info.GetElementType();
            _inputTensorShape = input_tensor_info.GetShape();

            //init output
            _outputNodesNum = _OrtSession->GetOutputCount();

#if ORT_API_VERSION < ORT_OLD_VISON
_output_name0 = _OrtSession->GetOutputName(0, allocator);
_outputNodeNames.push_back(_output_name0);
#else
_output_name0 = std::move(_OrtSession->GetOutputNameAllocated(0, allocator));
_outputNodeNames.push_back(_output_name0.get());

#endif
Ort::TypeInfo type_info_output0(nullptr);
type_info_output0 = _OrtSession->GetOutputTypeInfo(0); //output0

            auto tensor_info_output0 = type_info_output0.GetTensorTypeAndShapeInfo();
            _outputNodeDataType = tensor_info_output0.GetElementType();
            _outputTensorShape = tensor_info_output0.GetShape();

}
.
.
.
}
.
.
.
return 0;
}

Urgency

As soon as possible

Platform

Windows

OS Version

10 22h2

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.16.0

ONNX Runtime API

C++

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

[edgchen1]

I try to clear the variables initialised above and assign new values to them, but after changing their values the Session.Run function throws an exception: 0xC00000000005: Access violation reading location 0x0000000000000000000000000000000000000000000000000000.

Can you find out what nullptr is being dereferenced? Is _OrtSession null at that point? You can probably determine this by running in a debugger.

Do the variables Env, RunOptions, MemoryInfo need to be created again each time the model is updated or should they be initialised once when the program is started?

Ort::Env does not need to be recreated per session. Instantiating one for the duration of the program is enough.
If the Ort::RunOptions and Ort::MemoryInfo don't change then you also don't need to recreate them. I believe there should be no harm in doing so though.

is it possible to pass Ort variables by reference? because they contain smart pointers inside them and ownership is lost?

This should be possible. Simply passing by reference, e.g., Ort::Session&, should not transfer ownership.

Note 1:
You don't need to keep a pointer to Ort::Session. Should be able to do something like this:

// initialize session the first time
Ort::Session session = Ort::Session(...);

// set it to a new session instance
session = Ort::Session(...);

There's a move assignment operator:

onnxruntime/include/onnxruntime/core/session/onnxruntime_cxx_api.h

Lines 567 to 571 in b9fd9c5

	Base& operator=(Base&& v) noexcept {
	OrtRelease(p_);
	p_ = v.release();
	return *this;
	}

Note 2:
Calling _OrtSessionOptions.release() does not free the session options object, which seems to be the intent here. It will "disown" the owned pointer and return that pointer. Its semantics are like std::unique_ptr<T>::release(). The name is unfortunately confusing given there's also OrtRelease() which actually frees the C API object pointer.

onnxruntime/include/onnxruntime/core/session/onnxruntime_cxx_api.h

Lines 575 to 581 in b9fd9c5

	/// \brief Relinquishes ownership of the contained C object pointer
	/// The underlying object is not destroyed
	contained_type* release() {
	T* p = p_;
	p_ = nullptr;
	return p;
	}

You could unset something like this:

Ort::SessionOptions session_options = Ort::SessionOptions(...);
// unset to empty, not valid to use until reset to a valid session options object
session_options = Ort::SessionOptions(nullptr);

[yuslepukhin]

The name is unfortunately confusing

@edgchen1 : 🙃 It's also quite confusing given all COM objects have a Release method which frees the object if it's the last reference. So really it's more of a "Detach" semantic. I'd gladly add [[deprecated]] to this session method to prevent future faux pas.

This is not COM. This follows C++ patterns. In theory, release() should never be called. The intent of C++ API is to cleanup everything automatically, which, of course, is far superior to COM mess.

[xeeetu]

Я пытаюсь очистить инициализированные выше переменные и присвоить им новые значения, но после изменения их значений функция Session.Run выдает исключение: 0xC00000000005: Место чтения нарушения прав доступа 0x00000000000000000000000000000000000000000000000000.

Можете ли вы узнать, какой nullptr разыменовывается? Является _OrtSessionноль в этот момент? Вероятно, вы можете определить это, запустив отладчик.

Нужно ли создавать переменные Env, RunOptions, MemoryInfo каждый раз при обновлении модели или их следует инициализировать один раз при запуске программы?

Ort::Env не нужно пересоздавать для каждого сеанса. Достаточно создать экземпляр на время работы программы. Если Ort::RunOptions и Ort::MemoryInfo не изменяются, вам также не нужно их воссоздавать. Я считаю, что в этом не должно быть никакого вреда.

можно ли передавать переменные Ort по ссылке? потому что они содержат внутри себя умные указатели и право собственности теряется?

Это должно быть возможно. Просто передача по ссылке, например, Ort::Session&, не следует передавать право собственности.

Примечание 1: Вам не нужно хранить указатель на Ort::Session. Должен быть в состоянии сделать что-то вроде этого:

// initialize session the first time
Ort::Session session = Ort::Session(...);

// set it to a new session instance
session = Ort::Session(...);

There's a move assignment operator:

onnxruntime/include/onnxruntime/core/session/onnxruntime_cxx_api.h

Lines 567 to 571 in b9fd9c5

	Base& operator=(Base&& v) noexcept {
	OrtRelease(p_);
	p_ = v.release();
	return *this;
	}

Note 2: Calling _OrtSessionOptions.release() does not free the session options object, which seems to be the intent here. It will "disown" the owned pointer and return that pointer. Its semantics are like std::unique_ptr<T>::release(). The name is unfortunately confusing given there's also OrtRelease() which actually frees the C API object pointer.

onnxruntime/include/onnxruntime/core/session/onnxruntime_cxx_api.h

Lines 575 to 581 in b9fd9c5

	/// \brief Relinquishes ownership of the contained C object pointer
	/// The underlying object is not destroyed
	contained_type* release() {
	T* p = p_;
	p_ = nullptr;
	return p;
	}

You could unset something like this:

Ort::SessionOptions session_options = Ort::SessionOptions(...);
// unset to empty, not valid to use until reset to a valid session options object
session_options = Ort::SessionOptions(nullptr);

Thanks for the reply, your notes are very helpful.