I want to understand the allocations made on STM32s by Executorch

[ChristophKarlHeck]

📚 The doc issue

@tarun292, @kimishpatel, yesterday's issue was closed, so I opened a new one.
So here is the continuing of the #9447

There are two sections in the exeutor_runner.cpp that make heap allocations:

1. m_method_allocator_pool = (uint8_t*)malloc(m_allocator_pool_size);
2. planned_buffers.push_back(std::make_unique<uint8_t[]>(buffer_size));

The size for the second one is precise (33168). I get it from the execution plan.

executorch_program: exir.ExecutorchProgramManager = edge_program.to_executorch(
    ExecutorchBackendConfig(
        passes=[],  # User-defined passes
    )
)
print(executorch_program.executorch_program.execution_plan[0].non_const_buffer_sizes)

The first one is unclear yet. Since the CNN execution on the STM32 works with a m_allocator_pool_size of 8192 but not with a size of 4096, allocating 33168bytes is pointless since I want to save as much SRAM as possible.
I assume the method size is between 4096 and 8192, but where do I get the exact size? Since the error I get when I set `m_allocator_pool_size to 4096 is:

E executorch:memory_allocator.h:87] Memory allocation failed: 48B requested (adjusted for alignment), 44B available
I executorch:ModelExecutor.cpp:283] Loading of method forward failed with status 0x21

It happens in this code section:

Result<torch::executor::Method> method = program->load_method(method_name, &memory_manager);
if (!method.ok()) {
	ET_LOG(
		Info,
		"Loading of method %s failed with status 0x%" PRIx32,
		method_name,
		method.error());
}

Thank you for being so helpful in advance!
Christoph

cc @larryliu0820 @JacobSzwejbka

[kimishpatel]

Oh I am sorry. I closed because i thought it resolved the original issue.

[kimishpatel]

I assume the method size is between 4096 and 8192, but where do I get the exact size? Since the error I get when I set `m_allocator_pool_size to 4096 is:

This is a good question. @JacobSzwejbka do you have answer to this? As in for method allocator what is the size of method allocator pool?

[ChristophKarlHeck]

Understanding how memory planning influences the model's size, such as which nodes may overlap and which don't, would also be beneficial. Then, we can optimize the model's final architecture according to the size of the STM32.

Also, it would be interesting to know what the method_allocator is used for, for instance, the weights... Or what else?

[tarun292]

@ChristophKarlHeck the method allocator pool isn't used for weights, but it's mainly used to allocate memory for the internal data structures needed by ExecuTorch. Unfortunately for method allocator pool size we don't have a way of generating this number ahead of time.

If you're interested in understanding how memory planning works then you can refer to the code here which is quite extensively documented https://github.com/pytorch/executorch/blob/main/exir/memory_planning.py#L707

You can also write your own memory planning algo if you wish to do so.

[kimishpatel]

Understanding how memory planning influences the model's size, such as which nodes may overlap and which don't, would also be beneficial. Then, we can optimize the model's final architecture according to the size of the STM32.

For this see https://github.com/pytorch/executorch/blob/main/util/activation_memory_profiler.py

[ChristophKarlHeck]

@kimishpatel @tarun292 Thank you very much. I'll have a look at it :)