Summary
On MCU-class targets each CPU typically has its own tightly-coupled or local memory (TCM, DSPR, local RAM) with deterministic, zero-wait-state access latency. An important optimization for real-time workloads would be to place each CPU's struct cpu and the struct vcpus it runs in that CPU's local memory, eliminating cache variability from the hot path of every context switch, interrupt, and scheduler invocation.
Motivation
ARM explicitly recommends placing "call stack, RTOS control structures, interrupt data structures, and interrupt handler code" in TCM for deterministic latency [ARM TCM documentation]. The Cortex-R52 TCM offers zero-wait-state access [Cortex-R52 TRM], and Infineon describes DSPR as "high-speed, single-cycle SRAM core-private memory intended for stacks, local variables, and non-shared data to achieve maximum performance with deterministic access timing". Placing struct cpu and the associated struct vcpu in the local memory of the CPU that owns them would:
- Eliminate cache miss variability on every
cpu() dereference and vcpu context switch
- Improve VM exit and entry latencies, since guest state is saved to and restored from these low-latency memories on every world switch
- Improve worst-case interrupt latency bounds — critical for mixed-criticality systems
- Reduce cross-core memory traffic on shared buses
Current state
Several platforms already describe their per-CPU local memories in the platform descriptor as generic mem_region entries, but no mechanism exists to direct allocations there. struct cpu is always allocated linearly from _dmem_beg in global memory, regardless of available local memory.
- TriCore TC4DX — DSPR per core (240 KB each) described in
src/platform/tc4dx/tc4dx_desc.c, but _dmem_beg follows _image_end in global LMU memory; DSPR is unused by the hypervisor
- RH850 U2A16 — Local RAM per core (64 KB each) described in
src/platform/rh850-u2a16/u2a16_desc.c
- NXP S32Z270 — ATCM/BTCM/CTCM per core described in
src/platform/s32z270/s32z270_desc.c [S32Z2 datasheet]
The struct platform and all arch_platform structs have no dedicated per-CPU memory region fields, and the boot/allocation paths make no use of local memories.
Additionally, all struct vcpus belonging to a VM are currently allocated as a single contiguous block alongside the struct vm in src/core/vmm.c. This prevents each vcpu from being placed in the local memory of the CPU that will run it, as all vcpus are committed to one memory region at VM allocation time regardless of their CPU assignment.
Proposed design
Extend the platform description with a per-CPU local memory field — either in struct platform or arch_platform — that maps a CPU id to a memory region. During boot, each CPU would allocate its struct cpu (and ideally the struct vcpus pinned to it) from that region instead of the global heap. The latter would also require decoupling vcpu allocation from the monolithic VM struct allocation.
This should ideally also be configurable in the config file, allowing a system integrator to override or refine placement (e.g. for asymmetric TCM sizes or shared-TCM setups).
Affected platforms
TriCore (TC4DX), RH850 (U2A16), NXP S32Z270, ARMv8-R Cortex-R52 (MPS3-AN536, FVP-R, S32Z270, E3650)
Summary
On MCU-class targets each CPU typically has its own tightly-coupled or local memory (TCM, DSPR, local RAM) with deterministic, zero-wait-state access latency. An important optimization for real-time workloads would be to place each CPU's
struct cpuand thestruct vcpus it runs in that CPU's local memory, eliminating cache variability from the hot path of every context switch, interrupt, and scheduler invocation.Motivation
ARM explicitly recommends placing "call stack, RTOS control structures, interrupt data structures, and interrupt handler code" in TCM for deterministic latency [ARM TCM documentation]. The Cortex-R52 TCM offers zero-wait-state access [Cortex-R52 TRM], and Infineon describes DSPR as "high-speed, single-cycle SRAM core-private memory intended for stacks, local variables, and non-shared data to achieve maximum performance with deterministic access timing". Placing
struct cpuand the associatedstruct vcpuin the local memory of the CPU that owns them would:cpu()dereference and vcpu context switchCurrent state
Several platforms already describe their per-CPU local memories in the platform descriptor as generic
mem_regionentries, but no mechanism exists to direct allocations there.struct cpuis always allocated linearly from_dmem_begin global memory, regardless of available local memory.src/platform/tc4dx/tc4dx_desc.c, but_dmem_begfollows_image_endin global LMU memory; DSPR is unused by the hypervisorsrc/platform/rh850-u2a16/u2a16_desc.csrc/platform/s32z270/s32z270_desc.c[S32Z2 datasheet]The
struct platformand allarch_platformstructs have no dedicated per-CPU memory region fields, and the boot/allocation paths make no use of local memories.Additionally, all
struct vcpus belonging to a VM are currently allocated as a single contiguous block alongside thestruct vminsrc/core/vmm.c. This prevents each vcpu from being placed in the local memory of the CPU that will run it, as all vcpus are committed to one memory region at VM allocation time regardless of their CPU assignment.Proposed design
Extend the platform description with a per-CPU local memory field — either in
struct platformorarch_platform— that maps a CPU id to a memory region. During boot, each CPU would allocate itsstruct cpu(and ideally thestruct vcpus pinned to it) from that region instead of the global heap. The latter would also require decoupling vcpu allocation from the monolithic VM struct allocation.This should ideally also be configurable in the config file, allowing a system integrator to override or refine placement (e.g. for asymmetric TCM sizes or shared-TCM setups).
Affected platforms
TriCore (TC4DX), RH850 (U2A16), NXP S32Z270, ARMv8-R Cortex-R52 (MPS3-AN536, FVP-R, S32Z270, E3650)