Explore leveraging ITCM better (code and/or weights) — currently ~0% utilized

[apage224]

Summary

On AP5 boards, ITCM (Instruction TCM) is essentially unused today: for a
KWS model built with the heliaAOT engine, the linked binary places only
28 bytes into .itcm_text (a single AmbiqSuite HAL helper,
am_hal_utils.o), out of a 256 KB capacity — confirmed identical on
HPX's own build and on AutoDeploy's real kws_ref_model_aligned_power_aot.axf
(byte-for-byte same 28-byte section, same object). That's free, currently
unused silicon capability (0-wait-state instruction memory) that HPX
could exploit to show Ambiq silicon in the best light, independent of any
comparison to other tools.

What's been tried so far (2026-07-03)

1. heliaAOT has no code-placement capability today

Checked the installed helia_aot package (v0.18.0) source directly.
Its MemoryType enum (platforms/defines.py) includes ITCM and even
has an .executable property, but nothing in the package ever reads
.executable or exposes a per-operator/per-kernel "code memory" config —
PUT_IN_* qualifiers are only ever generated for tensors (weights /
scratch / persistent), never for operator/kernel code. Any ITCM code
placement has to happen at the linker level today, not via heliaAOT
config.

2. Linker-level prototype: placed hot CMSIS-NN kernels in ITCM — mixed/inconclusive results

Per-layer profiling for the KWS reference model shows CONV_2D +
DEPTHWISE_CONV_2D account for 99.1% of all cycles (58.3% / 40.8%);
FULLY_CONNECTED and SOFTMAX are negligible (0.03% / 0.1%).

Extended the NSX linker script's existing .itcm_text output section
(already has a *(.itcm_text*) wildcard) with explicit
*(.text.<kernel>) rules targeting the actual CMSIS-NN kernel functions
GCC's -ffunction-sections splits out per-function, tested two
iterations:

• Attempt 1 (wrong kernel set — included arm_softmax_s8 /
  arm_nn_vec_mat_mult_t_s8, missed the real depthwise inner loop):
  avg power 7.205 mW vs baseline 7.229-7.230 mW (~0.3-0.4% lower, small
  but repeatable across 2 runs).
• Attempt 2 (corrected kernel set, including the actual hot inner
  kernels arm_nn_depthwise_conv_nt_t_s8 and the 1x1-conv dispatcher
  arm_convolve_1x1_s8_fast): avg power 7.250-7.281 mW across 3 runs —
  flat to slightly worse than baseline, worse than attempt 1.

Working theory: partial placement can backfire — when a hot inner
function moves to ITCM but its caller/callees stay in MRAM, every call
now crosses memory regions, and that overhead can offset or exceed the
fetch-latency win, especially since AP5's MRAM cache may already serve
these small, tight, frequently-reused loops nearly as well as ITCM would.
Not a clean win as tested; results are noisy (~1-2% run-to-run even
within one binary) and inconsistent between the two attempts.

Open questions / ideas for a real follow-up investigation

1. Whole call-chain placement, not cherry-picked functions. Moving an
   entire hot call subtree (wrapper + inner kernel + any small helpers it
   calls) together, rather than a subset, to avoid the cross-region-jump
   penalty theorized above.
2. Weights/constants in ITCM, not just code (this issue's title scope,
   per user request). heliaAOT's MemoryType.ITCM already exists in its
   tensor-placement enum, so aot_args.memory tensor overrides pointing a
   weight tensor at ITCM may already be mechanically possible — untested.
   Open question to verify first: on Cortex-M55/AP5, is ITCM actually
   accessible from the D-side (load/store) bus at all, and if so, at what
   relative performance/power vs DTCM? If ITCM is I-side-only in this
   silicon's bus fabric, this idea is moot; if it's dual-accessible, put a
   hot, small, frequently-reused constant (e.g. quantization params, small
   bias vectors) there as a first test.
3. Proper before/after methodology: more repeats (5-10+) per
   configuration to separate real effect from the observed ~1-2% noise
   floor, and ideally a per-layer PMU comparison (cycles, not just gated
   power) to see whether ITCM placement changes cycle count at all before
   looking at power.
4. Consider whether this is better pursued as a heliaAOT feature request
   (compiler emits PUT_IN_ITCM-annotated code for identified hot ops)
   rather than an HPX-side linker hack — would generalize across models
   automatically using the AOT compiler's own graph/cost knowledge,
   instead of HPX guessing per-model which functions are hot.

Where this was found / prototyped

Repo memory: ap510-autodeploy-power-methodology-gap-resolved.md (ITCM
sections, 2026-07-03). Prototype linker scripts and measurements are not
committed anywhere (ad hoc /tmp scratch files from an interactive
session) — a real implementation would need to live in
src/helia_profiler/firmware/templates/CMakeLists.txt.j2 /
a new linker-overlay-generation mechanism, likely keyed off the
per-layer PMU profile or AOT operator manifest to auto-identify hot
kernels rather than hardcoding function names.