Merge pull request #4 from raws-labs/hardening/benchmark-provenance

Harden Cortex benchmark provenance

Author: asteinh

cortex-m-deployability/BUILD.md

@@ -39,7 +39,10 @@ BENCH_MODELS=ds_cnn BENCH_CONFIGS=cmsis_nn scripts/run_all.sh f446   # one cell
 This vendors CMSIS (first run), builds each (board, model, config), flashes +
 captures each on the SiliconRig remote lab until `BENCH_DONE`, writes
 `results/summary.json`, and runs the device-to-device parity gate. Raw serial logs
-land in `results/raw/` (gitignored; `summary.json` is the tracked artifact).
+land in `results/raw/` (gitignored; `summary.json` is the tracked artifact). Each
+raw capture also carries a machine-readable provenance record for the exact
+repositories, dependencies, tools, build invocation, model, firmware, and
+allocated board. Collection fails if that evidence is missing or incomplete.
 
 ## Manual steps
 
@@ -118,6 +121,7 @@ cat /dev/ttyACM0 > results/raw/pico_ts_cmsis.log   # stop at BENCH_DONE
 | `BENCH_OPT_LEVEL` | `-O2` | optimization for ALL benchmark code; consistent across boards + matches TFLM's `-O2` kernels (the clean-CMake default would otherwise be `-O0`) |
 | `TIGRIS_PLAN` | `ds_cnn_i8.tgrs` | the `.tgrs` plan embedded in flash (compile at a budget that fits the board) |
 | `TIGRIS_CODEGEN` | sibling TiGrIS venv CLI | `tigris` executable used to generate the backend deployment core during the build |
+| `TIGRIS_RUNTIME_ROOT` | sibling `tigris-runtime` | exact runtime checkout compiled into the firmware; `run_all.sh` binds this to the checkout it validates and records |
 | `TIGRIS_FAST_ARENA_BYTES` | `131072` | static fast arena backing store; the harness provisions only a tight slice of it (`peak + align`) so the compactor engages and the measured RAM is the true minimum |
 | `TIGRIS_SLOW_ARENA_BYTES` | `262144` | slow arena backing store; use `8192` on the F446 |
 | `TFLM_MODEL` | `ds_cnn` | TFLM model: `<name>_tflite_i8.h` in `MODELS_DIR` |
@@ -135,14 +139,15 @@ cat /dev/ttyACM0 > results/raw/pico_ts_cmsis.log   # stop at BENCH_DONE
   as its DWT under-counts XIP-stall cycles.)
 - **RAM metric:** `sram_peak_bytes` is the MEASURED runtime working set, directly
   comparable to TFLM's `arena_used_bytes`. For TiGrIS it is the fast + slow arena
-  high-water + CMSIS-NN scratch + the runtime tensor table; the harness provisions
-  a tight arena (`peak + alignment`) so the executor's reactive compactor engages
-  and the figure is the true minimum, not a lazy bump high-water. (Provision a
-  generous arena and the same model reports far more RAM - that is a measurement
-  artifact, not a real cost.)
-- **Vendor pins** (`fetch.sh`): CMSIS-NN `6d9d61d8` (full SHA, asserted after
-  checkout; matches TFLM's bundled pin), cmsis-device-h7 `master`, cmsis-device-f4
-  `3c77349`. Override via `DEV_*_REF`.
+  high-water + CMSIS-NN scratch + the runtime tensor table + the caller-owned
+  executor workspace; the harness provisions a tight arena (`peak + alignment`)
+  so the executor's reactive compactor engages and the figure is the true
+  minimum, not a lazy bump high-water. (Provision a generous arena and the same
+  model reports far more RAM - that is a measurement artifact, not a real cost.)
+- **Vendor pins** (`fetch.sh`): CMSIS-NN `6d9d61d8` (matches TFLM's bundled
+  pin), CMSIS-Core `45dab712`, cmsis-device-h7 `de8243d2`,
+  cmsis-device-f4 `3c77349`, and TFLM `074b75f8`. Every checkout is asserted
+  against its full commit. RP2350 runs likewise assert pico-sdk `a1438dff`.
 - **Parity** is checked device-to-device by `scripts/validate_accuracy.py` (reads
   `results/summary.json` by default, which carries each cell's `OUTPUT_I8`; pass a
   `results/raw/` dir to read logs instead). It groups by (board, model) and compares

cortex-m-deployability/CMakeLists.txt

@@ -13,6 +13,8 @@ set(TIGRIS_FAST_ARENA_BYTES "131072" CACHE STRING "fast arena size (bytes)")
 set(TIGRIS_SLOW_ARENA_BYTES "262144" CACHE STRING "slow arena size (bytes)")
 set(TIGRIS_CODEGEN "${CMAKE_CURRENT_SOURCE_DIR}/../../tigris/.venv/bin/tigris"
     CACHE FILEPATH "TiGrIS codegen executable")
+set(TIGRIS_RUNTIME_ROOT "${CMAKE_CURRENT_SOURCE_DIR}/../../tigris-runtime"
+    CACHE PATH "pinned TiGrIS runtime checkout")
 # Optimization level for ALL benchmark code (TiGrIS runtime, CMSIS-NN, harness, TFLM
 # glue). MUST be consistent across boards and frameworks for a fair comparison: TFLM's
 # own kernels build at -O2, and without this the clean-CMake path defaults to -O0 (no
@@ -50,7 +52,7 @@ if(BENCH_FRAMEWORK STREQUAL "tigris")
     add_subdirectory(third_party)                          # cmsis-nn static lib
 
     set(TIGRIS_HAS_CMSIS_NN ON CACHE BOOL "" FORCE)
-    add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/../../tigris-runtime
+    add_subdirectory(${TIGRIS_RUNTIME_ROOT}
                      ${CMAKE_BINARY_DIR}/tigris_runtime)
 
     if(BENCH_KERNEL STREQUAL "cmsis_nn")

cortex-m-deployability/README.md

@@ -23,9 +23,10 @@ ties differently from on-device CMSIS-NN, so it is not used as a baseline. All
 boards build at `-O2`; each runs at its rated clock, verified per run by a clock
 guard in `results.py`. Latency is the median of 30 runs (DWT cycle counter on
 M7/M4, hardware `time_us` on M33). RAM is the activation working set: TiGrIS
-`sram_peak` (fast-arena peak + slow-pool spill + scratch + tensor table) against
-TFLM `arena_used`; weights live in flash and are excluded, as is stack. Captured
-remotely on a SiliconRig hardware-in-the-loop lab via `scripts/run_all.sh`.
+`sram_peak` (fast-arena peak + slow-pool spill + scratch + tensor table +
+executor workspace) against TFLM `arena_used`; weights live in flash and are
+excluded, as is stack. Captured remotely on a SiliconRig hardware-in-the-loop
+lab via `scripts/run_all.sh`.
 
 For each TiGrIS cell, CMake runs `tigris codegen <plan> --backend
 {cmsis-nn|reference} --format core`. The generated, backend-specific deployment
@@ -46,44 +47,42 @@ int8 weights plus biases, measured from the committed plans.
 
 | Framework | Kernel | Latency | Cycles | RAM (work. set) | Flash (firmware) |
 |---|---|---|---|---|---|
-| TiGrIS | cmsis_nn | 11.21 ms | 5.38 M | 16.8 KB | 108 KB |
-| TFLM | cmsis_nn | 12.80 ms | 6.14 M | 22.2 KB | 166 KB |
-| TiGrIS | s8_ref | 63.12 ms | 30.30 M | 16.7 KB | 84 KB |
+| TiGrIS | cmsis_nn | 11.14 ms | 5.35 M | 25.5 KB | 118 KB |
+| TFLM | cmsis_nn | 12.80 ms | 6.14 M | 22.2 KB | 176 KB |
+| TiGrIS | s8_ref | 81.96 ms | 39.34 M | 25.3 KB | 93 KB |
 
 **Anomaly detection:**
 
 | Framework | Kernel | Latency | Cycles | RAM (work. set) | Flash (firmware) |
 |---|---|---|---|---|---|
-| TiGrIS | cmsis_nn | 1.19 ms | 569 K | 2.7 KB | 361 KB |
-| TFLM | cmsis_nn | 1.16 ms | 558 K | 15.5 KB | 408 KB |
-| TiGrIS | s8_ref | 3.02 ms | 1.45 M | 2.7 KB | 337 KB |
+| TiGrIS | cmsis_nn | 1.19 ms | 570 K | 11.6 KB | 370 KB |
+| TFLM | cmsis_nn | 1.16 ms | 558 K | 15.5 KB | 417 KB |
+| TiGrIS | s8_ref | 3.05 ms | 1.46 M | 11.4 KB | 346 KB |
 
 **Timeseries:**
 
 | Framework | Kernel | Latency | Cycles | RAM (work. set) | Flash (firmware) |
 |---|---|---|---|---|---|
-| TiGrIS | cmsis_nn | 0.306 ms | 147 K | 1.9 KB | 77 KB |
-| TFLM | cmsis_nn | 0.345 ms | 166 K | 2.9 KB | 135 KB |
-| TiGrIS | s8_ref | 1.13 ms | 544 K | 2.0 KB | 53 KB |
+| TiGrIS | cmsis_nn | 0.298 ms | 143 K | 11.4 KB | 87 KB |
+| TFLM | cmsis_nn | 0.345 ms | 166 K | 2.9 KB | 145 KB |
+| TiGrIS | s8_ref | 1.61 ms | 774 K | 10.7 KB | 62 KB |
 
 - Output is bit-exact device-to-device: every (model, framework, kernel) cell
   emits the identical INT8 vector (max abs diff 0), checked by
   `scripts/validate_accuracy.py`.
-- CMSIS-NN vs the portable reference kernels (same model, both `-O2`): 5.6x
-  (DS-CNN) / 2.5x (AD) / 3.7x (TS) on the M7.
+- CMSIS-NN vs the portable reference kernels (same model, both `-O2`): 7.4x
+  (DS-CNN) / 2.6x (AD) / 5.4x (TS) on the M7.
 - Cycles are clock-independent; ms is at 480 MHz.
-- Latency, cycles, and RAM come from the tracked `results/summary.json`. The
-  flash column is a build-time `arm-none-eabi-size` measurement that
-  `summary.json` does not carry, so it is not refreshed by a rerun; these values
-  are from the 2026-06-27 rig build and predate the codegen-core harness.
+- Latency, cycles, RAM, and firmware sizes come from the same tracked,
+  provenance-bearing run in `results/summary.json`.
 
 ## NUCLEO-F446RE (Cortex-M4F @ 180 MHz)
 
 | Model | TiGrIS cmsis | TFLM cmsis | TiGrIS s8 | RAM (TiGrIS / TFLM) |
 |---|---|---|---|---|
-| TS | 1.56 ms | 1.80 ms | 11.00 ms | 1.9 / 2.9 KB |
-| AD | 4.97 ms | 4.82 ms | 15.11 ms | 2.7 / 15.5 KB |
-| DS-CNN | 59.90 ms | 68.19 ms | 513.20 ms | 16.8 / 22.2 KB |
+| TS | 1.56 ms | 1.80 ms | 8.41 ms | 11.4 / 2.9 KB |
+| AD | 4.97 ms | 4.82 ms | 16.53 ms | 11.6 / 15.5 KB |
+| DS-CNN | 63.53 ms | 68.19 ms | 467.58 ms | 25.5 / 22.2 KB |
 
 Output is byte-identical to the H753 (same weights, two architectures). The
 128 KB SRAM holds every model.
@@ -96,11 +95,11 @@ byte-identical to the H753 and F446. Weights are read from QSPI flash via XIP.
 
 | Model | TiGrIS cmsis | TiGrIS s8 | RAM |
 |---|---|---|---|
-| TS | 2.73 ms | 9.02 ms | 1.9 KB |
-| AD | 35.08 ms | 44.52 ms | 2.7 KB |
-| DS-CNN | 62.94 ms | 412.77 ms | 16.8 KB |
+| TS | 2.68 ms | 8.37 ms | 11.4 KB |
+| AD | 35.01 ms | 44.56 ms | 11.6 KB |
+| DS-CNN | 67.83 ms | 412.06 ms | 25.5 KB |
 
-The FC-heavy AD is slower here (35.08 ms vs 4.97 ms on the F446): each of its
+The FC-heavy AD is slower here (35.01 ms vs 4.97 ms on the F446): each of its
 265 KB of weights is read once per inference from XIP flash with no reuse, so it
 is QSPI-bandwidth-bound. The conv models reuse weights across spatial positions
 and stay fast.
@@ -109,39 +108,41 @@ and stay fast.
 
 MobileNetV2 (alpha 0.35, 224x224, INT8, 591 KB weights, 52 convs with
 inverted-residual ADD skips) has a naive activation peak of 735 KB, larger than
-any of these boards' SRAM. TiGrIS tiles it to a 300 KB working set (129 KB fast +
-172 KB slow-pool spill, 2 tiled stages), with bit-exact output across boards.
+any of these boards' SRAM. TiGrIS tiles it to a 307.7 KB working set (127.1 KB
+fast + 171.5 KB slow-pool spill + 9.1 KB scratch/runtime metadata, 2 tiled
+stages), with bit-exact output across boards.
 
 | Board (SRAM) | TiGrIS (tiled) | TFLM (no tiling) |
 |---|---|---|
-| H753ZI (512 KB) | runs, 1.24 s, 300 KB | OOM at AllocateTensors |
-| RP2350 (520 KB) | runs, 6.55 s, 300 KB | n/a (no M33 lib) |
+| H753ZI (512 KB) | runs, 1.43 s, 307.7 KB | OOM at AllocateTensors |
+| RP2350 (520 KB) | runs, 7.03 s, 307.7 KB | n/a (no M33 lib) |
 | F446RE (128 KB) | does not fit | does not fit |
 
 - On the H753, TFLM given a 480 KB arena (nearly all of the 512 KB SRAM) fails
   `AllocateTensors` with `ARENA_TOO_SMALL`: with no tiling it needs the full
   735 KB. TiGrIS runs the identical model on the same board.
 - The F446 cannot hold MobileNetV2: the 591 KB weight blob exceeds its 512 KB
   flash, and the 300 KB tiled working set exceeds its 128 KB SRAM.
-- RP2350 is ~5.3x slower than the H753 on this model (XIP-bound: 591 KB of
+- RP2350 is ~4.9x slower than the H753 on this model (XIP-bound: 591 KB of
   weights streamed from QSPI flash each inference, plus the lower clock).
 
 ## Reproduce
 
 `SRIG_API_KEY=... ./scripts/run_all.sh` builds every cell, flashes and captures
 on the SiliconRig lab, aggregates, and runs the parity gate. Raw serial logs land
 in `results/raw/` (gitignored); the tracked artifact is `results/summary.json`,
-which carries each cell's numbers and `OUTPUT_I8` vector. See `BUILD.md` for the
-build knobs and a locally-attached-board (no-rig) path.
+which carries each cell's numbers, `OUTPUT_I8` vector, and deduplicated execution
+provenance. See `BUILD.md` for the build knobs and a locally-attached-board
+(no-rig) path.
 
 `results/provenance.json` binds that summary and its collector/validator to
-SHA-256 digests, records the paths and hashes of all 27 source captures, and
-distinguishes declared dependency pins from unavailable execution provenance.
-The captures are not shipped in a clone, but their hashes are retained so an
-obtained capture set can be checked. Fields marked `unknown` are intentionally
-unresolved rather than reconstructed from commit dates or current sibling
-checkouts. Host validation always checks tracked artifact/tool hashes and also
-checks source-capture hashes when those gitignored files are present. When the
-complete capture set is available, it reruns the collector in a temporary
-directory and requires byte-identical `summary.json` output; a clean clone
-skips that reconstruction and does not report it as verified.
+SHA-256 digests and records the paths and hashes of all 27 source captures. The
+summary embeds the actual compiler, runtime, TFLM, CMSIS and SDK revisions;
+tool and Python-environment versions; build invocations; model and firmware
+hashes/sizes; capture timestamps; and SiliconRig board identities. The captures
+are not shipped in a clone, but their hashes are retained so an obtained capture
+set can be checked. Host validation always checks tracked artifact/tool hashes
+and also checks source-capture hashes when those gitignored files are present.
+When the complete capture set is available, it reruns the collector in a
+temporary directory and requires byte-identical `summary.json` output; a clean
+clone skips that reconstruction and does not report it as verified.

cortex-m-deployability/boards/pico2_rp2350/CMakeLists.txt

@@ -23,7 +23,9 @@ project(tigris_pico_bench C CXX ASM)
 pico_sdk_init()
 
 set(BENCH_ROOT ${CMAKE_CURRENT_LIST_DIR}/../..)
-set(RT_DIR     ${BENCH_ROOT}/../../tigris-runtime)
+set(TIGRIS_RUNTIME_ROOT "${BENCH_ROOT}/../../tigris-runtime"
+    CACHE PATH "pinned TiGrIS runtime checkout")
+set(RT_DIR     ${TIGRIS_RUNTIME_ROOT})
 set(NN_DIR     ${BENCH_ROOT}/third_party/CMSIS-NN)
 
 set(BENCH_KERNEL "cmsis_nn" CACHE STRING "TiGrIS backend: cmsis_nn | s8_ref")

cortex-m-deployability/harness/main.c

@@ -63,6 +63,7 @@ extern const unsigned int  g_tigris_plan_len;
 static uint8_t s_fast_arena[TIGRIS_FAST_ARENA_BYTES] __attribute__((aligned(16)));
 static uint8_t s_slow_arena[TIGRIS_SLOW_ARENA_BYTES] __attribute__((aligned(16)));
 static void   *s_tensor_ptrs[BENCH_MAX_TENSORS];
+static tigris_executor_workspace_t s_executor_workspace;
 
 static const char *kernel_name(void)
 {
@@ -130,10 +131,12 @@ static uint32_t run_once(tigris_plan_t *plan, tigris_mem_t *mem,
     tigris_exec_stats_t stats;
     uint32_t c0 = platform_cycles();
 #ifdef BENCH_PROFILE_OPS
-    tigris_exec_error_t err = tigris_run(plan, mem, dispatch, NULL, &stats);
+    tigris_exec_error_t err = tigris_run_with_workspace(
+        plan, mem, dispatch, NULL, &stats, &s_executor_workspace);
 #else
     (void)dispatch;  /* generated core owns the normal dispatch call. */
-    tigris_exec_error_t err = tigris_codegen_run(plan, mem, &stats);
+    tigris_exec_error_t err = tigris_codegen_run(
+        plan, mem, &stats, &s_executor_workspace);
 #endif
     uint32_t c1 = platform_cycles();
 
@@ -268,7 +271,14 @@ int main(void)
     } else {
         base = TIGRIS_FAST_ARENA_BYTES;
     }
-    uint32_t fast_size = base + align_reserve + tigris_weight_decompression_overhead(&plan);
+    uint32_t fast_size = base + align_reserve
+        + tigris_weight_decompression_overhead(&plan);
+#if defined(BENCH_KERNEL_CMSIS_NN)
+    /* The plan budget covers activations and transient weights, not backend
+     * scratch. CMSIS-NN reserves its exact scratch requirement from the same
+     * physical backing store before inference. */
+    fast_size = tigris_cmsis_nn_fast_arena_required(&plan);
+#endif
 #ifdef TIGRIS_FAST_OVERRIDE
     /* Diagnostic: force a specific fast-arena size to probe the runtime minimum. */
     if (TIGRIS_FAST_OVERRIDE)
@@ -400,13 +410,16 @@ int main(void)
      *     framework needs in SRAM at runtime, not a compile-time estimate.
      *       act_peak  = fast + slow arena high-water (the live activation set)
      *       scratch   = CMSIS-NN scratch carved from the fast arena (0 for s8_ref)
-     *       meta      = the runtime tensor-pointer table (TiGrIS's RAM metadata;
-     *                   TFLM keeps its equivalent TfLiteEvalTensor table inside
-     *                   the arena, so arena_used already counts it - we add ours)
+     *       meta      = the runtime tensor-pointer table plus caller-owned
+     *                   executor workspace (TFLM keeps equivalent metadata
+     *                   inside its arena, so arena_used already counts it)
      *     plan.header->peak (the old reported value) is the compiler's compile-
      *     time activation estimate; kept as plan_act_peak_bytes for reference. */
     uint32_t scratch_bytes = fast_size - mem.fast_size;     /* prepare() carve; 0 if none */
-    uint32_t meta_bytes    = (uint32_t)plan.header->num_tensors * (uint32_t)sizeof(void *);
+    uint32_t tensor_table_bytes =
+        (uint32_t)plan.header->num_tensors * (uint32_t)sizeof(void *);
+    uint32_t executor_workspace_bytes = (uint32_t)sizeof(s_executor_workspace);
+    uint32_t meta_bytes = tensor_table_bytes + executor_workspace_bytes;
     uint32_t fast_peak     = mem.fast_peak;
     uint32_t slow_peak     = last_stats.slow_peak;
     uint32_t act_peak      = fast_peak + slow_peak;
@@ -439,6 +452,7 @@ int main(void)
            "sram_slow_peak_bytes=%lu,"
            "sram_scratch_bytes=%lu,"
            "sram_meta_bytes=%lu,"
+           "sram_executor_workspace_bytes=%lu,"
            "plan_act_peak_bytes=%lu,"
            "sram_budget_kb=%lu,"
            "sram_provisioned_kb=%lu,"
@@ -467,6 +481,7 @@ int main(void)
            (unsigned long)slow_peak,
            (unsigned long)scratch_bytes,
            (unsigned long)meta_bytes,
+           (unsigned long)executor_workspace_bytes,
            (unsigned long)plan.header->peak,
            (unsigned long)(plan.header->budget / 1024),
            (unsigned long)(fast_size / 1024),