Phase 6: Report generation

examples/beluga/README.md

@@ -166,3 +166,13 @@ On top of the [SDK-wide options](../../src/lambkin/README.md#cli), this benchmar
 | `--sensor-topic` | `/scan` | Declared as an option, but not currently read inside `nominal()` — check [`beluga_benchmark.py`](beluga_benchmark.py) before relying on it. |
 
 For everything else — listing options, selecting a subset of variants, dry-running, reading results back with `lambkin.data` — see the [SDK documentation](../../src/lambkin/README.md), which applies the same way to this example as to any other benchmark.
+
+### Explore Results
+
+After the benchmark completes, open the report notebook to visualise APE results across all variants and iterations:
+
+```bash
+jupyter notebook examples/beluga/report.ipynb
+```
+
+The notebook plots APE timeseries by variant, prints a stats summary table, and generates an RMSE comparison bar chart. All figures are saved under `results/`.

examples/beluga/report.ipynb

@@ -0,0 +1,205 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "7fb27b941602401d91542211134fc71a",
+   "metadata": {},
+   "source": [
+    "# Beluga AMCL Benchmark — APE Report\n",
+    "\n",
+    "Report template for a completed Beluga benchmark run. Scoped to the outputs\n",
+    "of `beluga_benchmark.py` (`output.ape.zip`). Open from `examples/beluga/` and\n",
+    "execute all cells after a benchmark run."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "acae54e37e7d407bbb7b55eff062a284",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from collections import defaultdict\n",
+    "from pathlib import Path\n",
+    "\n",
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "\n",
+    "from lambkin.data import access\n",
+    "from lambkin.data import evo as evo_data\n",
+    "\n",
+    "# Change this path if your results live elsewhere\n",
+    "RESULTS_DIR = Path.cwd() / \"results\"\n",
+    "\n",
+    "print(RESULTS_DIR)\n",
+    "APE_FILE = \"output.ape.zip\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9a63283cbaf04dbcab1f6479b197f3a8",
+   "metadata": {},
+   "source": [
+    "## Explore available data\n",
+    "\n",
+    "Inspect what variants and iterations are available before plotting."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8dd0d8092fe74a7c96281538738b07e2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "iterations = access.iterations(RESULTS_DIR)\n",
+    "\n",
+    "print(f\"Total iterations: {len(iterations)}\")\n",
+    "print()\n",
+    "for entry in iterations:\n",
+    "    label = \", \".join(f\"{k}={v}\" for k, v in sorted(vars(entry.params).items()))\n",
+    "    print(f\"  {entry.variant} / iter {entry.iteration}  —  {label}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "72eea5119410473aa328ad9291626812",
+   "metadata": {},
+   "source": [
+    "## APE timeseries by variant\n",
+    "\n",
+    "Each variant is drawn in a distinct color. Individual iterations are shown\n",
+    "at reduced opacity; the per-variant mean is overlaid in bold."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8edb47106e1a46a883d545849b8ab81b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "series = evo_data.series(RESULTS_DIR, APE_FILE)\n",
+    "\n",
+    "by_variant = defaultdict(list)\n",
+    "for entry in series:\n",
+    "    by_variant[entry.variant].append(entry)\n",
+    "\n",
+    "fig, ax = plt.subplots(figsize=(12, 5))\n",
+    "colors = plt.rcParams[\"axes.prop_cycle\"].by_key()[\"color\"]\n",
+    "for (_variant, entries), color in zip(sorted(by_variant.items()), colors, strict=False):\n",
+    "    label = \", \".join(f\"{k}={v}\" for k, v in sorted(vars(entries[0].params).items()))\n",
+    "    for entry in entries:\n",
+    "        ax.plot(entry.time, entry.error, color=color, alpha=0.3, linewidth=0.8)\n",
+    "    t_min = max(e.time[0] for e in entries)\n",
+    "    t_max = min(e.time[-1] for e in entries)\n",
+    "    t_grid = np.linspace(t_min, t_max, 300)\n",
+    "    mean_error = np.mean([np.interp(t_grid, e.time, e.error) for e in entries], axis=0)\n",
+    "    ax.plot(t_grid, mean_error, color=color, linewidth=2, label=label)\n",
+    "\n",
+    "ax.set_xlabel(\"Time (s)\")\n",
+    "ax.set_ylabel(\"APE (m)\")\n",
+    "ax.set_title(\"Absolute Pose Error — timeseries by variant\")\n",
+    "ax.legend(loc=\"upper left\", fontsize=8)\n",
+    "fig.tight_layout()\n",
+    "plt.savefig(RESULTS_DIR / \"report_ape_series.png\", dpi=150)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "10185d26023b46108eb7d9f57d49d2b3",
+   "metadata": {},
+   "source": [
+    "## Stats summary table\n",
+    "\n",
+    "RMSE, mean, and max APE aggregated across iterations for each variant."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8763a12b2bbd4a93a75aff182afb95dc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "all_stats = evo_data.stats(RESULTS_DIR, APE_FILE)\n",
+    "\n",
+    "by_variant_stats = defaultdict(list)\n",
+    "for entry in all_stats:\n",
+    "    label = \", \".join(f\"{k}={v}\" for k, v in sorted(vars(entry.params).items()))\n",
+    "    by_variant_stats[label].append(entry)\n",
+    "\n",
+    "header = (\n",
+    "    f\"{'Variant':<40} {'N':>4}\"\n",
+    "    f\" {'RMSE mean':>10} {'RMSE std':>10} {'Mean':>10} {'Max':>10}\"\n",
+    ")\n",
+    "print(header)\n",
+    "print(\"-\" * len(header))\n",
+    "for label in sorted(by_variant_stats):\n",
+    "    entries = by_variant_stats[label]\n",
+    "    rmse = [e.rmse for e in entries]\n",
+    "    print(\n",
+    "        f\"{label:<40} {len(entries):>4}\"\n",
+    "        f\" {np.mean(rmse):>10.4f} {np.std(rmse):>10.4f}\"\n",
+    "        f\" {np.mean([e.mean for e in entries]):>10.4f}\"\n",
+    "        f\" {np.mean([e.max for e in entries]):>10.4f}\"\n",
+    "    )"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7623eae2785240b9bd12b16a66d81610",
+   "metadata": {},
+   "source": [
+    "## RMSE comparison across variants\n",
+    "\n",
+    "Bar chart comparing RMSE per variant, with error bars showing ± std across iterations."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7cdc8c89c7104fffa095e18ddfef8986",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "labels = sorted(by_variant_stats.keys())\n",
+    "rmse_means = [np.mean([e.rmse for e in by_variant_stats[lbl]]) for lbl in labels]\n",
+    "rmse_stds = [np.std([e.rmse for e in by_variant_stats[lbl]]) for lbl in labels]\n",
+    "\n",
+    "fig, ax = plt.subplots(figsize=(max(6, len(labels) * 1.2), 4))\n",
+    "x = np.arange(len(labels))\n",
+    "ax.bar(x, rmse_means, yerr=rmse_stds, capsize=4)\n",
+    "ax.set_xticks(x)\n",
+    "ax.set_xticklabels(labels, rotation=25, ha=\"right\", fontsize=8)\n",
+    "ax.set_ylabel(\"RMSE (m)\")\n",
+    "ax.set_title(\"APE RMSE by variant (mean ± std across iterations)\")\n",
+    "fig.tight_layout()\n",
+    "plt.savefig(RESULTS_DIR / \"report_rmse_bars.png\", dpi=150)\n",
+    "plt.show()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

src/lambkin/README.md

@@ -14,7 +14,7 @@ The LAMBKIN Python SDK is the core library for building SLAM evaluation pipeline
 - [Logging](#logging)
 - [Results](#results)
   - [Metrics](#metrics)
-  - [Output Hooks](#output-hooks)
+  - [Report Generation](#report-generation)
   - [Reprocessing](#reprocessing)
 - [Cookbook](#cookbook)
 
@@ -304,7 +304,6 @@ Precedence (highest to lowest):
 - **`lambkin.data.evo.series(source, filename)`** — same traversal, plus loads the `evo` result file (e.g. `"output.ape.zip"`) from each iteration directory and exposes `time`, `error`, and `distance` arrays, ready to plot.
 - **`lambkin.data.evo.stats(source, filename)`** — same traversal, but exposes the aggregate statistics `evo` computes for each result: `rmse`, `mean`, `median`, `std`, `min`, `max`, `sse`.
 
-
 ### Metrics
 
 LAMBKIN doesn't compute trajectory metrics itself — it invokes `evo` through `ctx.shell`, the same way it invokes any other external process, and reads back whatever `evo` writes to disk. The field names exposed by `lambkin.data.evo` (`rmse`, `mean`, `median`, `std`, `min`, `max`, `sse`) are `evo`'s own, not LAMBKIN's.
@@ -410,3 +409,46 @@ def nominal(ctx):
 
 > [!NOTE]
 > `ShellProxy` converts keyword argument underscores to dashes (`save_as_tum=` → `--save-as-tum`), which `evo` won't recognize. Always pass `evo` flags that contain underscores as positional strings, as shown above.
+
+### Analysing results in a notebook
+
+`lambkin.data` functions accept a plain path, so results can be explored
+from a Jupyter notebook without re-running the benchmark:
+
+```python
+from collections import defaultdict
+
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+
+from lambkin.data import access, evo as evo_data
+
+RESULTS_DIR = "path/to/results"
+
+# List completed iterations
+for entry in access.iterations(RESULTS_DIR):
+    print(entry.variant, entry.iteration, vars(entry.params))
+
+# Plot APE timeseries per variant
+series = evo_data.series(RESULTS_DIR, "output.ape.zip")
+for entry in series:
+    plt.plot(entry.time, entry.error, label=entry.variant, alpha=0.5)
+plt.legend()
+plt.show()
+
+# Convert to a long-format DataFrame for seaborn
+rows = []
+for entry in evo_data.stats(RESULTS_DIR, "output.ape.zip"):
+    row = vars(entry.params).copy()
+    row.update({"variant": entry.variant, "iteration": entry.iteration,
+                "rmse": entry.rmse, "mean": entry.mean, "max": entry.max})
+    rows.append(row)
+df = pd.DataFrame(rows)
+```
+
+Export a notebook to HTML to share it without requiring Jupyter:
+
+```bash
+jupyter nbconvert --to html report.ipynb
+```