feat(install): observability stack provisioning (InfluxDB + Grafana) — co-located and separate-host

[CameronBrooks11]

What

Provision the observability stack (InfluxDB + Grafana) via the deployment engine, supporting two first-class topologies:

• Co-located — stack runs on the control Pi alongside the runtime (current default).
• Separate host — stack runs on a dedicated monitoring host; control Pis push telemetry to it (long-term preferred form).

Split out of #137 (G3): the observability half is Docker/topology-heavy and has a very different blocker profile from telemetry-export provisioning.

Context

Deep assessment (2026-07) established that the control Pi only needs the telemetry-export service pointed at an InfluxDB URL + token (that is #137). Where InfluxDB/Grafana actually run is this issue. Observability ships today as the anolis-<ver>-observability.tar.gz release asset deployed via docker compose; workbench currently runs it on the operator's host (a de-facto separate-host).

Scope / open questions

• Docker prerequisite: install.sh installs only i2c-tools; nothing installs Docker. Decide graceful-skip-if-absent vs fail-fast; documented operator precondition. install.sh must NOT apt-install Docker (offline guarantee).
• arm64 images: verify the influxdb/grafana image tags publish linux/arm64 manifests; pin by digest.
• Co-located mode: run docker compose as root (never add the nologin anolis user to the docker group). Note runtime/monitoring resource contention on the control Pi.
• Separate-host mode: provisioning target is the monitoring host, not the control Pi — define the invocation path.
• Extract/start flag: a dedicated flag — NOT the global --no-start (which gates the runtime's start + health; overloading it would stop the runtime).
• Secrets: InfluxDB admin token + Grafana creds via a 0600 root EnvironmentFile / generated; never logged (Define telemetry redaction/classification + secret-safe logging policy #96); no insecure defaults that ship silently.
• Offline images: docker save/load of arm64 images for air-gap — no CI produces these today; deferred to Define packaging & distribution strategy for shipped components (future) #100.
• Rollback/uninstall: observability out of the binaries-only rollback contract; do_uninstall gains docker compose down.

Child of #139.

[CameronBrooks11]

Design input harvested from deployment planning docs (Q6 research)

working/ is being retired; preserving the observability-provisioning design decisions here.

Decision: native apt, not Docker, on the Pi

InfluxDB 2.x ships a native arm64 binary via the InfluxData apt repo; Grafana is likewise available natively. There is therefore no reason to require Docker/Compose on the Pi for the observability layer — Docker was explicitly rejected in the research (memory overhead on Pi 4, extra dependency). Provision InfluxDB + Grafana as native apt packages + systemd services.

(The old Stage-1 draft sketched a docker compose up -d flow; that is superseded by this native-apt decision.)

Proposed --with-observability phase flow

Step	What	How
1	Install InfluxDB	native apt install influxdb2 from InfluxData apt repo
2	Create bucket + token	influx setup --force (non-interactive first-run)
3	Point runtime at Influx	set InfluxDB URL + token in runtime config
4	Install Grafana	native apt (Grafana apt repo), provision anolis dashboards + Influx datasource
5	Generate configs	render connection details; secrets into a root-only 0600 EnvironmentFile outside ${PREFIX} (same pattern as telemetry-export's /etc/anolis/telemetry-export.env)
6	systemd	enable --now influxdb grafana-server
7	Verify	InfluxDB reachable on :8086, Grafana on :3000

Relationship to the already-shipped telemetry-export

--with-telemetry-export (shipped, #163) provisions the export service (PyPI anolis-telemetry-export in a venv). --with-observability is the storage + dashboards layer beneath it (InfluxDB the runtime writes to + Grafana). They compose: export service reads runtime data → InfluxDB; Grafana visualizes. Keep them as separate opt-in flags.

Offline/air-gapped provisioning of the apt-sourced InfluxDB/Grafana is out of scope for v1 here (tracked alongside #100).

🤖 Generated with Claude Code

[CameronBrooks11]

Scoping addendum from the 2026-07-15 sprint: two inputs that didn't exist when this was scoped.

1. Per-device I/O health metrics are now on the HTTP surface (anolis 0.1.35, runtime status hides degraded providers: AVAILABLE with missing configured devices, install health passes #185/feat(http): pass provider-reported health through /v0/providers/health #186): /v0/providers/health carries provider-reported io_ok/io_failed/io_retried_attempts + real last_seen per device (bread 0.3.3+; ezo parity filed as anolis-provider-ezo#100). Any Grafana deployment this issue provisions should scrape/panel these — the retried-attempts slope is the leading indicator for bus degradation (it's what would have caught I2C: DCMT fails to serve ~30% of CRUMBS state queries (RLHT on the same bus: 0/360) feastorg/Slice_DCMT#3 in week one), and the degraded/failed_device_count pair is the alerting primitive for silent device loss.

2. The interim measurement practice this replaces: the actuation-bench work currently samples /v0/providers/health to CSV via an ad-hoc script for before/after comparisons (idle-vs-load retry slopes). That is exactly a dashboard + retention question — worth listing as an acceptance scenario for the co-located topology: 'operator can see per-device retry-rate over the last 24 h without SSH'.