Add steering-quality evaluation harness + CI metrics comment (#466)

* Add steering-quality evaluation harness + CI metrics comment

Deterministic mock-time evaluation suite (python -m
astrameter.simulator.evaluation) that wires CT002 active control against
the firmware-accurate battery simulators and scripted household scenarios
(load spikes, solar curves, single/multi/mixed batteries, both fair-share
and efficiency-optimization modes). Reports reaction (settling time),
oscillation (overshoot, band crossings, steady RMS) and energy
(avoidable import/export) metrics, with --json/--compare for diffing runs.

CI gains a steering-eval job that runs the suite on PR base and head and
posts the before/after table as a sticky PR comment (informational only).

Groundwork for the active-control redesign in #458.

https://claude.ai/code/session_01SeKwnaH74tzQoZEPhmhBu8

* eval: model meter latency, fix oven and cloud-dip scenario bugs

Three harness fixes that belong in the baseline rather than in the
controller-change PR stacked on top:

- The controller now reads the meter at a realistic ~1 s cadence
  (Scenario.meter_interval_s) while metrics see the true instantaneous
  grid; a zero-latency meter flattered the current controller.
- The oven thermostat schedule is emitted as paired on/off cycles; the
  previous loop could end stuck ON, stacking loads beyond the battery's
  ceiling for the rest of the run (the 281 W steady RMS in
  single_venus_steps was that saturation, not steering behavior).
- Solar is now curve x factor so the labeled cloud-dip transients
  compose with the per-minute day curve instead of being overwritten by
  its next tick mid-measurement-window.

https://claude.ai/code/session_01SeKwnaH74tzQoZEPhmhBu8

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: tomquist

.github/workflows/ci.yml

@@ -208,6 +208,109 @@ jobs:
           name: coverage-${{ matrix.python-version }}
           path: coverage.xml
 
+  # Steering-quality evaluation (issue #458): runs the deterministic
+  # active-control simulation suite (src/astrameter/simulator/evaluation.py)
+  # on the PR base and head and posts the before/after metrics as a sticky
+  # PR comment. Purely informational — thresholds that must hold live in
+  # pytest regressions, not here.
+  steering-eval:
+    needs: [ lint ]
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    permissions:
+      contents: read
+      pull-requests: write
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+
+      - name: Install dependencies
+        run: uv sync --frozen --extra dev
+
+      - name: Run evaluation (head)
+        run: uv run python -m astrameter.simulator.evaluation --json head.json
+
+      # The base run uses the base commit's own harness + controller. Until
+      # the harness exists on the base branch this produces an empty
+      # baseline and the comment shows head-only numbers.
+      - name: Run evaluation (base)
+        if: github.event_name == 'pull_request'
+        run: |
+          git worktree add /tmp/steering-base "${{ github.event.pull_request.base.sha }}"
+          if [ -f /tmp/steering-base/src/astrameter/simulator/evaluation.py ]; then
+            (cd /tmp/steering-base \
+              && uv sync --frozen --extra dev \
+              && uv run python -m astrameter.simulator.evaluation --json "$GITHUB_WORKSPACE/base.json")
+          else
+            echo "[]" > "$GITHUB_WORKSPACE/base.json"
+          fi
+
+      - name: Render comparison
+        if: github.event_name == 'pull_request'
+        run: |
+          uv run python -m astrameter.simulator.evaluation \
+            --input head.json --compare base.json > steering-eval-comment.md
+          cat steering-eval-comment.md >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Upload metrics artifacts
+        uses: actions/upload-artifact@v4
+        with:
+          name: steering-eval
+          path: |
+            head.json
+            base.json
+            steering-eval-comment.md
+          if-no-files-found: ignore
+
+      # Forks get a read-only token; they still see the table in the job
+      # summary above.
+      - name: Post sticky PR comment
+        if: >-
+          github.event_name == 'pull_request' &&
+          github.event.pull_request.head.repo.full_name == github.repository
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const fs = require('fs');
+            const marker = '<!-- steering-eval -->';
+            const body = marker + '\n' +
+              fs.readFileSync('steering-eval-comment.md', 'utf8');
+            const {data: comments} = await github.rest.issues.listComments({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: context.issue.number,
+              per_page: 100,
+            });
+            const existing = comments.find(
+              c => c.body && c.body.startsWith(marker));
+            if (existing) {
+              await github.rest.issues.updateComment({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                comment_id: existing.id,
+                body,
+              });
+            } else {
+              await github.rest.issues.createComment({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                issue_number: context.issue.number,
+                body,
+              });
+            }
+
   build:
     needs: [ validate ]
     permissions:

AGENTS.md

@@ -23,6 +23,17 @@ CI runs the same steps (see `.github/workflows/ci.yml`).
 
 `esphome/components/ct002/` is a C++ mirror of the Python CT002 stack. Any change to shared behavior must land on **both** sides in the same change. See `CONTRIBUTING.md` for the file mapping and what has no C++ counterpart. Verify with `uv run pytest tests/components/ct002/`.
 
+## Steering-quality evaluation (run when touching balancer behavior)
+
+`uv run python -m astrameter.simulator.evaluation` simulates hours of
+realistic household activity against the firmware-accurate battery plant and
+reports reaction/oscillation/energy metrics per scenario. When changing
+`src/astrameter/ct002/balancer.py` (or anything else in the active-control
+loop), capture a baseline first (`--json base.json` on the unchanged code),
+re-run after the change, and compare with `--input head.json --compare
+base.json`. CI runs the same suite on PR base + head (job `steering-eval`)
+and posts the comparison as a sticky PR comment.
+
 ## Changelog
 
 For user-facing work on a branch, keep **one bullet under `## Next`** that summarizes the **overall** outcome of that branch. **Add** it when you first document the change; on **later iterations** on the same branch, **edit that same bullet** if the scope or wording shifts—do **not** append extra bullets for each follow-up. Skip `CHANGELOG.md` entirely when nothing users would notice changes (refactors, tests-only, etc.).