Skip to content

Commit 3c15a29

Browse files
authored
feat(test-classifier): post-to-PR + --submit one-shot metrics (auto-mint, Testing Events) (#61)
* feat(test-classifier): let the zsh function pass flags through (post to PR) The test-classifier function hardcoded its arg mapping (no-arg → --unpushed, $1 → --against $1) and swallowed everything else, so there was no way to post a comment — you had to drop down to calling the dispatcher directly. That's a dead end for the common "classify and post to my PR" case. New routing (back-compatible): - no args → --unpushed (unchanged) - bare ref → --against <ref> (unchanged) - starts with - → passed straight through (NEW: --pr 42 --post-comment, --post-comment, --dry-run, --json-only, …) - ref + flags → --against <ref> <flags> (NEW combo) So `test-classifier --pr 42 --post-comment` now just works. Verified all six routing forms against a stub dispatcher; function is bash- and zsh-clean. Docs updated: function header, "what it does", Usage (with posting examples), and the report-only framing (now "report-only by default; opt in to posting"). Added a note that locally-posted comments are dev-authored, so metricsai needs --all-authors / METRICSAI_TESTING_GITHUB_AUTHORS to count them. * feat(test-classifier): --submit — classify, post, prompt, record in one shot Streamlines the manual local metrics loop (Foad's idea): instead of post comment → react 👍/👎 on GitHub → wait for the weekly harvest, capture the signal right after the run. --submit implies --post-comment, then on an interactive run prompts "Was the classification helpful? [y/n]" and appends ONE row to the Sheet's Testing Events tab — repo, pr, comment_id, verdict, category, confidence, thumbs_up, thumbs_down, reason (the same column contract the harvester writes). - Capture the new comment's id from the post (was discarded) for the row. - submit_metrics_row(): TTY-gated prompt; collapses multiple classifications to one representative verdict by most-actionable rank; appends via the Sheets values:append API to the "Testing Events" tab (NOT the weekly CXT/DMOD/… tabs). - Non-TTY / CI: posts the comment, skips the prompt + row (no hang) — CI metrics still come from the central weekly harvest. - No GOOGLE_SHEETS_TOKEN/SHEET_ID: records the answer to the terminal, warns no sink, still posts. Reuses the harvester's exact env var names. Works through the zsh function as-is (it passes flags through): test-classifier --pr 42 --submit. Verified: payload + Testing-Events URL build correctly, append fails gracefully on bad creds, non-TTY gate trips, verdict collapse picks the most-actionable. bash -n clean. * feat(test-classifier): --submit auto-mints token; align to Testing Events 10-col Make --submit set-and-forget and match the live sheet. Auto-mint (no per-session token juggling): - --submit mints a short-lived Sheets token via gcloud by impersonating the pilot SA (defaults to metrics-sheets-writer@nava-labs.iam.gserviceaccount.com, the SA the central sweep uses; override with METRICSAI_SA_EMAIL). An explicit GOOGLE_SHEETS_TOKEN still wins (CI). gcloud/impersonation failure → warns, still posts the comment. Column alignment (the Testing Events tab header): repo, pr, comment_id, comment_created_at, verdict, category, confidence, thumbs_up, thumbs_down, reason - --submit now emits comment_created_at (from the comment POST response, else date -u) — captured alongside the comment id. Adds the 10th column (reason). - The shell harvester (test_classifier_comments.sh) is aligned to the same 10 columns + order (it already had .created_at for windowing; now emits it). - Fix the A1 range: the tab name has a space, so it must be single-quoted — 'Testing Events'!A1 (matches the sweep). Both writers now default to it. Docs: SHEET_ID = the pilot sheet; one-line ~/.zshrc setup; auto-mint + override notes; two-writers-one-tab (deduped by comment_id) clarified. * fix(test-classifier): --submit posts via the metricsai webhook, not direct Sheets API Switch --submit to Brian's webhook transport (the one that actually works) and fix a false-failure bug found while probing the live endpoint. - Replace the SA / gcloud / direct Sheets values:append path with a POST to the metricsai Apps Script webhook: flat JSON named fields + reserved _tab / _key, same as metricsai's client. Env is now METRICSAI_WEBHOOK_URL + METRICSAI_WEBHOOK_KEY (both static — no service account, no short-lived token, no gcloud). The SA approach was a dead end: gcloud ignores --scopes for impersonated accounts, so the minted token lacked the spreadsheets scope. - The script aligns fields by header name, so column order no longer matters; _tab defaults to "Testing Events" (override: METRICSAI_WEBHOOK_TAB). - Fix success detection: the write happens at the /exec 302, and following that redirect can 405 on the final Drive hop even though the row landed. Stop using `curl -f -L` (false failure) — capture the first-hop status; 200/302 = success. Verified live against the real webhook: a full 10-field row posts and lands in the Testing Events tab with every value under its correct header. Also: gitignore .env.local / .env so local webhook creds never get committed.
1 parent a02f739 commit 3c15a29

4 files changed

Lines changed: 267 additions & 25 deletions

File tree

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,3 @@
11
.claude/settings.local.json
2+
.env.local
3+
.env

testing/classifier/.skills/test-classifier/scripts/test-classifier-dispatcher.sh

Lines changed: 161 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
# Usage:
3030
# test-classifier-dispatcher.sh # auto-discover PR; print only
3131
# test-classifier-dispatcher.sh --pr 1234 --post-comment # post the PR comment
32+
# test-classifier-dispatcher.sh --pr 1234 --submit # post + prompt "helpful?" + append a Testing Events row
3233
# test-classifier-dispatcher.sh --against origin/main # explicit base ref
3334
# test-classifier-dispatcher.sh --unpushed # local: committed+staged, NO PR (report-only)
3435
# test-classifier-dispatcher.sh --json-only # emit only the JSON block
@@ -40,6 +41,13 @@
4041
#
4142
# Required when --post-comment is used:
4243
# gh CLI installed and authenticated; or GH_TOKEN exported
44+
#
45+
# Required when --submit posts the metrics row (interactive runs only):
46+
# METRICSAI_WEBHOOK_URL + METRICSAI_WEBHOOK_KEY (the metricsai Apps Script
47+
# endpoint + the AI Metrics API key — both static, set once). Row posts to the
48+
# "Testing Events" tab (override: METRICSAI_WEBHOOK_TAB). Absent → posts the
49+
# comment + prompts but skips the row. Same transport metricsai uses; no
50+
# service account or gcloud token needed.
4351

4452
set -euo pipefail
4553

@@ -72,6 +80,10 @@ SKILL_PATH_CANONICAL=".skills/test-classifier/SKILL.md"
7280
#
7381
# --pr <number> Explicit PR number (overrides auto-discovery)
7482
# --post-comment Post ONE PR comment via gh api (omit to print only)
83+
# --submit Implies --post-comment; then (interactive only) prompts
84+
# "Was this helpful?" and appends one Testing Events row
85+
# to the Sheet with the verdict + 👍/👎. Non-TTY/CI: posts
86+
# and skips the prompt + row.
7587
# --gate Exit 1 if the result is CLASSIFIED (CI-blocking mode)
7688
# --json-only Print only the JSON block (machine consumption)
7789
#
@@ -84,6 +96,9 @@ POST_COMMENT=0
8496
GATE_MODE=0
8597
JSON_ONLY=0
8698
WANT_HELP=0
99+
SUBMIT=0
100+
POSTED_COMMENT_ID="" # set by the post functions; consumed by --submit
101+
POSTED_COMMENT_CREATED="" # the comment's created_at (ISO-8601) from the API
87102
REMAINING_FOR_LIB=()
88103

89104
while [[ $# -gt 0 ]]; do
@@ -108,6 +123,14 @@ while [[ $# -gt 0 ]]; do
108123
POST_COMMENT=1
109124
shift
110125
;;
126+
--submit)
127+
# Streamlined local loop: post the comment AND, when run interactively,
128+
# prompt "Was this helpful?" and append the answer as one Testing Events
129+
# row to the Sheet. Implies --post-comment (the row needs the comment_id).
130+
SUBMIT=1
131+
POST_COMMENT=1
132+
shift
133+
;;
111134
--gate)
112135
GATE_MODE=1
113136
shift
@@ -489,16 +512,21 @@ post_issue_comment_to_github() {
489512
local pr_number="$2"
490513
local body="$3"
491514

492-
# Pass the body via --field so gh handles JSON escaping for us.
493-
if ! gh api \
515+
# Pass the body via --field so gh handles JSON escaping for us. Capture id +
516+
# created_at (for --submit's metrics row); empty string on failure.
517+
local resp=""
518+
resp="$(gh api \
494519
"repos/${repo_slug}/issues/${pr_number}/comments" \
495520
--method POST \
496-
--field body="${body}" >/dev/null; then
521+
--field body="${body}" --jq '"\(.id)\t\(.created_at)"' 2>/dev/null || true)"
522+
if [[ -z "$resp" || "$resp" == $'\t' ]]; then
497523
echo "ERROR: 'gh api' issue-comment call failed." >&2
498524
echo " Check your gh auth status and that your token has 'pull-requests: write'" >&2
499525
echo " (or 'issues: write')." >&2
500526
return 1
501527
fi
528+
POSTED_COMMENT_ID="${resp%%$'\t'*}"
529+
POSTED_COMMENT_CREATED="${resp#*$'\t'}"
502530
}
503531

504532
# Post ONE classification comment to the PR. Args: PR number, comment body.
@@ -547,13 +575,19 @@ post_comment_to_github() {
547575

548576
if [[ -n "$anchor_path" && -n "$commit_id" ]]; then
549577
echo "[test-classifier] Posting classification as a review comment on ${repo_slug} PR #${pr_number} (anchored to ${anchor_path})..."
550-
if gh api \
578+
# Capture id + created_at from the response so --submit can record them.
579+
# jq joins them with a tab; empty string on failure.
580+
local resp=""
581+
resp="$(gh api \
551582
"repos/${repo_slug}/pulls/${pr_number}/comments" \
552583
--method POST \
553584
--field body="${body}" \
554585
--field commit_id="${commit_id}" \
555586
--field path="${anchor_path}" \
556-
--field subject_type=file >/dev/null 2>&1; then
587+
--field subject_type=file --jq '"\(.id)\t\(.created_at)"' 2>/dev/null || true)"
588+
if [[ -n "$resp" && "$resp" != $'\t' ]]; then
589+
POSTED_COMMENT_ID="${resp%%$'\t'*}"
590+
POSTED_COMMENT_CREATED="${resp#*$'\t'}"
557591
echo "[test-classifier] Review comment posted. Awaiting the developer's 👍/👎 reaction (and a reply reason on a 👎)."
558592
return 0
559593
fi
@@ -570,6 +604,121 @@ post_comment_to_github() {
570604
echo "[test-classifier] Comment posted (issue comment). Awaiting the developer's 👍/👎 reaction."
571605
}
572606

607+
# ── --submit: prompt "Was this helpful?" and post one row via the webhook ───
608+
# Streamlines the manual local loop: instead of (post comment → react 👍/👎 on
609+
# GitHub → a separate weekly harvest), capture the developer's signal right after
610+
# the run and POST it to the metricsai Google Apps Script webhook — the SAME
611+
# transport metricsai uses (flat JSON body, fields aligned by header name, plus
612+
# reserved `_tab` and `_key`). No service account, no gcloud, no token expiry —
613+
# just two static env vars the developer sets once.
614+
#
615+
# Fields (aligned by header name on the Apps Script side):
616+
# repo, pr, comment_id, comment_created_at, verdict, category, confidence,
617+
# thumbs_up, thumbs_down, reason (+ _tab="Testing Events", _key=<api key>)
618+
#
619+
# Env:
620+
# METRICSAI_WEBHOOK_URL the Apps Script /exec endpoint (required)
621+
# METRICSAI_WEBHOOK_KEY the "AI Metrics" API key (body _key) (required)
622+
# METRICSAI_WEBHOOK_TAB destination tab; defaults to Testing Events
623+
#
624+
# Gated: only runs interactively (TTY, not CI) — non-interactive runs post the
625+
# comment and skip the prompt + POST (no human signal to record, no hang).
626+
# Missing URL/key → records the answer to the terminal, warns, still posts the
627+
# comment.
628+
#
629+
# Args: PR number, the extracted classifier JSON.
630+
submit_metrics_row() {
631+
local pr_number="$1"
632+
local classifier_json="$2"
633+
634+
if [[ ! -t 0 ]] || [[ "${CI:-}" == "true" ]]; then
635+
ai_review::info "--submit: non-interactive run — comment posted; skipping the helpfulness prompt + metrics row." >&2
636+
return 0
637+
fi
638+
if ! command -v python3 &>/dev/null; then
639+
ai_review::warn "--submit: python3 not found; cannot build the metrics row — skipping it." >&2
640+
return 0
641+
fi
642+
643+
# Prompt for the tuning signal. Empty/invalid → skip (don't guess a verdict).
644+
local answer reason="" thumbs_up=0 thumbs_down=0
645+
printf '%s' " Was the classification helpful? [y/n] (enter to skip): " >&2
646+
read -r answer
647+
case "${answer}" in
648+
y|Y|yes|YES) thumbs_up=1 ;;
649+
n|N|no|NO)
650+
thumbs_down=1
651+
printf '%s' " Optional one-line reason (enter to skip): " >&2
652+
read -r reason
653+
;;
654+
*)
655+
ai_review::info "--submit: no answer — skipping the metrics row (comment still posted)." >&2
656+
return 0
657+
;;
658+
esac
659+
660+
local webhook_url="${METRICSAI_WEBHOOK_URL:-}"
661+
local webhook_key="${METRICSAI_WEBHOOK_KEY:-}"
662+
local webhook_tab="${METRICSAI_WEBHOOK_TAB:-Testing Events}"
663+
if [[ -z "${webhook_url}" || -z "${webhook_key}" ]]; then
664+
ai_review::warn "--submit: METRICSAI_WEBHOOK_URL / METRICSAI_WEBHOOK_KEY not set — recorded your answer but didn't post a row." >&2
665+
ai_review::log " Export both (the metricsai webhook URL + the AI Metrics API key) to enable the sink." >&2
666+
return 0
667+
fi
668+
669+
local repo_slug
670+
repo_slug="$(gh repo view --json nameWithOwner --jq '.nameWithOwner' 2>/dev/null || echo "")"
671+
672+
# The classifier may emit multiple classifications; the Testing Events row is
673+
# per-comment, so collapse to one representative verdict by most-actionable
674+
# rank (APPLICATION_BUG > TEST_BUG > FLAKY_FAILURE > ENVIRONMENT_ISSUE) — the
675+
# same rule the harvester uses. Returns: verdict<TAB>category<TAB>confidence.
676+
local repr
677+
repr="$(printf '%s' "${classifier_json}" | python3 -c '
678+
import json, sys
679+
RANK = {"APPLICATION_BUG":3,"TEST_BUG":2,"FLAKY_FAILURE":1,"ENVIRONMENT_ISSUE":0}
680+
try:
681+
cls = (json.load(sys.stdin) or {}).get("classifications") or []
682+
except Exception:
683+
cls = []
684+
if not cls:
685+
print("\t\t"); sys.exit(0)
686+
best = max(cls, key=lambda c: RANK.get(c.get("verdict",""), -1))
687+
print("\t".join([best.get("verdict","") or "", best.get("category","") or "", best.get("confidence","") or ""]))
688+
' 2>/dev/null || printf '\t\t')"
689+
local verdict category confidence
690+
IFS=$'\t' read -r verdict category confidence <<< "${repr}"
691+
692+
# comment_created_at comes from the comment POST response; fall back to now.
693+
local created_at="${POSTED_COMMENT_CREATED:-}"
694+
[[ -n "${created_at}" ]] || created_at="$(date -u +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || echo "")"
695+
696+
# Flat JSON body: named fields (Apps Script aligns by header) + reserved
697+
# _tab / _key. Built with python3 so values are safely JSON-escaped.
698+
local body
699+
body="$(python3 -c '
700+
import json, sys
701+
keys = ["repo","pr","comment_id","comment_created_at","verdict","category",
702+
"confidence","thumbs_up","thumbs_down","reason","_tab","_key"]
703+
print(json.dumps(dict(zip(keys, sys.argv[1:1+len(keys)]))))
704+
' "${repo_slug}" "${pr_number}" "${POSTED_COMMENT_ID}" "${created_at}" "${verdict}" "${category}" "${confidence}" "${thumbs_up}" "${thumbs_down}" "${reason}" "${webhook_tab}" "${webhook_key}")"
705+
706+
# The Apps Script writes the row in doPost and answers /exec with a 302 to a
707+
# googleusercontent URL. The WRITE has already happened at that 302 — following
708+
# it can 405 on the final Drive hop even though the row landed, so we must NOT
709+
# use `-f -L` (that reports a false failure). Capture the first-hop status and
710+
# treat 200/302 as success; don't follow the redirect.
711+
local http_code
712+
http_code="$(curl -sS -o /dev/null -w '%{http_code}' -X POST "${webhook_url}" \
713+
-H "Content-Type: application/json" \
714+
-d "${body}" 2>/dev/null || echo "000")"
715+
if [[ "${http_code}" == "200" || "${http_code}" == "302" ]]; then
716+
ai_review::ok "--submit: posted a row to the metrics webhook (tab=${webhook_tab}, verdict=${verdict:-?}, $([ "${thumbs_up}" = 1 ] && echo 👍 || echo 👎))."
717+
else
718+
ai_review::warn "--submit: webhook POST failed (HTTP ${http_code}; your answer was not recorded). Check METRICSAI_WEBHOOK_URL / METRICSAI_WEBHOOK_KEY." >&2
719+
fi
720+
}
721+
573722
# ── Custom run loop (mirrors the security PR dispatcher) ────────────────────
574723
test_classifier::run() {
575724
# Discover the PR (and inject --against into REMAINING_FOR_LIB).
@@ -592,6 +741,7 @@ test_classifier::run() {
592741
ai_review::log " PR number: ${AI_REVIEW_PR_NUMBER:-(none — using --against directly)}"
593742
ai_review::log " Diff source: $(ai_review::diff_command_description)"
594743
ai_review::log " Post comment: ${POST_COMMENT}"
744+
ai_review::log " Submit metrics: ${SUBMIT}"
595745
ai_review::log " Gate mode: ${GATE_MODE}"
596746
ai_review::log " Changed files:"
597747
ai_review::changed_files | sed 's/^/ /'
@@ -679,6 +829,12 @@ test_classifier::run() {
679829
local comment_body
680830
comment_body="$(render_pr_comment_body "${json_block}")"
681831
post_comment_to_github "${AI_REVIEW_PR_NUMBER}" "${comment_body}"
832+
833+
# --submit: prompt for the helpfulness signal and append a Testing Events
834+
# row (interactive only; no-ops cleanly in CI / non-TTY).
835+
if (( SUBMIT == 1 )); then
836+
submit_metrics_row "${AI_REVIEW_PR_NUMBER}" "${json_block}"
837+
fi
682838
fi
683839
fi
684840

testing/classifier/docs/LOCAL_TEST_CLASSIFIER.md

Lines changed: 92 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,9 @@ and uses the same `--unpushed` scope rule. This is the ergonomics layer over
2424
while you're still iterating.
2525
- **It's a local preview, not the gate.** The PR run (Path B) is the recorded
2626
backstop that feeds the metrics loop. The local run is for the developer's own
27-
fast feedback before pushing; it is **report-only** — it never posts a PR
28-
comment (there's no PR to post to) and never blocks anything.
27+
fast feedback before pushing; it never blocks anything. By default it's
28+
**report-only** (prints to your terminal, posts nothing) — but you can opt in
29+
to posting against a real PR with `--pr N --post-comment` (see Usage).
2930
- **OBSERVED on demand.** By default a local run is INFERRED (it predicts from
3031
the diff — read-only, never touches your machine's toolchain). Opt into
3132
OBSERVED with `AI_RUN_SUITE=1` to have the agent actually locate, install, and
@@ -60,9 +61,12 @@ it's only needed for the PR-based modes (`--pr` / auto-discovery / `--post-comme
6061
Paste this block into `~/.zshrc`, then `source ~/.zshrc` (or open a new shell):
6162

6263
```zsh
63-
# AI test classifier — local, report-only.
64-
# <no arg> → classify everything not yet pushed (committed + staged)
65-
# <ref> → classify the committed range <ref>..HEAD
64+
# AI test classifier.
65+
# <no arg> → classify everything not yet pushed (committed + staged); report-only
66+
# <ref> → classify the committed range <ref>..HEAD; report-only
67+
# <flags...> → passed straight to the dispatcher. This is how you post to a PR:
68+
# test-classifier --pr 42 --post-comment (or --post-comment to
69+
# auto-discover the current branch's PR). Needs gh authed.
6670
# Prefix with AI_RUN_SUITE=1 to run the suite locally (OBSERVED) instead of
6771
# inferring from the diff: AI_RUN_SUITE=1 test-classifier
6872
test-classifier() {
@@ -72,12 +76,19 @@ test-classifier() {
7276
disp="$root/testing/classifier/.skills/test-classifier/scripts/test-classifier-dispatcher.sh"
7377
[ -x "$disp" ] \
7478
|| { echo "✗ test-classifier dispatcher not found/executable: $disp"; return 1; }
75-
if [ -n "$1" ]; then
76-
echo "▶ test-classifier: ${1}..HEAD (committed range)"
77-
"$disp" --against "$1"
78-
else
79+
if [ "$#" -eq 0 ]; then
80+
# No args → classify everything not yet pushed (committed + staged).
7981
echo "▶ test-classifier: all not-yet-pushed changes (committed + staged)"
8082
"$disp" --unpushed
83+
elif [ "${1#-}" != "$1" ]; then
84+
# First arg is a flag (starts with -) → pass everything straight through.
85+
# This is how you post to a PR: test-classifier --pr 42 --post-comment
86+
"$disp" "$@"
87+
else
88+
# First arg is a bare ref → --against <ref>; pass any remaining flags too.
89+
local ref="$1"; shift
90+
echo "▶ test-classifier: ${ref}..HEAD (committed range)"
91+
"$disp" --against "$ref" "$@"
8192
fi
8293
}
8394
```
@@ -89,13 +100,18 @@ What the function does:
89100
function works from any subdirectory.
90101
2. **Locates the dispatcher** under `testing/classifier/` and confirms it's
91102
executable.
92-
3. **Runs the classifier** by calling the dispatcher directly:
103+
3. **Runs the classifier** by calling the dispatcher:
93104
- **no argument**`--unpushed`: everything not yet pushed (committed +
94105
staged). The base is your branch's upstream, falling back to the merge-base
95106
with the remote default branch. If neither can be determined (e.g. a
96107
brand-new branch with no remote), it errors and asks for an explicit ref
97108
rather than silently classifying less than you expect.
98-
- **a ref**`--against <ref>`: the committed range `<ref>..HEAD`.
109+
- **a bare ref**`--against <ref>`: the committed range `<ref>..HEAD`.
110+
- **anything starting with `-`** → passed straight through to the dispatcher.
111+
This is how you post to a PR — `test-classifier --pr 42 --post-comment`
112+
(or just `--post-comment` to auto-discover the current branch's PR), plus
113+
any other dispatcher flag (`--dry-run`, `--json-only`, …). You can combine a
114+
ref with flags too: `test-classifier origin/main --post-comment`.
99115

100116
---
101117

@@ -107,7 +123,72 @@ test-classifier origin/main # the committed range origin/main..HEAD
107123
test-classifier HEAD~1 # just the last commit
108124

109125
AI_RUN_SUITE=1 test-classifier # run the suite locally and triage REAL failures (OBSERVED)
126+
127+
# Post the result as a PR comment (needs an open PR + gh authed):
128+
test-classifier --post-comment # auto-discovers the current branch's PR
129+
test-classifier --pr 42 --post-comment # explicit PR number
130+
AI_RUN_SUITE=1 test-classifier --pr 42 --post-comment # OBSERVED + post
131+
132+
# Streamlined: post AND record the metric in one shot (see --submit below):
133+
test-classifier --pr 42 --submit
134+
```
135+
136+
> **Posting authorship.** A locally-posted comment is authored by **you**, not
137+
> `github-actions[bot]`. For the comment to be picked up by the weekly metrics
138+
> harvest, `metricsai` must be told to count your author (`--all-authors` or
139+
> `METRICSAI_TESTING_GITHUB_AUTHORS`); by default it only counts the CI bot. For
140+
> just *posting to the PR*, this doesn't matter.
141+
142+
---
143+
144+
## `--submit` — classify, post, and record the metric in one shot
145+
146+
The slow path to metrics is: post a comment → react 👍/👎 on GitHub later → wait
147+
for the weekly harvest. `--submit` collapses that into the run you're already
148+
doing. After classifying and posting, it prompts right in the terminal:
149+
110150
```
151+
Was the classification helpful? [y/n] (enter to skip):
152+
```
153+
154+
Your answer is appended **immediately** as one row to the Sheet's **Testing
155+
Events** tab — verdict, file context, confidence, and your 👍/👎 (with an optional
156+
one-line reason on a 👎). No GitHub round-trip, no separate harvest.
157+
158+
```bash
159+
test-classifier --pr 42 --submit # classify (INFERRED) → post → ask → record
160+
AI_RUN_SUITE=1 test-classifier --pr 42 --submit # OBSERVED → post → ask → record
161+
test-classifier --submit # auto-discovers the current branch's PR
162+
```
163+
164+
**Requirements:** everything `--post-comment` needs (open PR + `gh` authed),
165+
**plus** the metricsai webhook creds — two static values, set once in `~/.zshrc`:
166+
167+
```bash
168+
export METRICSAI_WEBHOOK_URL="https://script.google.com/macros/s/<id>/exec"
169+
export METRICSAI_WEBHOOK_KEY="<the AI Metrics API key>"
170+
```
171+
172+
That's the whole setup. `--submit` POSTs the row to the same Google Apps Script
173+
webhook `metricsai` uses (flat JSON body; the script aligns fields to the sheet
174+
by header name, so column order never matters). **No gcloud, no service account,
175+
no token to refresh** — both values are permanent.
176+
177+
The row lands in the **Testing Events** tab by default; override with
178+
`METRICSAI_WEBHOOK_TAB`.
179+
180+
**Behavior notes:**
181+
- **Interactive only.** In CI or any non-TTY run, `--submit` posts the comment
182+
and **skips** the prompt + row (no human to ask, no hang). CI metrics still
183+
come from the central weekly harvest.
184+
- **No creds → no crash.** If `METRICSAI_WEBHOOK_URL` / `METRICSAI_WEBHOOK_KEY`
185+
aren't set, it records your answer to the terminal, warns, and **still posts**
186+
the comment.
187+
- **Writes to Testing Events, not the weekly tabs.** These are per-event rows
188+
(one per run), distinct from the `metricsai` weekly aggregate rows in the
189+
CXT / DMOD / EMMY / OSRE tabs. The central sweep writes to the same tab — two
190+
writers, one tab, deduped by `comment_id`.
191+
- **Pressing enter skips** the row entirely — no verdict is guessed.
111192

112193
**Scope details.**
113194

0 commit comments

Comments
 (0)