Commit 3c15a29
authored
feat(test-classifier): post-to-PR + --submit one-shot metrics (auto-mint, Testing Events) (#61)
* feat(test-classifier): let the zsh function pass flags through (post to PR)
The test-classifier function hardcoded its arg mapping (no-arg → --unpushed,
$1 → --against $1) and swallowed everything else, so there was no way to post a
comment — you had to drop down to calling the dispatcher directly. That's a dead
end for the common "classify and post to my PR" case.
New routing (back-compatible):
- no args → --unpushed (unchanged)
- bare ref → --against <ref> (unchanged)
- starts with - → passed straight through (NEW: --pr 42 --post-comment,
--post-comment, --dry-run, --json-only, …)
- ref + flags → --against <ref> <flags> (NEW combo)
So `test-classifier --pr 42 --post-comment` now just works. Verified all six
routing forms against a stub dispatcher; function is bash- and zsh-clean.
Docs updated: function header, "what it does", Usage (with posting examples),
and the report-only framing (now "report-only by default; opt in to posting").
Added a note that locally-posted comments are dev-authored, so metricsai needs
--all-authors / METRICSAI_TESTING_GITHUB_AUTHORS to count them.
* feat(test-classifier): --submit — classify, post, prompt, record in one shot
Streamlines the manual local metrics loop (Foad's idea): instead of post comment
→ react 👍/👎 on GitHub → wait for the weekly harvest, capture the signal right
after the run. --submit implies --post-comment, then on an interactive run
prompts "Was the classification helpful? [y/n]" and appends ONE row to the
Sheet's Testing Events tab — repo, pr, comment_id, verdict, category, confidence,
thumbs_up, thumbs_down, reason (the same column contract the harvester writes).
- Capture the new comment's id from the post (was discarded) for the row.
- submit_metrics_row(): TTY-gated prompt; collapses multiple classifications to
one representative verdict by most-actionable rank; appends via the Sheets
values:append API to the "Testing Events" tab (NOT the weekly CXT/DMOD/… tabs).
- Non-TTY / CI: posts the comment, skips the prompt + row (no hang) — CI metrics
still come from the central weekly harvest.
- No GOOGLE_SHEETS_TOKEN/SHEET_ID: records the answer to the terminal, warns no
sink, still posts. Reuses the harvester's exact env var names.
Works through the zsh function as-is (it passes flags through):
test-classifier --pr 42 --submit. Verified: payload + Testing-Events URL build
correctly, append fails gracefully on bad creds, non-TTY gate trips, verdict
collapse picks the most-actionable. bash -n clean.
* feat(test-classifier): --submit auto-mints token; align to Testing Events 10-col
Make --submit set-and-forget and match the live sheet.
Auto-mint (no per-session token juggling):
- --submit mints a short-lived Sheets token via gcloud by impersonating the
pilot SA (defaults to metrics-sheets-writer@nava-labs.iam.gserviceaccount.com,
the SA the central sweep uses; override with METRICSAI_SA_EMAIL). An explicit
GOOGLE_SHEETS_TOKEN still wins (CI). gcloud/impersonation failure → warns,
still posts the comment.
Column alignment (the Testing Events tab header):
repo, pr, comment_id, comment_created_at, verdict, category, confidence,
thumbs_up, thumbs_down, reason
- --submit now emits comment_created_at (from the comment POST response, else
date -u) — captured alongside the comment id. Adds the 10th column (reason).
- The shell harvester (test_classifier_comments.sh) is aligned to the same 10
columns + order (it already had .created_at for windowing; now emits it).
- Fix the A1 range: the tab name has a space, so it must be single-quoted —
'Testing Events'!A1 (matches the sweep). Both writers now default to it.
Docs: SHEET_ID = the pilot sheet; one-line ~/.zshrc setup; auto-mint + override
notes; two-writers-one-tab (deduped by comment_id) clarified.
* fix(test-classifier): --submit posts via the metricsai webhook, not direct Sheets API
Switch --submit to Brian's webhook transport (the one that actually works) and
fix a false-failure bug found while probing the live endpoint.
- Replace the SA / gcloud / direct Sheets values:append path with a POST to the
metricsai Apps Script webhook: flat JSON named fields + reserved _tab / _key,
same as metricsai's client. Env is now METRICSAI_WEBHOOK_URL +
METRICSAI_WEBHOOK_KEY (both static — no service account, no short-lived token,
no gcloud). The SA approach was a dead end: gcloud ignores --scopes for
impersonated accounts, so the minted token lacked the spreadsheets scope.
- The script aligns fields by header name, so column order no longer matters;
_tab defaults to "Testing Events" (override: METRICSAI_WEBHOOK_TAB).
- Fix success detection: the write happens at the /exec 302, and following that
redirect can 405 on the final Drive hop even though the row landed. Stop using
`curl -f -L` (false failure) — capture the first-hop status; 200/302 = success.
Verified live against the real webhook: a full 10-field row posts and lands in
the Testing Events tab with every value under its correct header.
Also: gitignore .env.local / .env so local webhook creds never get committed.1 parent a02f739 commit 3c15a29
4 files changed
Lines changed: 267 additions & 25 deletions
File tree
- testing
- classifier
- .skills/test-classifier/scripts
- docs
- metrics
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
| 2 | + | |
| 3 | + | |
Lines changed: 161 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
| |||
40 | 41 | | |
41 | 42 | | |
42 | 43 | | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
43 | 51 | | |
44 | 52 | | |
45 | 53 | | |
| |||
72 | 80 | | |
73 | 81 | | |
74 | 82 | | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
75 | 87 | | |
76 | 88 | | |
77 | 89 | | |
| |||
84 | 96 | | |
85 | 97 | | |
86 | 98 | | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
87 | 102 | | |
88 | 103 | | |
89 | 104 | | |
| |||
108 | 123 | | |
109 | 124 | | |
110 | 125 | | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
111 | 134 | | |
112 | 135 | | |
113 | 136 | | |
| |||
489 | 512 | | |
490 | 513 | | |
491 | 514 | | |
492 | | - | |
493 | | - | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
494 | 519 | | |
495 | 520 | | |
496 | | - | |
| 521 | + | |
| 522 | + | |
497 | 523 | | |
498 | 524 | | |
499 | 525 | | |
500 | 526 | | |
501 | 527 | | |
| 528 | + | |
| 529 | + | |
502 | 530 | | |
503 | 531 | | |
504 | 532 | | |
| |||
547 | 575 | | |
548 | 576 | | |
549 | 577 | | |
550 | | - | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
551 | 582 | | |
552 | 583 | | |
553 | 584 | | |
554 | 585 | | |
555 | 586 | | |
556 | | - | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
557 | 591 | | |
558 | 592 | | |
559 | 593 | | |
| |||
570 | 604 | | |
571 | 605 | | |
572 | 606 | | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
| 718 | + | |
| 719 | + | |
| 720 | + | |
| 721 | + | |
573 | 722 | | |
574 | 723 | | |
575 | 724 | | |
| |||
592 | 741 | | |
593 | 742 | | |
594 | 743 | | |
| 744 | + | |
595 | 745 | | |
596 | 746 | | |
597 | 747 | | |
| |||
679 | 829 | | |
680 | 830 | | |
681 | 831 | | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
682 | 838 | | |
683 | 839 | | |
684 | 840 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
28 | | - | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| |||
60 | 61 | | |
61 | 62 | | |
62 | 63 | | |
63 | | - | |
64 | | - | |
65 | | - | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
66 | 70 | | |
67 | 71 | | |
68 | 72 | | |
| |||
72 | 76 | | |
73 | 77 | | |
74 | 78 | | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
| 79 | + | |
| 80 | + | |
79 | 81 | | |
80 | 82 | | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
81 | 92 | | |
82 | 93 | | |
83 | 94 | | |
| |||
89 | 100 | | |
90 | 101 | | |
91 | 102 | | |
92 | | - | |
| 103 | + | |
93 | 104 | | |
94 | 105 | | |
95 | 106 | | |
96 | 107 | | |
97 | 108 | | |
98 | | - | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
99 | 115 | | |
100 | 116 | | |
101 | 117 | | |
| |||
107 | 123 | | |
108 | 124 | | |
109 | 125 | | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
110 | 150 | | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
111 | 192 | | |
112 | 193 | | |
113 | 194 | | |
| |||
0 commit comments