Skip to content

docs(plans): nightly AI verdict review workflow#1591

Open
ytallo wants to merge 1 commit intomainfrom
docs/verdict-review-plan
Open

docs(plans): nightly AI verdict review workflow#1591
ytallo wants to merge 1 commit intomainfrom
docs/verdict-review-plan

Conversation

@ytallo
Copy link
Copy Markdown
Contributor

@ytallo ytallo commented May 2, 2026

Summary

Adds the implementation plan for a nightly GitHub Actions workflow that runs an AI architectural verdict against the iii repo and posts the finding count to the existing release Slack channel.

Plan only — no code changes. Implementation lands in a follow-up PR.

The plan covers:

  • Single-job workflow on a 0 7 * * * cron with workflow_dispatch escape hatch
  • Static prompt at .github/verdict-prompt.md so the brief can be tuned without touching YAML
  • Python stdlib extractor with a 9-test pytest suite covering 7 scenarios (happy path, malformed/missing JSON contract, missing exec file, mid-body JSON noise, body truncation, zero-findings goal state)
  • Day-over-day finding-count delta in the Slack header (model itself stays stateless — only one integer leaks to the formatting layer)
  • Explicit if: failure() Slack ping so silent days can't hide drift
  • Notify-only — workflow never fails CI; pressure to fix comes from Slack visibility
  • Estimated cost: $150–$600/month at nightly + Opus

Goal state: 7 consecutive nights of iii nightly verdict — 0 finding(s) (no change).

The plan is decomposed into 10 TDD-shaped tasks (~10 commits) with full code in every step. Reuses existing Slack notification patterns from _homebrew.yml and the workflow-doc style from WORKFLOWS.md.

Test plan

  • Document renders correctly on GitHub
  • No code changes — CI should pass via path filters

Summary by CodeRabbit

  • Documentation
    • Added implementation documentation for the AI Verdict Review Workflow, detailing the nightly review execution process, artifact retention policies, verdict parsing specifications, and Slack notification handling. Includes testing guidelines and deployment verification procedures.

Adds the implementation plan for a GitHub Actions workflow that runs an
AI architectural verdict against the repo nightly and posts the finding
count + day-over-day delta to the existing release Slack channel.

Plan only — no code changes. Implementation lands in a follow-up PR.
@mintlify
Copy link
Copy Markdown

mintlify Bot commented May 2, 2026

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
iii 🟢 Ready View Preview May 2, 2026, 1:16 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@vercel
Copy link
Copy Markdown
Contributor

vercel Bot commented May 2, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
iii-website Ready Ready Preview, Comment May 2, 2026 1:16pm

Request Review

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 2, 2026

📝 Walkthrough

Walkthrough

A comprehensive implementation plan document for a nightly cron-triggered GitHub Actions workflow that invokes Claude Code Action, extracts findings via Python parsing, persists results as markdown and JSON artifacts, and posts Slack notifications. Includes 10 sequential tasks with test specifications, validation steps, and a pre-merge checklist.

Changes

Verdict Review Workflow Implementation Plan

Layer / File(s) Summary
Planning & Architecture
docs/superpowers/plans/2026-05-01-verdict-review-workflow.md (lines 1–49)
High-level workflow purpose, architecture overview (cron job, prompt loading, Claude invocation, extraction, artifact upload, Slack messaging), file structure, and pre-flight operator constraints (stdlib-only Python, Python/pytest/actionlint/jq/gh tooling).
Extractor Implementation & Happy Path
docs/superpowers/plans/2026-05-01-verdict-review-workflow.md (lines 132–355)
Task 1–2 scaffolding: create .github/scripts/, .github/verdict-prompt.md with JSON schema, and extract_verdict.py. Specify extractor logic for parsing Claude transcript, extracting JSON contract tail, byte-size-capped body, and writing verdict.md and verdict.json with fallback behavior and exit code 0.
Test Coverage (Error & Edge Cases)
docs/superpowers/plans/2026-05-01-verdict-review-workflow.md (lines 358–542)
Tasks 3–6: specify pytest suite for malformed JSON, missing JSON tail, missing/empty/null execution file, mid-body noise, 50KB truncation with marker, and goal-state (clean verdict with score=100, findings=[]) regression test.
Workflow Definition & Slack Integration
docs/superpowers/plans/2026-05-01-verdict-review-workflow.md (lines 599–766)
Task 7: author .github/workflows/verdict-review.yml with schedule, permissions, concurrency, prompt injection, prior-run delta computation, Claude Code Action invocation, extraction, artifact upload (90-day retention), and conditional Slack success/failure messaging.
Validation & Documentation Update
docs/superpowers/plans/2026-05-01-verdict-review-workflow.md (lines 769–871)
Tasks 8–9: run actionlint to validate workflow YAML and fix flags; update .github/workflows/WORKFLOWS.md with new workflow section and ANTHROPIC_API_KEY secret row.
Pre-Merge & Deployment Checklist
docs/superpowers/plans/2026-05-01-verdict-review-workflow.md (lines 874–1025)
Task 10: manual verification including extractor test execution, actionlint re-validation, secret configuration, dry-run success/failure paths, schedule confirmation post-merge, and deferred out-of-scope items.

Estimated Code Review Effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 A blueprint for verdicts, so nightly and bright,
Claude whispers through workflows from dawn until night,
Python extracts wisdom, and Slack spreads the word,
Ten tasks mapped out clear—implementation absurd!
The checklist awaits, and the artifacts gleam,
✨ A code-review dream! ✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: adding a documentation plan for a nightly AI verdict review workflow.
Description check ✅ Passed The description covers all required template sections with substantive content, clearly explains the plan scope and intent, and addresses reviewer concerns.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch docs/verdict-review-plan

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
Review rate limit: 7/8 reviews remaining, refill in 7 minutes and 30 seconds.

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@docs/superpowers/plans/2026-05-01-verdict-review-workflow.md`:
- Around line 604-605: Replace the private local spec reference
`~/.claude/plans/verdict-i-d-treat-cheerful-starlight.md` (appears as
`~/.claude/...` in the YAML) with a repository-tracked source or inline the
required constraints: search for the literal
`~/.claude/...`/`~/.claude/plans/verdict-i-d-treat-cheerful-starlight.md`,
update those occurrences to point to a repo-relative path (or paste the
constraint block directly into the plan), and ensure the new reference is
accessible to other reviewers and matches the original constraints.
- Around line 758-759: Update the sanity-check expectation that currently reads
“4 `uses:` references” to “5 `uses:` references”; specifically adjust the text
that lists expected actions so the count includes both Slack steps (checkout@v4,
claude-code-action@v1, upload-artifact@v4, slack-github-action@v2.0.0 twice),
ensuring the documented `uses:` total matches the listed actions.
- Around line 32-35: Replace the hard-coded cd line ("cd
/Users/ytallolayon/workspaces/personal/motia/iii") with a repo-agnostic
approach: either remove the absolute path and assume the user runs the script
from the repo root, or programmatically resolve the repository root (e.g., using
git to determine the top-level directory) and change directory to that instead;
update the subsequent commands (git checkout main && git pull and git checkout
-b feat/verdict-review-workflow) to run relative to the resolved repo root.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 51a4d05c-2556-4f59-8594-f783c02a1f3f

📥 Commits

Reviewing files that changed from the base of the PR and between 19a6f7e and 1445b05.

📒 Files selected for processing (1)
  • docs/superpowers/plans/2026-05-01-verdict-review-workflow.md

Comment on lines +32 to +35
cd /Users/ytallolayon/workspaces/personal/motia/iii
git checkout main && git pull
git checkout -b feat/verdict-review-workflow
```
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Replace machine-specific repo path with a repo-agnostic command

Line 32 hard-codes a local filesystem path, which makes the plan non-portable for other contributors.

Suggested edit
-cd /Users/ytallolayon/workspaces/personal/motia/iii
+cd <path-to-your-local-iii-repo>
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
cd /Users/ytallolayon/workspaces/personal/motia/iii
git checkout main && git pull
git checkout -b feat/verdict-review-workflow
```
cd <path-to-your-local-iii-repo>
git checkout main && git pull
git checkout -b feat/verdict-review-workflow
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@docs/superpowers/plans/2026-05-01-verdict-review-workflow.md` around lines 32
- 35, Replace the hard-coded cd line ("cd
/Users/ytallolayon/workspaces/personal/motia/iii") with a repo-agnostic
approach: either remove the absolute path and assume the user runs the script
from the repo root, or programmatically resolve the repository root (e.g., using
git to determine the top-level directory) and change directory to that instead;
update the subsequent commands (git checkout main && git pull and git checkout
-b feat/verdict-review-workflow) to run relative to the resolved repo root.

Comment on lines +604 to +605
The YAML below is final per the eng review (D1=leak-count-only, D2=Slack failure ping, D3=nightly+Opus). Every line is intentional — do not "tidy up" without re-checking against the spec at `~/.claude/plans/verdict-i-d-treat-cheerful-starlight.md`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick win

Avoid private local spec paths; point to repo-tracked source

Lines 604 and 997 depend on ~/.claude/..., which is not accessible/reproducible for other reviewers. The plan should reference a repository path or embed the required constraints inline.

Suggested edit
-The YAML below is final per the eng review (D1=leak-count-only, D2=Slack failure ping, D3=nightly+Opus). Every line is intentional — do not "tidy up" without re-checking against the spec at `~/.claude/plans/verdict-i-d-treat-cheerful-starlight.md`.
+The YAML below is final per the eng review (D1=leak-count-only, D2=Slack failure ping, D3=nightly+Opus). Every line is intentional — do not "tidy up" without re-checking against this plan and the repo-tracked workflow documentation.

-After completing all 10 tasks, verify against the spec at `~/.claude/plans/verdict-i-d-treat-cheerful-starlight.md`:
+After completing all 10 tasks, verify against this plan and `.github/workflows/WORKFLOWS.md`:

Also applies to: 997-998

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@docs/superpowers/plans/2026-05-01-verdict-review-workflow.md` around lines
604 - 605, Replace the private local spec reference
`~/.claude/plans/verdict-i-d-treat-cheerful-starlight.md` (appears as
`~/.claude/...` in the YAML) with a repository-tracked source or inline the
required constraints: search for the literal
`~/.claude/...`/`~/.claude/plans/verdict-i-d-treat-cheerful-starlight.md`,
update those occurrences to point to a repo-relative path (or paste the
constraint block directly into the plan), and ensure the new reference is
accessible to other reviewers and matches the original constraints.

Comment on lines +758 to +759
Expected: ~115 lines; 4 `uses:` references (`checkout@v4`, `claude-code-action@v1`, `upload-artifact@v4`, two of `slack-github-action@v2.0.0`).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Fix the uses: count in the sanity-check expectation

Line 758 says “4 uses: references” but lists five actions (including two Slack steps). This can cause false confusion during verification.

Suggested edit
-Expected: ~115 lines; 4 `uses:` references (`checkout@v4`, `claude-code-action@v1`, `upload-artifact@v4`, two of `slack-github-action@v2.0.0`).
+Expected: ~115 lines; 5 `uses:` references (`checkout@v4`, `claude-code-action@v1`, `upload-artifact@v4`, and two `slack-github-action@v2.0.0` steps).
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
Expected: ~115 lines; 4 `uses:` references (`checkout@v4`, `claude-code-action@v1`, `upload-artifact@v4`, two of `slack-github-action@v2.0.0`).
Expected: ~115 lines; 5 `uses:` references (`checkout@v4`, `claude-code-action@v1`, `upload-artifact@v4`, and two `slack-github-action@v2.0.0` steps).
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@docs/superpowers/plans/2026-05-01-verdict-review-workflow.md` around lines
758 - 759, Update the sanity-check expectation that currently reads “4 `uses:`
references” to “5 `uses:` references”; specifically adjust the text that lists
expected actions so the count includes both Slack steps (checkout@v4,
claude-code-action@v1, upload-artifact@v4, slack-github-action@v2.0.0 twice),
ensuring the documented `uses:` total matches the listed actions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant