Design for review — nothing here is built yet. Phase 1 targets the sandbox fork only; humans merge everything the factory produces, always.
- Inference: dedicated API keys (Anthropic + OpenAI) as repo secrets on the sandbox fork; per-provider usage alerts. Model tiering: judgment work on top-tier models, triage/classification on cheap tiers.
- Guardrails are part of every prompt:
docs/AI_GUARDRAILS.mdis injected verbatim. Two rules are load-bearing here: never weaken an assertion, validation, or auth check to make something pass, and findings in the human-owned domains (security, availability, scalability) are flagged, never auto-fixed. - Output is always a draft PR or a PR comment, labeled
ai-factory. The factory has no merge rights.
Trigger: an e2e-blocking or e2e-advisory job fails on a preview run.
Pipeline:
- Collect (plain CI, no AI): bundle the Playwright
error-context.mdaria snapshot, failing spec source, the PR diff, and the run URL into a triage artifact. - Classify (cheap model, structured output): one of
app-bug | test-bug | spec-mismatch | flake | infra, with confidence and a one-paragraph rationale. Dedupe by error fingerprint (normalized failure hash) so a repeated failure never re-bills. - Route:
flake→ append to the flake ledger issue; if it's a blocking-tier test, open a fix-or-demote PR proposal (theE2E_TIERS.mdone-day rule, mechanized).test-bug/spec-mismatch→ open a fix PR against the failing branch, and when the spec is wrong, a companion spec correction. This session already produced three worked examples by hand (the AM/PM empty-state string, the muted-icon Permissions-API assumption, the copied-link deferral) — they become the few-shot examples and the eval set.app-bug→ analysis comment on the PR; a fix PR only at high confidence and never in auth/RBAC paths.infra→ comment + ping (no automation).
- Report: every action links the classification rationale.
Controls: daily run cap; fingerprint dedupe; precision metric tracked weekly (% of factory PRs merged without rework) — if precision drops, the factory demotes itself to comment-only.
Trigger: PR labeled ai-review (opt-in in phase 1; all sandbox PRs in
phase 2). Complements CodeRabbit, which stays for code smells.
Pipeline:
- Reviewer A (Anthropic, top tier, spec-anchored): loads the relevant
specs/*.md,AI_GUARDRAILS.md, and the diff. Hunts violations of acceptance criteria, the Never-Do list, and RBAC expectations from the spec's authorization table. - Reviewer B (OpenAI, top tier, code-anchored): sees only the diff + surrounding code. Hunts logic bugs, injection/validation gaps, perf footguns (N+1, unbounded queries).
- Judge (cheap tier): merges A+B findings, drops duplicates — including anything CodeRabbit already commented — ranks by severity, discards style nits, caps at N findings.
- Output: one PR review with inline comments, each tagged
[A],[B], or[A+B](independent agreement = highest confidence). Findings touching security/availability/scalability get ahuman-review-requiredtag instead of a suggested patch.
Why two independent models: different vantage points (spec-down vs code-up) and different failure modes; agreement is signal, and neither sees the other's output until the judge stage — that's the "adversarial" part.
Cost envelope: roughly 2–6 top-tier calls per reviewed PR; estimated well under a dollar per typical PR, capped by the label opt-in in phase 1.
- Phase 1 (sandbox): F3-62 behind the
ai-reviewlabel; F3-61 collector + classification comment only (no auto-PRs yet). - Phase 2 (sandbox): F3-61 fix-PRs for
test-bug/flakeclasses; F3-62 on all non-draft sandbox PRs. - Phase 3: port with the roll-up; F3 org owns keys/budget decisions.
All merges; any suggestion touching auth/RBAC; provider spend alerts; the weekly precision review. The factory's job is to shrink human review to judgment calls — not to remove it.