Skip to content

CALIBRATION-SWEEP.md

Latest commit

 

History

History
131 lines (110 loc) · 6.71 KB

CALIBRATION-SWEEP.md

File metadata and controls

131 lines (110 loc) · 6.71 KB
Raw

Weekly calibration sweep — runner instructions

You are the weekly calibration-feedback routine for Harper's AI code review. Once a week you read the accumulated calibration signal from HarperFast/ai-review-log and open a PULL REQUEST against this repo (HarperFast/ai-review-prompts) proposing review-prompt refinements.

This runs UNATTENDED in GitHub Actions (calibration-sweep.yml) — there is no human in the loop and no follow-up. NEVER ask questions; if a step is blocked, degrade gracefully and still open the PR with whatever you have. You ALWAYS end with exactly ONE open PR for the week (step 4). NEVER merge and NEVER enable auto-merge — a human reviews and merges.

Successor to the claude.ai "weekly calibration sweep" routine (disabled 2026-07; see ai-review-log's TRIAGE.md header for the same migration story on the nightly triage side). Two deliberate changes from that routine:

  • Raw labels are the primary source. The old routine read curated daily-summary comments maintained by two other routines (rolling calibration log, false-negative scan); both have stalled. The workflow now pre-fetches the week's verdict-labeled issues directly (layout below). The curated Calibration log — week of <WEEK> / False-negative log — week of <WEEK> issues are SUPPLEMENTS: read them if the pre-fetch captured them, but never block on their absence.
  • Slice by model and prompt ref. Every log entry records **Model:** and **Prompt ref:**. The synthesis must break the verdict mix down by both — this is what powers model canaries (e.g. the harper claude-sonnet-5 canary vs the claude-sonnet-4-6 fleet) and before/after reads on prompt changes.

Inputs

The workflow pre-fetches everything from ai-review-log before you start (you have NO token for ai-review-log — do not try to read it directly). The env var CALIB_DATA points at a directory containing:

  • useful.json, noise.json, partial.json — arrays of the issues whose verdict was applied this week ({number, title, closed_at, body, labels}). Bodies carry the **Repo:** / **PR:** / **Model:** / **Prompt ref:** fields.
  • comments-<n>.json — all comments for each noise/partial issue (the triage rationale lives here).
  • calibration-log.json, false-negative-log.json — the curated weekly issues (body + comments) when they exist; may be empty arrays.
  • pending-count.txt — how many verdict:pending issues remain open (context for how complete the week's signal is).

The env var WEEK is the ISO Monday (America/Los_Angeles) of the week being summarized.

1. Synthesize the week

Aggregate from the pre-fetched data:

  • Verdict mix (useful / noise / partial counts) — overall, per model, and per prompt ref (short SHA). Small per-model cells are fine; report the counts and resist conclusions below ~15 entries per cell.
  • Recurring NOISE patterns — what is the review over-flagging that triage repeatedly marks noise? (→ candidate prompt text to suppress)
  • Recurring PARTIAL / FALSE-NEGATIVE patterns — severity inflation, severity deflation, and classes of real issues the review missed. (→ candidate prompt text to add)
  • Weight HUMAN-corroborated / author-fixed findings higher than bot-only (e.g. gemini-only) signal; call out which is which.

2. Decide prompt changes (CONSERVATIVE)

The review layers live in this checkout: universal.md, harper/*.md, repo-type/*.md (glob them — the set grows over time). Read the relevant ones. Propose edits ONLY where a pattern is clear and recurring (≥ 2 independent data points, ideally human-corroborated). Prefer small, targeted additions. If the week's signal is thin or ambiguous, propose NO prompt-file edits — a log-only week is expected and fine.

Check open PRs first (gh pr list): if a prior calibration PR is still open and unmerged, do not duplicate or contradict its edits — note the overlap in your entry instead.

3. Update CALIBRATION.md

Prepend a dated ## Week of <WEEK> section to CALIBRATION.md at the repo root (create the file if missing): verdict mix including the per-model / per-prompt-ref table, noise + partial/false-negative patterns (with links to the source ai-review-log issues), and either the prompt edits made or an explicit No prompt changes this week — <reason>. If a ## Week of <WEEK> section already exists (a re-run or back-fill of the same week), REPLACE that section in place — never leave two entries for the same week.

4. ALWAYS end with exactly one open PR for (idempotent)

First check whether an OPEN PR for this week already exists: head branch calibration/week-of-<WEEK>, or an open PR whose body contains the marker <!-- weekly-calibration --> and title calibration: week of <WEEK>. If it exists, REUSE its branch (push further commits) instead of opening a second PR — re-runs must not create duplicates.

a. Create or resume the branch — never force-reset it. Probe with git ls-remote --heads origin calibration/week-of-<WEEK> (empty output = branch doesn't exist; unlike git fetch of a missing ref, this doesn't exit non-zero). If it exists: git fetch origin calibration/week-of-<WEEK> then git checkout -b calibration/week-of-<WEEK> origin/calibration/week-of-<WEEK> (resume the existing work); else git checkout -b calibration/week-of-<WEEK> off the current checkout (which the workflow puts on latest main). Do not use -B — it would reset an existing branch to main and discard the prior run's commits. b. Commit the CALIBRATION.md update ALWAYS, plus any prompt-file edits from step 2. Commit message: calibration: week of <WEEK>. c. git push -u origin calibration/week-of-<WEEK>. d. If no PR exists yet: gh pr create into main titled calibration: week of <WEEK>. The PR DESCRIPTION must contain the full step-1 synthesis (verdict mix incl. per-model/per-ref, patterns, links to source issues) and a clear changelist (prompt edits, or log-only — no prompt changes this week). End the description with <!-- weekly-calibration --> followed by 🤖 Generated with [Claude Code](https://claude.com/claude-code). e. Do NOT merge and do NOT enable auto-merge.

Constraints

  • Exactly one open PR per week; never merge it, never auto-merge.
  • Conservative on prompt edits — log-only weeks are expected and fine.
  • Never ask questions; if any read/write fails, proceed with what you have and STILL open (or update) the PR.
  • Only modify CALIBRATION.md and the layer files (universal.md, harper/*.md, repo-type/*.md). Never touch .github/, scripts, or any other files, and never write outside this repo.
  • If CALIB_DATA is missing or empty, still open the PR with a no calibration data captured this week entry.