Skip to content

Night Watch Component Auditor

Jump to bottom
cixzhang edited this page Aug 11, 2026 · 24 revisions

Night Watch — Component Auditor

Assigned to: Cindy's Navi (cixzhang)

Goal: Ensure all components follow Astryx conventions — theming compatibility, API consistency, accessibility contracts, and export hygiene.

Source of truth documents:

⚠️ Keep in sync. When updating conventions in those pages, update the corresponding check in this auditor. When the auditor finds a pattern not covered by those pages, add it there first, then update the check here.

Why This Role Exists

Component quality regresses silently. A hardcoded color, a missing themeProps, a boolean prop without is prefix, a missing displayName — these don't break tests or CI but they break the developer and theming contracts. This role catches the drift across all dimensions in a single pass per component.

Scope

Each night this role takes one component and runs a full audit of it against the Component Audit Rubric — every section, including the rendered-design half, which means driving a real browser and capturing screenshots. Then it fixes what it can, re-audits to prove the fixes worked, and records the result.

One component done properly, rather than five skimmed. A pass that only runs the mechanical checks finds the drift a linter could have found; the expensive half — does this state actually render, is this reachable by keyboard, does the hover do anything — is where the real defects are, and it is worth a night.

The checks themselves are not restated here. The rubric is the single source for what is checked, how it is verified, and how it is scored. This page owns only the nightly mechanics: how the component is chosen, how fixes are made and proven, and how the result is recorded.

The [Component Scores ledger] is the only record of progress. There is no separate state file, no queue kept on the side, no list of completed components. What has been audited, when, under which rubric version, and what is still open — all of it lives in the ledger, and the ledger is what the next night reads. If it is not in the ledger, it did not happen.

The nightly run

1. Pick the component

Ask the ledger:

node scripts/score-ledger.mjs --queue --limit 1

It orders never-audited first, then oldest-audited, then lowest-scoring — so coverage grows before anything gets re-audited, and the weakest components come back around soonest. Take the one it names.

Already audited today? The ledger's lastAudited says so. Stop; do not run twice.

2. Audit it — the full rubric

Work Component Audit Rubric end to end, as a full component grading rather than a mechanical sweep. That means the automated checks and the judgement ones, and the screenshots: build Storybook, drive real Chromium, and capture every state the component has in light and dark. The rendered-design half is graded from pixels or it is reported not measured — never guessed, never scored zero.

Write the scorecard before changing a single line. It is the before half of the night's evidence, and you cannot prove a fix worked without it.

3. Fix what you can

Fix the findings that have a clear right answer — a raw value that should be a token, a missing themeProps, a prop name that breaks convention, an unguarded hover, a missing test. Work in one branch: night-watch-component-audit/<Component>/YYYY-MM-DD.

Leave alone anything that needs a decision rather than a correction: a naming dispute, a new prop, a changed API shape, a visual judgement call, a system-level design question. Those are not yours to settle in a fix pass — carry them in the scorecard and say so in the PR's Needs Review section. The Component Lifecycle routing rules say where each kind goes.

If a finding is real but you cannot fix it safely, that is fine. Leave it in the ledger as an open blocking finding with a one-line explanation. An honest open finding is worth more than a rushed fix.

4. Re-audit, and prove it

Run the audit again on the fixed component. This is the point of the whole exercise: the second score is what tells you the fixes worked, and it is what goes in the ledger.

  • Capture the screenshots again, so you have a genuine before/after pair for anything visual.
  • The score must move. If you fixed something blocking and the score is unchanged, then either the fix does not do what you think or the first audit was wrong. Find out which — do not record a number you cannot explain.
  • If a fix made something else worse, revert it. A nightly pass must never leave a component worse than it found it.

5. Record it

Record the post-fix score, per Recording an audit:

<your scorecard> | node scripts/score-ledger.mjs --record <Component> \
  --from - \
  --ledger /tmp/astryx-wiki/component-scores.json \
  --push

Pipe the scorecard straight in — do not leave JSON files lying around. The ledger is the only stored form; a scorecard on disk is a second copy that will be stale within the hour.

Only record what you actually measured. A section you could not verify is not_measured — never a guess, never a zero. Note the before score in the PR so the improvement is legible, but the ledger holds the current truth, which is the score after your fixes.

Do not file issues. Findings you did not fix live in the ledger row against the component, where the next audit and the scores page will both find them. A finding does not need a ticket to be tracked, and a nightly pass that files a dozen issues a week buries the ones that matter. The only exception is a system-level problem — something affecting many components that no single component's PR can fix. That gets one issue, and every affected ledger row links it.

6. Open the PR, and drive it green

Publish it ready for review, not as a draft. The description carries:

  • the component, the before and after scores, and what moved;
  • before/after screenshots for anything visual;
  • the findings you fixed, each citing its rubric rule id;
  • a Needs Review section for everything you deliberately did not fix, and why.

Then drive CI to green yourself: run pnpm test, pnpm build and pnpm lint:strict locally before pushing — CI is not your test runner. On failure, check the branch HEAD is still the sha you pushed (if it changed, someone else is working on it — stop), fix, and push again. Three attempts, then stop and leave it for a human.

Does not do

  • File hardening issues. Unfixed findings live in the ledger row, not in a ticket queue. Only a genuinely system-level problem gets an issue.
  • Keep its own state. The ledger is the record of progress — there is no state file to update.
  • Settle judgement calls. Naming, API shape, new capability, and visual taste route to the Component Specification Protocol, a vibe test, or Component Hardening Protocol — never resolved in a fix pass.
  • Batch components. One a night, audited properly.
  • Review other people's PRs — that is Night Watch Reviewer.

Clone this wiki locally