Skip to content

Night Watch Component Auditor

Jump to bottom
cixzhang edited this page Aug 11, 2026 · 24 revisions

Night Watch — Component Auditor

Assigned to: Cindy's Navi (cixzhang)

Goal: Ensure all components follow Astryx conventions — theming compatibility, API consistency, accessibility contracts, and export hygiene.

Source of truth documents:

  • API Conventions — API rules (naming, structure, composition, a11y, exports)
  • Theming Infrastructure — Theming rules (tokens, themeProps, variant maps, sub-element targeting)
  • Design Conventions — Visual/interaction rules. Audit only the objectively checkable conventions (the 🟢 rows in its "What's checkable, and where it's checked" table); leave subjective visual judgment to Component Audit Rubric §5b.

⚠️ Keep in sync. The checks live in the Component Audit Rubric, not here. When a convention changes on one of those pages, update the rubric check that cites it. When a pass finds a pattern no page covers, add it to the owning page first, then give the rubric a check for it.

Why This Role Exists

Component quality regresses silently. A hardcoded color, a missing themeProps, a boolean prop without is prefix, a missing displayName — these don't break tests or CI but they break the developer and theming contracts. This role catches the drift across all dimensions in a single pass per component.

Scope

Each night this role takes one component and runs a full audit of it against the Component Audit Rubric — every section, including the rendered-design half, which means driving a real browser and capturing screenshots. Then it fixes what it can, re-audits to prove the fixes worked, and records the result.

One component done properly, rather than five skimmed. A pass that only runs the mechanical checks finds the drift a linter could have found; the expensive half — does this state actually render, is this reachable by keyboard, does the hover do anything — is where the real defects are, and it is worth a night.

The checks themselves are not restated here. The rubric is the single source for what is checked, how it is verified, and how it is scored. This page owns only the nightly mechanics: how the component is chosen, how fixes are made and proven, and how the result is recorded.

The score ledger — component-scores.json in this wiki — is the only record of progress. There is no separate state file, no queue kept on the side, no list of completed components. What has been audited, when, under which rubric version, and what is still open — all of it lives in the ledger, and the ledger is what the next night reads. If it is not in the ledger, it did not happen.

The nightly run

1. Pick the component

Read the ledger — component-scores.json in this wiki — and pick against the live roster of components in packages/core/src and packages/lab/src. The order is never-audited first, then oldest-audited, then lowest-scoring, so coverage grows before anything gets re-audited and the weakest components come back around soonest. Take the first one.

node scripts/score-ledger.mjs --queue --limit 1

Already audited today? The ledger's lastAudited says so. Stop; do not run twice.

2. Audit it — the full rubric

Work Component Audit Rubric end to end, as a full component grading rather than a mechanical sweep. That means the automated checks and the judgement ones, and the screenshots: build Storybook, drive real Chromium, and capture every state the component has in light and dark. The rendered-design half is graded from pixels or it is reported not measured — never guessed, never scored zero.

Write the scorecard before changing a single line. It is the before half of the night's evidence, and you cannot prove a fix worked without it.

3. Fix what you can

Fix the findings that have a clear right answer — a raw value that should be a token, a missing themeProps, a prop name that breaks convention, an unguarded hover, a missing test. Work in one branch: night-watch-component-audit/<Component>/YYYY-MM-DD.

Leave alone anything that needs a decision rather than a correction: a naming dispute, a new prop, a changed API shape, a visual judgement call, a system-level design question. Those are not yours to settle in a fix pass — carry them in the scorecard and say so in the PR's Needs Review section. The Component Lifecycle routing rules say where each kind goes.

If a finding is real but you cannot fix it safely, that is fine. Leave it in the ledger as an open blocking finding with a one-line explanation. An honest open finding is worth more than a rushed fix.

4. Re-audit, and prove it

Run the audit again on the fixed component. This is the point of the whole exercise: the second score is what tells you the fixes worked, and it is what goes in the ledger.

  • Capture the screenshots again, so you have a genuine before/after pair for anything visual.
  • The score must move. If you fixed something blocking and the score is unchanged, then either the fix does not do what you think or the first audit was wrong. Find out which — do not record a number you cannot explain.
  • If a fix made something else worse, revert it. A nightly pass must never leave a component worse than it found it.

5. Record it

Record the post-fix score, per Recording an audit:

git clone https://github.com/facebook/astryx.wiki.git /tmp/astryx-wiki

<your scorecard JSON> | node scripts/score-ledger.mjs --record <Component> \
  --from - --ledger /tmp/astryx-wiki/component-scores.json --push

--push commits and pushes the wiki for you and prints the commit URL. Pipe the scorecard in rather than writing it to a file — the ledger is the only stored form, and a scorecard on disk is a second copy that goes stale within the hour. Use --dry-run first if you want to see the diff.

Your result appears on the Component Scores page without a rebuild — it fetches the ledger at runtime.

Only record what you actually measured. A section you could not verify is not_measured — never a guess, never a zero. Note the before score in the PR so the improvement is legible, but the ledger holds the current truth, which is the score after your fixes.

Do not file issues. Findings you did not fix live in the ledger row against the component, where the next audit will find them. A finding does not need a ticket to be tracked, and a nightly pass that files a dozen issues a week buries the ones that matter. The only exception is a system-level problem — something affecting many components that no single component's PR can fix. That gets one issue, and every affected ledger row links it.

6. Open the PR, and drive it green

Publish it ready for review, not as a draft. The description carries:

  • the component, the before and after scores, and what moved;
  • before/after screenshots for anything visual;
  • the findings you fixed, each citing its rubric rule id;
  • a Needs Review section for everything you deliberately did not fix, and why.

Then drive CI to green yourself: run pnpm test, pnpm build and pnpm lint:strict locally before pushing — CI is not your test runner. On failure, check the branch HEAD is still the sha you pushed (if it changed, someone else is working on it — stop), fix, and push again. Three attempts, then stop and leave it for a human.

Does not do

  • File hardening issues. Unfixed findings live in the ledger row, not in a ticket queue. Only a genuinely system-level problem gets an issue.
  • Keep its own state. The ledger is the record of progress — there is no state file to update.
  • Settle judgement calls. Naming, API shape, new capability, and visual taste route to the Component Specification Protocol, a vibe test, or Component Audit Rubric — never resolved in a fix pass.
  • Batch components. One a night, audited properly.
  • Review other people's PRs — that is Night Watch Reviewer.

Clone this wiki locally