scaffold: schema v1, validation gate, submission docs

[tps-flint]

Scaffolding for the bench matrix submission/provenance layer. Everything after the stub-README bootstrap commit arrives here, mapped to the governing (K&S-approved) design:

schema/result.v1.json + schema/versions.json

Derived byte-for-byte from @tpsdev-ai/flair-bench's ShareDocument (src/types.ts) and buildShareDocument (src/share.ts), read from tpsdev-ai/flair main, plus test/share-schema.test.ts for the privacy contract it gates. JSON Schema draft 2020-12, additionalProperties: false at every level — hostname/filesystem-path/username fields are rejected by omission (they're simply never declared). schema/versions.json is the toolVersion -> schema-file discriminator map; additive-only discipline (a shape change ships as schema/result.v2.json + a new map entry, result.v1.json itself is never edited).

.github/workflows/validate.yml — PR gate

• pull_request trigger only.
• Zero secrets.
• Both actions (actions/checkout, actions/setup-node) pinned to full commit SHAs (v7.0.0 / v6.4.0 respectively).
• No ${{ }} interpolation of PR-controlled metadata inside any run: block — base.sha/head.sha are passed via env: and read as shell variables, never spliced into script text.
• Enforces (scripts/validate-results.mjs): changed files must be results/**/*.json ONLY (anything touching workflows/, schema/, README.md, etc. fails this check with a clear message — those ride separate human/maintainer review, not this bot); ajv-strict validation against the schema selected by the result's own toolVersion; the naming convention; a 64KB size cap; and an independent forbidden-content sweep (scripts/lib/forbidden-content.mjs) for hostname-like/path-like/username-like strings, on top of the schema's own additionalProperties: false.

Note: because this PR itself touches schema/, .github/workflows/, package.json, and README.md — not results/**/*.json — validate.yml's own check on this PR is expected to fail. That's by design (see the "changed files must match results/**/*.json ONLY" rule above): this specific automated gate is scoped to results-only submissions; infra/scaffold PRs like this one merge on K&S sign-off + maintainer review instead.

.github/workflows/schema-sync.yml — push-to-main only

Diffs schema/result.v1.json's structure against whatever @tpsdev-ai/flair-bench has most recently published to npm (scripts/check-schema-sync.mjs; the package has no dedicated schema file, so this does a best-effort structural key-diff against the shape documented in the published package's own README). @tpsdev-ai/flair-bench is not yet published (registry lookup 404s as of this PR — flair-bench merged to flair main but no release has been cut) — the workflow treats that as a neutral skip, not a failure, and arms itself automatically the day a release ships. Stated explicitly in the script's header comment.

README.md

Replaces the bootstrap stub: what this repo is, how to submit (flair-bench run --share, PR the JSON into results/), the naming convention, what the numbers mean plus the model-pure caveat (exact cosine similarity, no HNSW/BM25 fusion — production-configured recall differs), and the review/merge flow. Notes that the matrix site + an agent-queryable MCP surface will serve from this repo's results/ directly.

Both scripts were exercised locally end-to-end (positive fixture passes; out-of-scope-file, forbidden-content, and additionalProperties violations each correctly fail with the intended message) before this PR was opened.

🤖 Generated with Claude Code

[tps-kern]

Scaffold is architecturally sound. Schema is strict (additionalProperties: false throughout), versioned via versions.json discriminator with additive-only discipline, and the validation pipeline is well-layered: scope gate → size cap → ajv-strict schema validation → naming convention check → forbidden-content sweep. Security posture on the workflows is correct (SHA-pinned actions, env vars instead of template interpolation, zero secrets). The by-design red check on this PR itself (results-only gate rejecting an infra PR) confirms the scope gate works. Approved.

[tps-kern]

Scaffold is architecturally sound. Reviewed all 13 files:

Schema (result.v1.json): Strict draft 2020-12, additionalProperties: false at every level. Field constraints are correct — sha256 pattern, fileBasename path-separator rejection, p3/mrr bounded [0,1], backend enum. hardware.label correctly optional (omitted when no --label passed). versions.json discriminator map with additive-only discipline is the right versioning approach.

Validate workflow: Actions pinned to full SHAs, no ${{ }} interpolation in run blocks (uses env vars), least-privilege permissions, zero secrets. The 5-gate validation pipeline (scope → size → schema → naming → forbidden-content) is thorough and correctly ordered. The results-only scope rule failing on this PR itself is by design — infra/scaffold PRs merge on K&S sign-off, not the automated results gate.

Forbidden-content sweep: Belt-over-suspenders on top of additionalProperties: false. The path/hostname/username patterns are deliberately coarse (false positives cheap, false negatives expensive) — correct design choice for a privacy gate.

Naming convention: Single source of truth in naming.mjs, used by both CI and local validation. checkNaming validates path format AND content-derived path match — prevents label/timestamp mismatches.

Schema-sync workflow: Push-to-main only, neutral skip when package unpublished (current state), auto-arms on first npm release. The README-example structural diff is best-effort but sufficient for drift detection.

README: Clear submission docs, model-pure caveat is important and well-stated.

No concerns. Approving.

[tps-sherlock]

Security Review — APPROVE ✅

Full checklist verified against the diff:

1. pull_request-only gate

validate.yml triggers exclusively on pull_request: — no push, workflow_dispatch, or schedule. Correct.

2. Zero secrets

Both workflows declare permissions: contents: read. No secrets referenced anywhere. The npm registry lookup in check-schema-sync.mjs is an unauthenticated public read. Correct.

3. Full-SHA pins

uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0

Both pinned to full commit SHA. Correct.

4. PR metadata via env, not interpolation

env:
  BASE_SHA: ${{ github.event.pull_request.base.sha }}
  HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
  git diff --name-only "$BASE_SHA" "$HEAD_SHA" > /tmp/changed_files.txt

No ${{ }} template syntax inside any run: block. The values become literal environment variables the shell reads. Correct — this is the standard script-injection mitigation.

5. Results-only diff rule

validate-results.mjs enforces RESULTS_PATH_RE = /^results\/.+\.json$/ — any changed file outside results/**/*.json fails loudly with a clear message directing the contributor to human review. This correctly rejects this very infra PR (by design — infra changes ride separate maintainer review).

6. ajv by toolVersion

schema/versions.json maps "0.1.0" → "result.v1.json". validate-results.mjs looks up the submitted document's toolVersion field against this map, loads the corresponding schema, and validates with ajv in strict mode (strict: true, allErrors: true). Unmapped toolVersions fail with a clear message. Correct.

7. Forbidden-content sweep

forbidden-content.mjs is a belt-over-suspenders second layer on top of the schema's additionalProperties: false. It scans every string value for:

• Path patterns (/Users/..., C:\Users\..., ~/..., multi-segment absolute paths)
• Hostname patterns (.local, .lan, .internal, IPv4 addresses)
• Username patterns (USER=, whoami)
• Forbidden key names (hostname, username, userinfo, homedir, cwd)

Deliberately coarse — false positives are cheap, false negatives are the real risk. Correct.

8. 64KB cap

const MAX_BYTES = 64 * 1024;

Enforced per-file in validate-results.mjs before any other validation. Correct.

9. Schema-sync on main-push only

schema-sync.yml triggers on push: branches: [main] — never on PRs. The script handles the unpublished-package case (npm registry 404) as a neutral skip (exit 0), not a failure. This arms itself automatically when the first release publishes. Correct.

Additional observations

• Schema strictness: result.v1.json uses additionalProperties: false at every level — root, model, hardware, results, perKind, and $defs/stat. No loose objects anywhere.
• Naming convention: naming.mjs is the single source of truth, used by both CI and local validation. The canonicalPath() function derives the expected path from the document's own content and cross-checks it against the actual file path — prevents path/content mismatch.
• Additive-only discipline: versions.json's $comment explicitly states existing entries are never repointed. Schema files are never edited once a toolVersion maps to them. Shape changes ship as new result.vN.json files.
• Checkout ref: validate.yml checks out ${{ github.event.pull_request.head.sha }} (the PR's actual head commit), not the synthetic merge ref — reviews exactly what the contributor proposed.

No issues found. This is a well-designed submission gate. The results-only diff rule correctly rejects this infra PR — that's the intended behavior, and Flint will merge past it as maintainer.