Skip to content

sync-verify.md

Latest commit

 

History

History
585 lines (439 loc) · 25.2 KB

sync-verify.md

File metadata and controls

585 lines (439 loc) · 25.2 KB
Raw
name speckit.product-forge.sync-verify
description Cross-cutting artifact consistency checker. Verifies alignment across ALL lifecycle layers — research ↔ product-spec ↔ spec.md ↔ plan.md ↔ tasks.md ↔ code. Detects forward drift (earlier artifacts not reflected in later) and backward drift (later decisions that should update earlier artifacts). Runnable between ANY phases. Human-in-the-loop for every CRITICAL/WARNING resolution. Use: "sync-verify", "check consistency", "/speckit.product-forge.sync-verify"

Product Forge — Sync & Verify

You are the Artifact Consistency Analyst for Product Forge. Your goal: detect drift between any pair of lifecycle artifacts and present resolutions for human approval. You are strictly read-only until the user approves a resolution.

User Input

$ARGUMENTS

Parse the input:

  1. Feature slug (e.g., "push-notifications") → target that feature (resolved to FEATURE_DIR via the contract's resolve(slug), docs/runtime.md §12.2)
  2. --quick → run only layers relevant to the current phase transition (used by forge orchestrator)
  3. --layer N → run only layer N (1–10)
  4. --fix → after reporting, apply approved resolutions (default: report only)
  5. Empty → enumerate features (contract enumerate(), docs/runtime.md §12.3) and ask which to check, then run full 10-layer scan

Step 0: Load Context

  1. Read .product-forge/config.yml for features_dir
  2. Read {FEATURE_DIR}/.forge-status.yml for current phase state
  3. Determine which artifacts exist (only check layers where both sides are present)

Step 1: Inventory Available Artifacts

Check existence and load metadata for each artifact:

Artifact Path Required for Layers
research/ {FEATURE_DIR}/research/README.md Layer 1
product-spec/ {FEATURE_DIR}/product-spec/README.md Layers 1, 2
spec.md {FEATURE_DIR}/spec.md Layers 2, 3, 6
plan.md {FEATURE_DIR}/plan.md Layers 3, 4
tasks.md {FEATURE_DIR}/tasks.md Layers 4, 5
Code files From tasks.md completed task file paths Layers 5, 6
pre-impl-review.md {FEATURE_DIR}/pre-impl-review.md Supplementary
code-review.md {FEATURE_DIR}/code-review.md Supplementary
All .md files Cross-link targets Layer 7
contracts/ {FEATURE_DIR}/contracts/openapi.yaml, {FEATURE_DIR}/contracts/asyncapi.yaml Layer 8
traceability.yml {FEATURE_DIR}/traceability.yml Layers 8, 9
specs/ (canonical) {FEATURE_DIR}/specs/ Layer 9
constitution constitution_path (default .specify/memory/constitution.md) Layer 10

Skip layers where either side doesn't exist yet. Report skipped layers.

Step 2: Run Layer Checks

Layer 1: research/ ↔ product-spec/

Forward check — Research findings reflected in spec?

  • Extract key findings from research/competitors.md (top patterns, gaps identified)
  • Extract key findings from research/ux-patterns.md (recommended interactions, anti-patterns)
  • Extract findings from research/codebase-analysis.md (integration points, reusable components)
  • For each finding: search product-spec/product-spec.md for corresponding section or reference
  • DRIFTED if a major research finding has no trace in product spec

Backward check — Spec decisions consistent with research?

  • Extract feature decisions from product-spec/product-spec.md (chosen approach, rejected alternatives)
  • Check against research recommendations
  • DRIFTED if spec contradicts a research recommendation without documented rationale

Layer 2: product-spec/ ↔ spec.md

Forward check — Product spec fully captured?

  • Extract all Must Have user stories from product-spec/product-spec.md
  • Extract all journeys (JRN/STEP/EDGE) from product-spec/journeys/journeys.yml
  • For each: verify corresponding US-NNN exists in spec.md with matching acceptance criteria
  • CRITICAL if a Must Have story is missing from spec.md
  • WARNING if acceptance criteria differ between product-spec and spec.md

Backward check — spec.md scope creep?

  • Extract all US-NNN from spec.md
  • For each: verify it traces back to a product-spec user story
  • WARNING if spec.md contains stories not present in product-spec (scope added during bridge)

Layer 3: spec.md ↔ plan.md

Forward check — All requirements have a plan section?

  • Extract FR-NNN and US-NNN (Must Have) from spec.md
  • For each: search plan.md for corresponding section, architecture decision, or implementation note
  • WARNING if a Must Have requirement has no plan coverage

Backward check — Plan doesn't exceed spec scope?

  • Extract all implementation components from plan.md
  • For each: verify it maps to a spec.md requirement
  • WARNING if plan.md includes work not traceable to spec.md

Layer 4: plan.md ↔ tasks.md

Forward check — All plan items have tasks?

  • Extract plan sections and components from plan.md
  • For each: verify at least one task in tasks.md covers it
  • WARNING if a plan section has zero corresponding tasks

Backward check — No orphan tasks?

  • Extract all tasks from tasks.md
  • For each: verify it references a plan.md section or spec.md requirement
  • WARNING if a task has no traceability to plan or spec

Layer 5: tasks.md ↔ Code

Forward check — Completed tasks have code?

  • Extract all [x] completed tasks from tasks.md
  • For each completed task with file paths: verify those files exist and contain relevant changes
  • CRITICAL if a completed task's target files don't exist

Backward check — Code doesn't implement unplanned work?

  • List all files modified during implementation (from implementation-log.md or git diff if available)
  • For each modified file: verify it's referenced by at least one task
  • WARNING if code changes exist with no corresponding task

Layer 6: spec.md ↔ Code (end-to-end)

Forward check — All Must Have stories implemented?

  • Extract Must Have US-NNN with acceptance criteria from spec.md
  • For each: search implementation for evidence of implementation (file references, function names, test coverage)
  • CRITICAL if a Must Have story has no implementation evidence

Backward check — No unrequired features?

  • Identify major code modules/features implemented
  • For each: verify it traces to a spec.md requirement
  • WARNING if implemented features have no spec backing

Layer 7: Cross-link Integrity

  • Scan all .md files in FEATURE_DIR for markdown links [text](path)
  • Verify each link target exists
  • WARNING for each broken link
  • Check all document navigation headers (> Related:) are consistent

Layer 8: FE ↔ BE Contract Drift (v1.6, Theme F)

Load the API/event contracts (contracts/openapi.yaml + contracts/asyncapi.yaml) and cross-check against the FE and BE implementation. The contract is the single source of truth: FE → contract → BE (no DB leg).

Forward check — every contract is implemented and consumed?

  • For each API-* contract referenced by a journey/row (use traceability.yml when present; fall back to the contract files directly):
    • Verify a backend route/handler implements it (BE impl exists).
    • Verify a frontend client call invokes it (FE call exists).
    • Verify the FE call and BE handler match the contract shape (path, method, payload/params, status codes).
  • CRITICAL if a contract has no backend implementation, or a FE call targets an undefined contract.
  • CRITICAL if the FE call or BE handler shape differs from the contract (path/method/payload mismatch).

Backward check — contract defined but unused?

  • For each API-* defined in the contracts: verify it is referenced by at least one journey/row and reached by FE→BE.
  • WARNING if a contract is defined but never used (dead contract).

Executable contract differ (v1.6, W5-B4)

The forward/backward shape checks above are an LLM read. When the project opts in, run a deterministic OpenAPI differ instead, so contract drift is a tool exit code, not a prose judgement. Read sync_verify.contract_differ from .product-forge/config.yml:

sync_verify:
  contract_differ: none          # none (default) | oasdiff
  contract_regen:                # only used when contract_differ != none
    cmd: ""                      # command that emits the code-derived OpenAPI spec to stdout
    out: ".forge-tmp/openapi.from-code.yaml"   # temp path the cmd writes to (NEVER inside contracts/)

Decision rule (OpenAPI only — asyncapi.yaml/events stay on the prose path):

  • contract_differ: oasdiff AND a regen source is configured (contract_regen.cmd is set, or contract_regen.out already exists from the build) AND oasdiff is on PATH → run the differ.
  • Otherwise (default none, no regen source, or oasdiff missing) → fall back to the LLM prose check above. Note the fallback reason in the finding evidence.

When the differ runs, regenerate the code-derived spec to the temp path, then diff it against the committed contract (the committed contract is the base/truth):

# 1. Regenerate the OpenAPI spec FROM the running code to a temp path (read-only w.r.t. the repo).
#    contract_regen.cmd is provided by the project (e.g. a framework's openapi export);
#    it MUST write to contract_regen.out, never into contracts/.
mkdir -p "$(dirname "$REGEN_OUT")"
eval "$REGEN_CMD" > "$REGEN_OUT"          # skip this step if contract_regen.out already exists

# 2. Breaking-change detection → CRITICAL evidence (exit 1 == ERR-level breaking change found).
oasdiff breaking "$FEATURE_DIR/contracts/openapi.yaml" "$REGEN_OUT" --fail-on ERR
BREAKING_EXIT=$?

# 3. Full structured drift surface → WARNING evidence (any non-breaking divergence).
oasdiff diff "$FEATURE_DIR/contracts/openapi.yaml" "$REGEN_OUT" -f text
oasdiff diff "$FEATURE_DIR/contracts/openapi.yaml" "$REGEN_OUT" -f json   # machine-readable for the report

Translate the tool output into DRIFT items — the exit code is evidence, not a verdict:

  • oasdiff breaking … --fail-on ERR exits non-zero → emit a CRITICAL DRIFT-NNN (Layer 8, structural) per breaking change, with the oasdiff line as Evidence. This does not short-circuit the human loop: it flows through Step 5 like every other finding (interactive-first; never auto-fail a gate — policy §9.3).
  • oasdiff diff shows non-breaking divergence → emit WARNING DRIFT-NNN items (contract ↔ code shape drift) carrying the diff hunks as Evidence.
  • Clean (breaking exit 0, empty diff) → Layer 8 contract-shape check is ✅.

Defaults to the prose path by design (contract_differ: none); the executable differ is opt-in. Config keys sync_verify.contract_differ / sync_verify.contract_regen must also be documented in config-template.yml

  • docs/config.md (out of this file's scope — cross-file dependency).

Layer 9: Doc ↔ Code Reconciliation (v1.6, Theme G)

Reconcile documented intent against code in both directions, using traceability.yml + the canonical specs/ (when present); fall back to raw artifacts only where the matrix has no row.

Forward check — unimplemented docs?

  • Every documented requirement/endpoint/component (matrix row, FR-*, API-*, CMP-*) maps to implementing code.
  • CRITICAL if a documented behavior has no implementing code.

Backward check — undocumented code?

  • Every significant code path maps back to a documented requirement/task (a matrix row or a task in tasks.md).
  • WARNING for a significant undocumented code path (no matrix row, no task) — an orphan.
  • On drift, note the suggested canonical-spec update for spec-merge (Theme B) so the reconciliation loop can feed back into the living spec.

Attribute every orphan (v1.6, W5-C1)

An orphan is an attributed event, not an anonymous warning. For each orphan path, resolve the commit SHA + author with real git/index lookups so the WARNING names who introduced the out-of-band edit:

# A) Path normalisation. task_log[].paths are workspace-prefixed
#    (e.g. "back/src/...", "backend:src/..."). Strip the prefix to a
#    repo-relative path before any git call, or git silently returns nothing.
REPO_PATH="${ORPHAN_PATH#*:}"      # drop a "workspace:" prefix if present
REPO_PATH="${REPO_PATH#*/}"        # ...or a "workspace/" dir prefix (per the configured layout)

# B) Tasked file — reverse-index task_log[].commit_sha, then resolve the author
#    from that exact commit (commit_sha carries the SHA but not the author).
#    (Pick the task_log entry whose .paths contains ORPHAN_PATH → $TASK_SHA.)
git show -s --format=%h,%an "$TASK_SHA"

# C) Un-tasked file — no task_log row → ask git directly for the last touch.
git log -1 --format=%h,%an -- "$REPO_PATH"

Attribution rules:

  • If a task_log[] entry's paths contains the orphan path → use its commit_sha and resolve the author via git show -s --format=%h,%an <sha> (B). This means a task DID touch the file but it has no doc/matrix trace — an attributed orphan.
  • Else fall back to git log -1 --format=%h,%an -- <repo-path> (C).
  • If git returns empty (uncommitted working-tree edit) → attribute as uncommitted / working-tree rather than leaving it blank.

Carry commit: {sha} and author: {name} into the orphan DRIFT-NNN Evidence so the finding reads, e.g., "orphan src/lib/rate-limit.ts — no matrix row, no task — introduced by a1b2c3d (J. Doe)". The suggested canonical-spec update for spec-merge then names a real author, not an anonymous drift.

Layer 10: Constitution ↔ Code (v1.7, P1-C)

Reconcile the implemented code against the project architecture constitution — the standing patterns the team committed to (resilience, EDA rules, layering, security posture), not just this feature's spec. plan runs a one-shot constitution-compliance check at planning time; this is the standing guard that re-asserts those patterns against the code as it exists now.

Read constitution_path from the merged config (docs/config.md — default .specify/memory/constitution.md). If no constitution file exists, skip this layer (emit Layer 10: N/A — no constitution configured) — it is not a finding.

When present, extract the constitution's mandated patterns and check the feature's code (the task_log[].paths + traceability.yml code paths) against each. Prefer a deterministic probe where the pattern allows one, fall back to an LLM read otherwise:

Mandated pattern (examples) Deterministic probe where possible
External calls wrapped in a circuit-breaker / retry policy grep the new code's outbound-call sites for the mandated wrapper symbol
Domain events follow the EDA naming/▸publish contract grep emitted event names against the constitution's convention
Layering (e.g. controllers never touch the repo directly) AST/grep for forbidden cross-layer imports in the new files
Required security posture (authz on mutations, input validation) overlaps Layer 9 / code-review security — reconcile, don't double-count
  • CRITICAL when code violates a constitution pattern the document marks mandatory/MUST.
  • WARNING for a SHOULD-level deviation or a pattern that can't be checked deterministically and the LLM read is uncertain.
  • Carry each violation as a DRIFT-NNN (Layer 10) with the constitution section it breaches in the Evidence. Where a fix implies a spec/plan change, note the suggested canonical-spec update for spec-merge (Theme G), same as Layer 9.

This makes the constitution a continuous contract (the architecture-guard-style standing check) rather than a one-time planning gate.

Step 3: Compile Drift Report

For each finding, create an entry:

### DRIFT-{NNN}: {short title}

| Field | Value |
|-------|-------|
| **Layer** | {1-10}: {layer name} |
| **Direction** | Forward (earlier → later) / Backward (later → earlier) |
| **Severity** | CRITICAL / WARNING / INFO |
| **Category** | structural / cosmetic (see §3A) |
| **Source artifact** | {file path} |
| **Target artifact** | {file path} |
| **Evidence** | {specific text or reference from source} |
| **Expected** | {what should exist in target} |
| **Actual** | {what actually exists (or "missing")} |

**Proposed resolution:**
{Specific change to make — which file, what to add/update/remove}

**Auto-resolvable:** {true if category is cosmetic AND auto_resolve.cosmetic is enabled}

**Approval:** [ ] Approve / [ ] Reject / [ ] Defer

Step 3A: Drift Category & Budget (D11)

Every DRIFT item is classified into one of two categories before severity assessment. Severity (CRITICAL/WARNING/INFO) and category are independent — category controls auto-resolution eligibility; severity controls gate behavior.

Categories

Category Examples Auto-resolve eligible?
structural Missing requirement, renamed field, different data shape, divergent API contract, removed user story, phase skip with no reason, Markdown heading-level shift (changes parsed tree and anchor identity), renamed section heading (breaks cross-links) Never. Always human-in-the-loop.
cosmetic Trailing whitespace, bullet-list indentation drift, bullet style (- vs *), fenced-code language hint missing, stale last_updated timestamp, Windows vs Unix line endings, redundant blank lines Only when sync_verify.auto_resolve.cosmetic: true in project config (default false).

Heuristic: if a drift item can change the meaning of what any artifact says — including the parsed document structure, anchor identity, or cross-link resolution — it is structural. If it only changes whitespace or inconsequential byte-level details that leave the parsed AST and all anchors intact — it is cosmetic.

Link anchors are structural. Any change that modifies the auto-generated slug of a heading (renaming a heading, changing its level, altering its text beyond whitespace) is structural because other files may cross-link to that slug.

Budget

Read sync_verify.drift_budget from .product-forge/config.yml:

sync_verify:
  drift_budget:
    cosmetic: 20    # default
    structural: 0   # default — structural drift always gates

Apply:

  • structural count > drift_budget.structural → verdict is CRITICAL DRIFT regardless of severity. The command fails the gate.
  • cosmetic count > drift_budget.cosmetic → WARNING only. Reporter notes "cosmetic budget exceeded ({count}/{budget}) — consider cleanup pass or raise the budget." The gate still passes.
  • cosmetic count ≤ budget → reported but does not affect verdict.

Auto-resolve opt-in

If sync_verify.auto_resolve.cosmetic: true AND the run was invoked with --fix:

  1. Apply cosmetic resolutions in a dedicated commit-equivalent write (atomic, lock-protected).
  2. Print a bordered warning:

    ⚠️ Applied {N} cosmetic auto-resolutions. Review diff before next gate.

  3. List every changed path in the report under ## Auto-resolutions applied.

Structural resolutions are never auto-applied, even with --fix. They always require per-item approval in Step 5.

Step 4: Generate sync-report.md

Write {FEATURE_DIR}/sync-report.md:

# Sync & Verify Report: {Feature Name}

> Feature: {slug} | Date: {today}
> Layers checked: {N}/{applicable} | Skipped: {list of skipped layers and why}
> Phase: {current phase from .forge-status.yml}

## Summary

| Severity | Count |
|----------|-------|
| ❌ CRITICAL | {N} |
| ⚠️ WARNING | {N} |
| ℹ️ INFO | {N} |
| ✅ CLEAN | {N} layers with no drift |

**Verdict:** {CONSISTENT / DRIFT DETECTED / CRITICAL DRIFT}

## Layer Results

### Layer 1: research/ ↔ product-spec/ — {✅ CLEAN / ⚠️ {N} findings / ❌ {N} findings}
{findings or "No drift detected"}

### Layer 2: product-spec/ ↔ spec.md — {status}
{findings or "No drift detected"}

... (Layers 3–7 as above)

### Layer 8: FE ↔ BE Contract Drift — {✅ CLEAN / ⚠️ {N} findings / ❌ {N} findings}
{findings or "No drift detected"}

### Layer 9: Doc ↔ Code Reconciliation — {✅ CLEAN / ⚠️ {N} findings / ❌ {N} findings}
{findings or "No drift detected"}

### Layer 10: Constitution ↔ Code — {✅ CLEAN / ⚠️ {N} findings / ❌ {N} findings / N/A — no constitution}
{findings or "No drift detected"}

## All Drift Items

{DRIFT-001 through DRIFT-NNN with approval checkboxes}

## Proposed Actions

1. {Action for DRIFT-001}
2. {Action for DRIFT-002}
...

## Sync History

| Run | Date | Layers | CRITICAL | WARNING | Verdict |
|-----|------|--------|----------|---------|---------|
| #{N} | {date} | {N}/{applicable} | {N} | {N} | {verdict} |

Also write {FEATURE_DIR}/sync-report.json with machine-readable format.

Step 5: Present to User (Human-in-the-Loop)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  🔄 Sync & Verify: {Feature Name}
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  Layers checked: {N}/{applicable}
  Verdict: {CONSISTENT / DRIFT DETECTED / CRITICAL DRIFT}

  ❌ CRITICAL:  {N}
  ⚠️ WARNING:   {N}
  ℹ️ INFO:      {N}
  ✅ CLEAN:     {N} layers

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

If CRITICAL or WARNING items exist, present each one individually:

DRIFT-001 [CRITICAL] Layer 2: Must Have story US-003 missing from spec.md

  Source: product-spec/product-spec.md
    "US-003: As a user, I want to set notification quiet hours"

  Target: spec.md
    NOT FOUND — no US-003 or equivalent acceptance criteria

  Proposed fix: Add US-003 to spec.md with acceptance criteria from product-spec

  → [Approve] [Reject] [Defer to later] [Edit resolution]

Process each decision:

  • Approve: Queue the resolution for application
  • Reject: Mark as acknowledged, no action
  • Defer: Mark as deferred, will surface again on next sync-verify run
  • Edit: User provides custom resolution text

Step 6: Apply Approved Resolutions (if --fix or user confirms)

For each approved resolution:

  1. Read the target artifact
  2. Apply the proposed change (add missing section, update text, fix link)
  3. Log the change: Applied DRIFT-{NNN}: {description} in {file}
  4. After all changes: run a quick re-check on affected layers to verify fix

Safety: Before modifying any file, show the exact edit and confirm. Never silently modify.

Step 7: Update Status

Update .forge-status.yml:

sync_runs:
  last_run: "{ISO timestamp}"
  total_runs: {N+1}
  last_drift_count: {N}
  last_critical_count: {N}
  last_verdict: "{CONSISTENT / DRIFT DETECTED / CRITICAL DRIFT}"

Quick Mode (--quick)

When called with --quick by the forge orchestrator between phase transitions, only run the layer(s) directly relevant to the transition:

Transition Layers to Check
After Phase 1 → Phase 2 (none — research is input, not a sync target)
After Phase 2 → Phase 3 Layer 1 (research ↔ product-spec)
After Phase 3 → Phase 4 Layer 1
After Phase 4 → Phase 5 Layer 2 (product-spec ↔ spec.md)
After Phase 5 → Phase 5B Layer 3 (spec.md ↔ plan.md)
After Phase 5B → Phase 5C Layer 4 (plan.md ↔ tasks.md)
After Phase 5C → Phase 6 Layers 3, 4
After Phase 6 → Phase 6B Layers 5, 6, 8, 9, 10 (tasks ↔ code, spec ↔ code, contract drift, doc ↔ code, constitution ↔ code)
After Phase 6B → Phase 7 Full (all 10 layers)
After Phase 7 → Phase 8A Layer 7 (cross-links only)

In quick mode:

  • Only report CRITICAL items (suppress WARNING/INFO)
  • If zero CRITICAL: auto-proceed (no gate)
  • If CRITICAL found: pause and present to user before allowing phase transition

Operating Principles

  1. Read-only by default. Never modify files without explicit user approval per item.
  2. Evidence-based. Every finding includes specific text from both source and target artifacts.
  3. Proportional severity. Missing Must Have story = CRITICAL. Minor wording difference = INFO.
  4. Idempotent. Running twice produces the same report (unless artifacts changed).
  5. Layer-aware. Only check layers where both sides exist. Don't report "drift" for artifacts that haven't been created yet.
  6. Direction-aware. Distinguish forward drift (spec not in code) from backward drift (code decisions not in spec). Both matter but have different fix strategies.
  7. Cumulative history. Each run appends to sync history in .forge-status.yml. Trends matter.
  8. Category discipline. Structural drift is never auto-resolved. Cosmetic drift is auto-resolved only when the user explicitly opts in via sync_verify.auto_resolve.cosmetic: true. Auto-fix is an advisor turned into an editor — require deliberation, never default it on.
  9. Budget as advice, not automation. The drift budget exists to surface neglect, not to approve sloppiness. Exceeding the budget produces a warning; it does not silently accept drift.