Testing

Related docs:

Authoritative Commands

npm run compile: build the extension from src/ to out/.
npm run lint: type-check src/ and test/ without emitting files.
npm run check:docs: run deterministic documentation structure, link, ownership, and lightweight code-doc alignment checks.
npm run check:ledger: verify that the task and claims ledger files in .ralph/ are internally consistent; a drifted ledger blocks the full validation gate before tests run.
npm run prompt:calibrate -- <workspace-path>: render one prompt of each kind and target against a representative Ralph workspace and report estimated token counts versus the checked-in codex targets. This is a manual operator aid, not a validation gate.
npm test: run npm run compile:tests and then execute the Node test suite from out-test/test/.
npm run test:activation: compile the extension and test harness, launch a real Extension Development Host smoke test through @vscode/test-electron against a seeded temporary .ralph/ workspace, then verify activation, command registration, and Dashboard/PRD Wizard React webview readiness.
RALPH_E2E=1 npm run test:e2e-pipeline: create a fresh temp workspace, execute ralphCodex.runPipeline through the VS Code test harness with a deterministic fake Codex executable, and run the shipped scaffold, multi-agent loop, review-agent, and SCM-agent command chain end-to-end. The smoke asserts the delegated phase-command sequence, final pipeline artifact PR URL, and inspectable latest loop evidence surfaces (latest-prompt.md, latest-cli-invocation.json, latest-result.json, latest-provenance-bundle.json) plus run-bundle artifacts. Ralph's verifier treats the leading RALPH_E2E=1 assignment as a portable env override instead of shell-specific syntax, but when running the command manually in a terminal you still need to use a shell form your OS understands. Without RALPH_E2E=1, the command exits quickly with a skip message.
RALPH_E2E_ORCHESTRATION=1 npm run test:e2e-orchestration: create a fresh temp workspace and exercise the orchestration control plane end-to-end without touching the VS Code host. The smoke walks an OrchestrationGraph through a fan-out/fan-in traversal (writing state.json and per-node span artifacts), validates plan-graph fan-in blocking against incomplete children and then passing once they complete, drives a handoff through propose→accept and a second handoff through propose→reject, asserts that the reviewer role policy disallows the in_progress→done mutation (the same check that produces a policy_violation stop reason in reconciliation.ts), and triggers the scope_expansion human choke point via executeReplanNode so the gate artifact is written. Without RALPH_E2E_ORCHESTRATION=1, the command exits quickly with a skip message. Set RALPH_E2E_ORCHESTRATION_KEEP_WORKSPACE=1 to preserve the temp workspace for inspection.
npm run test:real-cli-smoke: run one temp-workspace Ralph iteration through the real codex exec path and print the preserved artifact paths. This command is optional and requires a working Codex CLI environment.
npm run test:offline-evals (or npm run test:evals): run the offline evaluation harness fixtures from test/evals/fixtures/ and emit a compact Markdown plus JSON summary; it fails only when fixture expectations drift, not when a fixture intentionally models a failing verifier outcome.
npm run test:ui-harness: run deterministic UI fixture rendering + accessibility-oriented checks for dashboard/sidebar webviews (uiFixtureHarness, panel/sidebar HTML regression checks, and shared theme contrast checks).
npm run build:webview: bundle the local React + TypeScript webview shell to out/webview-ui/main.js and out/webview-ui/main.css.
npm run watch: run the TypeScript compiler watcher and the webview bundle watcher together so Extension Development Host debugging does not serve missing or stale dashboard/sidebar assets.
npm run evidence:ui-fixtures: compile and export deterministic sidebar/dashboard fixture HTML plus a hash manifest to .ralph/artifacts/ui-fixtures/ for UX review evidence.
npm run dogfood: run one repeatable local dogfood pass through the real CLI smoke harness and persist a structured report under .ralph/artifacts/dogfood/ (provider, model, task count, stop reason, duration, failure category) without recording secrets.
npm run validate: run compile, check:docs, check:ledger, check:prompt-budget, lint, and test.
npm run package: verify the Node runtime and then build a .vsix package with vsce.
npm run publish:dry-run: currently an alias to npm run package (runtime check + vsce package). It validates packaging readiness without publishing.

What Is Covered

test/commandShell.smoke.test.ts: lightweight extension-shell smoke coverage for key command registration plus Show Status, latest-summary/latest-provenance/latest-prompt-evidence/latest-CLI-transcript and approved-decomposition-apply commands, stale latest-summary/latest-provenance repair after manual deletion, remediation-summary repair from latest-result.json, transcript-to-last-message fallback, latest-bundle reveal behavior, fresh-workspace doctrine scaffolding, established-workspace doctrine-pack initialization/repair without overwriting existing doctrine Markdown files, scoped runtime-artifact cleanup that preserves stable latest evidence surfaces, prompt clipboard auto-copy, non-claiming prepare/handoff behavior, and Open Codex IDE handoff behavior across clipboard-only, IDE-command, missing-command fallback, and cliExec warning modes with mocked Ralph state/artifacts.
scripts/run-e2e-pipeline-smoke.js: opt-in temp-workspace pipeline smoke coverage that runs the shipped codex exec provider path against a deterministic fake executable, exercises the full scaffold, loop, review, and SCM chain without network access, verifies inspectable loop evidence pointers and run-bundle artifacts, and emits the delegated phase-command sequence plus the resulting pipeline artifact PR URL.
scripts/run-e2e-orchestration-smoke.js: opt-in temp-workspace orchestration smoke that threads the orchestration supervisor, plan-graph fan-in gate, handoff manager lifecycle, role-policy check, and the scope_expansion human choke point through a single deterministic end-to-end run. The accompanying test/e2eOrchestrationSmokeScript.test.ts harness drives both the skip path and the full opt-in path and asserts the emitted JSON summary so orchestration regressions surface inside the default npm test gate.
test/pipelineDriver.test.ts: default-gate contract coverage for the Run Full Workflow phase sequencer (src/ralph/pipelineDriver.ts) using fake loop/review/SCM runners — no provider, verifier, Git, or Extension Host. It pins the loop→review→SCM→done role order, the stop reasons (a loop failure halts before review/SCM and finalizes failed; a review failure skips SCM but stays complete), the loop/review/scm resume entry points, and the finalized PR-artifact shape (PR URL and review-transcript path propagate only when reported), so the full-workflow contract regresses inside npm run validate without the opt-in RALPH_E2E smokes.
test/statusReport.test.ts: focused rendering coverage for status output, including task/backlog separation, provenance trust level messaging, inspection-root overrides, repeated-stop remediation reporting for decomposition, blocked, and human-review cases, recent iteration/run history, live retention summaries for generated artifacts and provenance bundles, and latest-artifact repair or stale-surface reporting.
test/artifactStore.test.ts: targeted retention cleanup coverage proving protected bundles survive cleanup when a latest pointer still references them, proving generated prompt/run/iteration artifacts stay protected when .ralph/state.json last*, runHistory[], or iterationHistory[] entries or latest result, preflight-report, prompt-evidence, execution-plan, or CLI-invocation records still reference them, including root-by-root regressions for latest-linked prompt, summary, and preflight references plus state-referenced transcript, last-message, and iteration-directory protection, including summary-only and path-only iteration references plus transcript-only and last-message-only run references, proving latest summary, preflight-summary, and provenance-summary Markdown surfaces can still protect the implied iteration directory when the matching JSON pointer is absent, including both iteration markers and rendered artifact-path lines, proving mixed latest-pointer and raw-state protection can preserve older summary/preflight iteration directories separately from older prompt/transcript/last-message artifacts in the same cleanup pass, proving latest provenance bundle/failure pointers keep only the referenced iteration directory, proving handoff-note cleanup keeps only the newest retained .ralph/handoff/*.json entries, including run-only state fallbacks that omit explicit iteration records and raw state run references that carry only explicit file paths, and proving generated-artifact conflicts resolve deterministically by keeping the newest parsed-iteration window first, then adding older protected entries without reordering it, while also reporting which retained entries survived only because of that protection.
test/vscode/runActivationSmoke.ts and test/vscode/suite/index.ts: optional real activation smoke coverage for extension activation, command registration, seeded .ralph/ workspace loading, and theme-sensitive Dashboard/PRD Wizard coverage in a real Extension Development Host across Default Light Modern, Default Dark Modern, and High Contrast. The React shell emits a typed webview-readiness message on mount; the suite waits on hidden activation-smoke diagnostics so bundle/load failures produce actionable timeout output instead of silently passing after command invocation.
scripts/run-real-cli-smoke.js: optional real codex exec smoke coverage that seeds a temp Ralph workspace, runs one CLI iteration against the actual Codex binary, and prints the resulting artifact pointers or preserved temp path for inspection.
test/evals/offlineEvalHarness.ts, test/evals/fixtures/*.fixture.json, and scripts/run-offline-evals.js: deterministic offline evaluation-harness baseline covering a simple success fixture, a verifier-block fixture with retry/fallback evidence, and a doctrine-context inclusion fixture while recording per-dimension pass/fail results.
test/promptBuilder.test.ts: deterministic prompt-kind selection, file-based template rendering, verifier-informed prompt refinement, prior-context trimming, prompt-budget omission behavior, and prompt-output determinism.
test/promptBuilder.snapshot.test.ts: golden snapshot coverage for each valid prompt-kind fixture scenario using readable .md snapshots under test/fixtures/snapshots/.
test/promptBudget.golden.test.ts: focused golden fixture coverage for prompt-budget policy wording, key prompt kinds and role-specific section exclusions, deterministic prompt-budget evidence fields, and stable omitted-section metadata when trimming occurs.
test/cliExecStrategy.test.ts: CLI argument shaping, medium-versus-high reasoning-effort selection, transcript generation, missing-CLI error wording, and fail-fast prompt-hash mismatch detection before launch.
test/iterationEngine.integration.test.ts: temp-workspace loop execution with mocked Codex exec covering progress, repeated no-progress, repeated failure classification, persisted remediation artifacts for decomposition, reframe, and human-review paths, proposal-child generation versus non-decomposition artifacts, verifier-driven completion, human-review-needed stops, execution-plan/CLI-invocation artifact emission, persisted state across iterations, durable session handoff-note writes on clean terminal stops plus next-session prompt carry-forward, non-blocking IDE preparation before later CLI selection, and blocked integrity evidence for executionPlanHash, promptArtifactHash, and stdinPayloadHash.
test/doctrine.test.ts: deterministic doctrine-pack coverage for scaffold creation, partial-pack completion, non-overwrite behavior, explicit invalid-evidence-index.json repair, protected-file identification, required-heading validation, and minimal evidence-index.json validation.
test/doctrineProposals.test.ts: doctrine proposal parsing, target validation, protected-target classification, aggregate risk calculation, and artifact-shaping coverage for proposal-only doctrine updates.
test/preflight.test.ts: targeted preflight coverage for likely task-schema drift diagnostics, doctrine-health warnings, validation-command readiness wording, and stale latest-artifact plus retention-readiness warnings before CLI loops start.
test/stateManager.test.ts: non-destructive workspace inspection, workspace seeding, task-file parse diagnostics, iteration-result serialization, and scoped runtime-artifact cleanup that preserves durable state, stable latest evidence surfaces, and the latest protected provenance bundle while pruning older generated artifacts.
test/taskContract.test.ts: executable contract tests for the normalized-task shape defined in docs/invariants.md § Normalized Task Contract, covering required-field enforcement, optional-string and optional-array coercion to undefined, dependency deduplication via Set, enum-field rejection for invalid priority/mode/tier values, unknown-field drop after normalization, child-task conversion field mapping (status forcing, rationale → notes, null validation → undefined, parent mode inheritance), and parent status promotion on child application.
test/taskSeeder.test.ts, test/taskCreation.test.ts, and the seeding scenarios in test/commandShell.smoke.test.ts: focused coverage for epic/feature task seeding, including durable artifacts under .ralph/artifacts/task-seeding/, shared appendNormalizedTasksToFile persistence, duplicate-id remapping or rejection behavior, and failure paths that leave tasks.json unchanged.
test/taskFile.test.ts: task-file parsing, deterministic task selection, explicit parent/dependency behavior, approved decomposition-apply validation, legacy normalization, status counting, preflight graph diagnostics, and file-backed claim acquisition and release semantics including contested claims, legacy IDE-claim reclamation across one or many active handoff records, lock-mediated concurrent acquisition and release, idempotent release, and stale-claim detection.
test/verifier.test.ts: cheap validation-command readiness probing for explicit executables, unresolved PATH commands, and portable execution of leading KEY=value env-prefix validation commands.
test/docsValidator.test.ts: deterministic coverage for required doc presence, heading rules, AGENTS.md ownership guardrails, local doc links, verifier-doc alignment checks, and the operator-facing task-seeding command or artifact contract across README, workflows, invariants, and testing docs.
test/ui/ and test/webview/: shipped dashboard/sidebar rendering, host, and bridge regression coverage for the production src/webview-ui/, src/ui/, and src/webview/ ownership boundary. The historical UXrefresh/ prototype bundle is reference-only and is not part of the validation gate.
test/ui/reactUiOwnership.test.ts: React-first UI ownership guard that asserts the authoritative src/webview-ui/ React shell is present and fails on drift — if a reference/archive UI tree (e.g. UXrefresh/) is reintroduced as a directory under the repo root or src/, or imported by any production module under src/. Keeps the single-renderer boundary enforced in the default validation gate, not just documented.
src/ui/fixtureEvidence.ts, scripts/export-ui-fixture-catalog.js, and test/ui/uiFixtureEvidenceCatalog.test.ts: deterministic fixture-catalog evidence export coverage for sidebar/dashboard HTML, including stable sort order, stable hashing, and baseline accessibility/webview hooks.
test/loopLogic.test.ts: outcome classification, no-progress detection, failure signatures, and stop-decision logic.
test/workspaceInspection.test.ts: package-manager, lifecycle, and validation-command inference.
test/workspaceScanner.test.ts: end-to-end workspace inspection heuristics from real temp dirs.
test/workspaceSupport.test.ts: explicit-path and PATH-lookup Codex CLI support inspection.

Stub Smoke Vs Real Activation Smoke

The default npm test path keeps using the lightweight stubbed harness because it is fast, deterministic, and good enough for most command-shell and artifact assertions.
npm run test:activation is intentionally narrower than the default Node test gate but more realistic: it verifies the packaged extension can activate under VS Code, confirms key Ralph commands are registered, opens a seeded temporary .ralph/ workspace, switches the host through Default Light Modern, Default Dark Modern, and High Contrast, opens the Dashboard settings tab plus the PRD Wizard in each theme, and waits for each React webview surface to report that it mounted.
If the default @vscode/test-electron download does not launch cleanly in your environment, rerun npm run test:activation with RALPH_VSCODE_EXECUTABLE_PATH=/absolute/path/to/code so the smoke can target a known-good local VS Code executable.
Neither path introduces heavy UI automation, browser automation, or a richer VS Code integration framework. The only webview bridge path exercised by the activation smoke is the deterministic readiness diagnostic; deeper command-bridge behavior remains covered by the default test/webview/ host tests.
npm run test:real-cli-smoke is intentionally separate from npm run validate because it depends on live Codex auth/network reachability and may preserve the temp workspace on failure for inspection.

What Is Not Covered

heavy Extension Development Host UI automation beyond lightweight command-shell and webview-readiness smoke coverage
live clipboard integration in a real host OS session
live VS Code command handoff behavior in a real Extension Development Host session
Real codex exec process execution is only covered by the optional npm run test:real-cli-smoke path, not by the default npm test or npm run validate gate.
Prompt-budget calibration is a manual operator workflow through npm run prompt:calibrate -- <workspace-path> and is not exercised by npm test or npm run validate.
Live Git checkpoint behavior in a real repository
live .vsix install behavior after packaging

When changing those areas, rely on the authoritative commands above plus manual verification in the Extension Development Host.

Test Runtime Notes

npm test preloads test/register-vscode-stub.cjs so extension modules can run under plain Node without a heavyweight VS Code test harness.
The smoke tests intentionally stay thin: they verify command registration and simple command behavior, not full UI rendering or live VS Code integration.
The integration suite uses temp directories and mocked codex exec behavior instead of spawning the real Codex CLI.
The optional real CLI smoke persists the temp workspace when execution or verification fails so operators can inspect .ralph/artifacts/, stderr.log, and the latest summary surfaces directly.
npm run test:real-cli-smoke accepts RALPH_REAL_CLI_SMOKE_COMMAND, RALPH_REAL_CLI_SMOKE_MODEL, and RALPH_REAL_CLI_SMOKE_KEEP_WORKSPACE=1 when you need a non-default Codex binary, model, or preserved temp workspace.
Prompt-template tests may point ralphCodex.promptTemplateDirectory at temp directories so rendering stays thin and deterministic without pulling in a heavier templating engine.
Prompt snapshot updates require npm test -- --updateSnapshot; reviewers should read .md snapshot diffs with the same care as code diffs because they capture the durable prompt contract.
The activation smoke still stays thin: it exercises only the owned theme-sensitive surfaces and does not attempt deep UI automation inside native VS Code dropdown popups.

Manual Theme Verification

The select legibility guardrail is split between deterministic coverage and one manual host check:

npm run validate proves the React webview shell bundles locally, the shared webview CSS and generated Dashboard/PRD wizard HTML keep select open-state colors tied to VS Code dropdown and active-selection theme tokens, and the dashboard/sidebar host message paths still dispatch through the typed bridge.
npm run test:activation proves those owned surfaces open cleanly in a real Extension Development Host under Default Light Modern, Default Dark Modern, and High Contrast.
One manual check is still required because native VS Code select popups are rendered by the host rather than the HTML test harness: open the Dashboard settings tab and the PRD wizard in each theme, expand every select/optionset, and confirm the option rows and selected row stay readable with no near-invisible foreground/background pair.

Recommended operator flow:

Run npm run validate.
Run npm run test:activation or RALPH_VSCODE_EXECUTABLE_PATH=/absolute/path/to/code npm run test:activation when the bundled VS Code download is unreliable on the current machine.
In the Extension Development Host, switch through Default Light Modern, Default Dark Modern, and High Contrast, open the Dashboard settings tab plus the PRD wizard, and visually inspect each expanded select/optionset popup.
For sidebar/dashboard UX changes, run npm run evidence:ui-fixtures and review .ralph/artifacts/ui-fixtures/manifest.json plus selected panel-*.html/sidebar-*.html outputs alongside test results.

Packaging Runtime

Packaging is supported on Node 22+.
scripts/ensure-node-version.js fails fast when npm run package is invoked on an older runtime.
The same runtime gate also protects npm run publish:dry-run because that script currently delegates to npm run package.
Node 18 is intentionally treated as unsupported for packaging because the modern @vscode/vsce toolchain requires a newer runtime.
The packaged .vsix must include runtime dependencies and exclude development-only inputs such as src/, test/, .ralph/, .worktrees/, local .tgz files, and prior .vsix artifacts.
npm run package proves the repo can emit a .vsix, but manual .vsix install still needs an operator check through Extensions: Install from VSIX... or code --install-extension.

FilesExpand file tree

testing.md

Latest commit

History