Skip to content

11_testing_strategy.md

Latest commit

 

History

History
95 lines (73 loc) · 4.38 KB

11_testing_strategy.md

File metadata and controls

95 lines (73 loc) · 4.38 KB

Testing Architecture

Overview

Testing is based on Vitest with strict TypeScript type checking. Suites track the three-phase pipeline refactor and ensure parity with historical behaviour.

Test Layers

graph LR
    UNIT[Unit] --> PIPE[Core Pipeline]
    UNIT --> PLUGINS[Remark Plugins]
    UNIT --> UTILS[Utilities]

    INTEGRATION[Integration] --> PIPELINE3[3-Phase Pipeline]
    INTEGRATION --> CLI[CLI]
    INTEGRATION --> PLUGIN_ORDER[Plugin Order]

    E2E[End-to-End] --> VISUAL[Visual Contracts]
    E2E --> PLAYGROUND[Browser Smoke]
Loading
  • Unit - pure functions and modules (parsers, plugin helpers, generators)
  • Integration - end-to-end runs through key entry points
  • E2E - scenario tests exercising real documents and visual diff tooling

Key Suites

Path Purpose
tests/unit/core/pipeline/context-builder.test.ts Validates Phase 1 option merging, force-command resolution, validation helpers
tests/unit/core/pipeline/format-generator.test.ts Ensures Phase 3 orchestrates HTML/PDF/Markdown generation (skipped where I/O heavy)
tests/integration/pipeline-3-phase.integration.test.ts Confirms the new pipeline executes once per run and compares against legacy timings
tests/integration/plugin-order-validation.integration.test.ts Verifies metadata registry + validator catch ordering violations and suggest fixes
tests/unit/plugins/remark/* Cover plugin-specific logic (imports, headers, css classes, html comments, etc.)
tests/unit/cli/interactive/service.test.ts Tests interactive CLI mapping and archiving behaviour
tests/integration/legacy-remark-parity.integration.test.ts Guards compatibility with historical remark outputs
tests/e2e scripts Generate PDF/HTML artefacts for manual review (run on demand)
tests/playground/playground.spec.ts Playwright UI smoke tests for the web playground (6 scenarios, not in CI)

All suites run under Vitest; heavier PDF/visual checks are opt-in via env flags.

Tooling & Configuration

  • vitest.config.ts defines project groups (unit/integration, e2e, golden, and web) and introduces path aliases and coverage instrumentation via @vitest/coverage-v8
  • npm run test, npm run test:unit, npm run test:integration, npm run test:e2e provide targeted entry points
  • npm run test:playground:ui - Playwright smoke tests for the web playground (not part of CI; run manually after npm run build:web)
  • npm run typecheck (tsc --noEmit) runs alongside tests in CI to keep typing honest
  • Coverage thresholds are enforced by Vitest configuration (see package.json scripts for test:coverage)

CI Expectations

CI pipelines (GitHub Actions) execute:

  1. npm ci
  2. npm run lint
  3. npm run typecheck
  4. npm run test:ci (Vitest in non-watch mode with JSON reporter)
  5. npm run build

Failing suites block merge; release automation (semantic-release) runs the same sequence in prerelease checks.

Data & Fixtures

  • Fixtures live under tests/fixtures for cross-suite reuse
  • Integration suites create temporary directories for output inspection and clean them up after runs
  • Visual/E2E tests place generated artefacts under tmp/ or tests/output/ for manual inspection, ensuring they do not pollute source control

Testing discipline focuses on guarding pipeline invariants (single remark run, plugin order, deterministic output) while keeping regression coverage for legacy constraints.

Golden Test Harness

  • 38+ fixtures in tests/golden/fixtures/
  • Runner: tests/golden/runner.test.ts
  • Regenerate: UPDATE_GOLDEN=1 npx vitest run tests/golden/
  • Naming: NN-feature-scenario/