feat(test): PTY-based interactive CLI snapshot tests#2052
Conversation
✅ Deploy Preview for viteplus-preview canceled.
|
How to use the Graphite Merge QueueAdd the label auto-merge to this PR to add it to the merge queue. You must have a Graphite account in order to use the merge queue. Sign up using this link. An organization admin has enabled the Graphite Merge Queue in this repository. Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue. This stack of pull requests is managed by Graphite. Learn more about stacking. |
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
Implements rfcs/interactive-snapshot-tests.md: - crates/vite_cli_snapshots: libtest-mimic runner that executes every step in a real PTY (vt100 grid capture), synchronizes interactive input on OSC 8 milestones, and compares Markdown snapshots with real pass/fail semantics (UPDATE_SNAPSHOTS=1 to accept) - one fixture tree with per-case vp flavor (local, global, or both for parity), per-case VP_HOME/HOME isolation, and managed-runtime seeding - vpt test multitool (vtt-aligned subcommands plus json-edit, chmod, probe) - tool migrate-snap-tests: one-click conversion of old steps.json cases, validated by migrating the check-pass cases end to end - packages/prompts: milestone emission (VP_EMIT_MILESTONES=1) wired into select, confirm, and text renders - just snapshot-test recipe, pnpm snapshot-test wrapper, and a CI step on the Rust test job (global flavor only until the JS build joins during migration) Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
Full both-flavor snapshot coverage now runs in cli-snap-test, which builds packages/cli/dist for the local flavor and reuses the installed release binary for the global flavor via the new VP_SNAP_GLOBAL_VP override (no second vite_global_cli compile). The Rust test job keeps its fast global-only leg for early signal, with a comment explaining the split. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
The archive's package loop is kept in sync with the justfile test recipe, which already excluded vite_cli_snapshots; the archive copy was missed, so the relocated cli_snapshots binary panicked at nextest list time on the Windows runner (its compile-time CARGO_MANIFEST_DIR is a Linux path). Also prefer the runtime CARGO_MANIFEST_DIR over the compile-time value in the harness, which is what relocated nextest archives rewrite; this is the groundwork for the planned Windows legs. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
The e2e job's repo-wide vp check flagged the markdown table alignment. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
The fixtures under crates/vite_cli_snapshots/tests are workspaces under test, not repo code, same as the existing snap-tests excludes. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
Adds the harness reference README (case/step/interaction schema, vpt helpers, milestone conventions, env overrides, migration workflow) and points AGENTS.md and CONTRIBUTING.md at it, marking the legacy snap trees as migration-only so new cases land in the new harness. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
build-windows-tests now cross-compiles a dedicated -p vite_cli_snapshots archive (test binary + vpt), and the new cli-snapshot-test-windows job runs it on windows-latest with no Rust toolchain: prebuilt vp via VP_SNAP_GLOBAL_VP, JS CLI built on the runner for the local flavor, managed runtime prewarmed for seed-runtime. vpt resolution now prefers the runtime CARGO_BIN_EXE_vpt that nextest rewrites under --workspace-remap, matching the CARGO_MANIFEST_DIR handling. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
Reuse: milestone hex encoding via Buffer.toString('hex'); the migrator now
imports the legacy Steps schema from snap-test.ts instead of redeclaring it.
Simplification: shared makeTodo/redirect handling and a verbatim-vpt set in
the migrator; dead NewStep fields dropped; shared find_beside_test_exe and
manifest_dir helpers in the harness; the always-true separator guard from
the upstream port removed.
Efficiency: redaction regexes compiled once per run (LazyLock), diagnostic
sort skipped when no blocks exist, fixture staging filters harness metadata
instead of copy-then-delete, per-step env only cloned when a step overrides
it, and the prompts milestone flag is cached at module load (per-keystroke
path).
Suite output is unchanged: all 8 trials pass against existing snapshots and
the migrator reproduces byte-identical fixtures.
Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
fefbe9d to
e6ed290
Compare
std's canonicalize returns a \\?\ verbatim path on Windows; CMD.EXE,
which runs the local flavor's .cmd shims, rejects verbatim/UNC working
directories ('UNC paths are not supported'), so every local-flavor case
failed instantly on the Windows snapshot job. Also run the Windows suite
with --no-fail-fast: on a snapshot suite every diff is diagnostic signal.
Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
The full both-flavor suite already runs in cli-snap-test (linux/mac) and cli-snapshot-test-windows; the extra leg only re-ran the global cases and cost a vite_global_cli build inside the Rust test job. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
On Windows ~/.vite-plus/bin/vp.exe is the trampoline, which re-execs %VP_HOME%/current/bin/vp.exe; the harness gives each case an isolated VP_HOME with no install inside, so vp_help::help::global failed with 'failed to execute ...current/bin/vp.exe'. Use current/bin/vp.exe (the real CLI) as VP_SNAP_GLOBAL_VP, matching what bin/vp is on Unix. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
Converted with tool migrate-snap-tests (15 help/version steps, zero hand conversions). The new snapshot keeps every legacy assertion and adds the banner line the old pipe capture missed; vp -V now records the isolated workspace state (tools Not found) since the new harness does not symlink the checkout node_modules into fixtures. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
…llisions Successfully converted case directories are now removed from the legacy tree automatically (git history keeps the originals; --keep-old defers). A case whose target fixture already exists is skipped and reported instead of clobbering it: the same name in both legacy trees means a hand merge. Also migrates packages/cli/snap-tests/cli-helper-message as the first such merge: the fixture gains a cli_helper_message_local case (vp -h / -V) next to the 15-step global one, sharing the legacy package.json; the global snapshot is unchanged by the added file. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
Extracted from the cli-snap-test matrix per review: a dedicated Linux/macOS job with no runner.os/shard filter conditions, mirroring the Windows job's structure (build-upstream for dist + release vp, bootstrap-cli:ci, runtime prewarm for seed-runtime, then cargo test). Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
build-upstream and the snapshot suite never touch docs/ (the Windows snapshot job already runs green without it); the step exists in cli-e2e-test for pnpm tsgo, which this job does not run. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
Converted with tool migrate-snap-tests (env-prefixed commands became step envs, win32 skip became skip-platforms; zero hand conversions; the old case dir was removed by the migrator). The task-cache flow asserts identically: cold miss, cache hit with replay trailer, and env-changed miss. Build sizes and the asset hash are now recorded concretely instead of masked; they are deterministic per vite version. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
The win32 skip was inherited from the legacy case (added in #544 with no stated reason). Its plausible causes are gone in the new harness: env prefixes are structured step envs instead of shell syntax, the task engine supports Windows, .gitattributes forces LF so the asset content hash is checkout-stable, and vite prints forward-slash paths on every OS. The Windows snapshot job is the arbiter; if it disagrees, the skip returns with a documented reason. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
- Backslash-to-slash normalization now runs unconditionally on Windows instead of only when an absolute-path redaction matched the screen; tools print OS-native separators for relative paths too. Debug-escaped separators collapse BEFORE conversion so https:// URLs survive. - Every rendered row is trimmed of trailing whitespace on all platforms: ConPTY repaints rows padded to the grid width when a second console client attaches (global-flavor vp spawning node). - Byte sizes and content-hash asset suffixes are now redacted (<size>, <hash>): emitted bundle bytes differ across OSes, as #2031's 0.10 vs 0.11 kB pack output showed. Aligns the implementation with the RFC's normalization list; build_vite_env re-recorded accordingly. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
A redaction fixture drives the guarantees end to end through the PTY (a new vpt print-native-path payload prints OS-native separators, so one snapshot proves normalization on every platform; sizes/hashes masked; lowercase stems and URLs survive). A redact_unit test target covers the edges fixtures cannot exercise deterministically: ConPTY row padding, Debug-escaped separator collapse, and URL survival, with the Windows-gated assertions running in the nextest-archive job. Runner entry points now run the whole package (both test targets). Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
<size> kB instead of <size>, matching the legacy <variable> kB convention: the unit only changes when content crosses a magnitude boundary, which is real signal. Durations stay fully masked because their unit flips with timing (999ms vs 1.00s). Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f046797698
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The run tempdir is dropped before Conclusion::exit so runs no longer leak staged trees. vpt cp copies into existing directories like real cp. Real-tool resolution happens after per-step envs apply, so a step PATH override selects the tool the child sees. after-cleanup steps get their per-step envs (and PATH-aware resolution) too. detect-changes ORs in a crates/vite_cli_snapshots filter so baseline-only .md changes still run the snapshot jobs. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bfad895b80
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The push trigger's paths-ignore becomes a paths list with negations so baseline-only .md pushes still run CI. list-dir hides dot entries like plain ls (--all opts in) and the migrator TODOs ls flags. vp/vpr/vpx resolve through the case PATH first so VP_HOME/bin shims shadow the harness aliases. A failing step now stops the case by default (shell-like && semantics) with continue-on-failure as the opt-out; the migrator restores legacy semantics exactly by marking each command line's final step. Bare runtime-tool versions (Node 24.x, pnpm 10.x from vp create) redact by tool-name context while user semver stays assertable. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7660d0533f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…build Round-8 review follow-up: absent dist stays a deliberate fail-fast with remedies in the message (auto-building would drag the rolldown chain into the recipe; auto-skipping would hide coverage), and toolchain-only contributors get a first-class entry point instead. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: df5abf79ad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: df5abf79ad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Failure flow gains line boundaries: a failing step skips the rest of its line (up to the next continue-on-failure step) and the following line resumes, reproducing legacy line-level continuation exactly; proven by a new failure_semantics fixture. The dist-staleness check also covers packages/core, which is linked into every staged workspace. The migrator emits seed-runtime = false for runtime-provisioning cases (vp env install/uninstall) so their empty-home coverage survives migration. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b750991666
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Wording preference: say runner (the executing test binary) or suite (the collection) instead. Identifiers are unchanged; the cargo manifest key harness = false stays, it is syntax, not prose. The config_import snapshot is re-recorded because its case comment is rendered into it. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
vpt cp -r nests into an existing directory and touch-file creates every operand, both matching coreutils and locked in by vpt_selftest steps. The run root aliases vite to the core package (as the vite -> core override does in migrated projects), proven by a new config_import case. The local-flavor freshness check covers packages/prompts, which is bundled into the CLI dist. Piped steps spawn as process-group leaders and a timeout kills the whole tree, so descendants holding the pipes can no longer hang the reader threads. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ab99aad510
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
stat-file gains --assert/--assert-not, failing on state mismatch, and the migrator emits them for test expressions, so guards like test -f x && cmd keep the shell's short-circuit through the line-boundary flow (proven by a new failure_semantics case). after-cleanup steps honor the step timeout with tree-kill instead of blocking on Command::output. chmod validates the mode and target on Windows too, instead of a blind no-op. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 222abdcc73
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
grep-file and print-file keep their coreutils exit semantics: nonzero on a missing pattern, unreadable file, or missing operand, so content guards and cat assertions short-circuit like the shell (locked in by new vpt_selftest steps). The README documents oxfmt/oxlint as local-flavor only; they are JS shims from the CLI build, and global cases use vp fmt / vp lint like real global-binary users. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c060a5d7e1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Real-tool resolution runs from the step cwd, so relative PATH entries resolve exactly as the child sees them (after-steps included). The migrator omits only the ROOT steps.json/snap.txt; a project file with the same name in a subdirectory carries over. Windows global provisioning copies vp-shim.exe beside vp.exe when the source build has one, so trampoline-creating cases (vp env setup) work. Claude-Session: https://claude.ai/code/session_01NRgjMi2Vus3iJctudGEWPT
|
@codex review |

Implements the RFC included in this PR as
rfcs/interactive-snapshot-tests.md.crates/vite_cli_snapshots: PTY-based CLI snapshot suite (libtest-mimic). Every step runs in a real pseudo-terminal with vt100 grid capture; interactive steps script keystrokes synchronized on OSC 8 milestones; snapshots are Markdown with real pass/fail semantics (UPDATE_SNAPSHOTS=1to accept,.md.newplus unified diff on mismatch). Built on the pty_terminal/snapshot_test crates from vite-task at the already-pinned rev.vp = "local" | "global" | ["local", "global"]; the parity matrix already caught real drift between the two help outputs. Per-caseVP_HOME/HOME/npm-prefix isolation removesserialand the bootstrap byte-match requirement;seed-runtimesymlinks a provisioned managed runtime; failure flow is shell-like with line boundaries (continue-on-failure).vpttest multitool (vtt-aligned subcommands plusjson-edit,chmod,probe) so fixtures run without a shell and behave identically on every platform.tool migrate-snap-tests <dir> --vp <flavor> [filter]: one-click migration of oldsteps.jsoncases with a report; TODO-free conversions auto-remove the legacy dir. Migrated so far: the fourcheck-pass*cases,cli-helper-message(local + global merged into one fixture), andbuild-vite-env.packages/prompts: milestone emission forselect/confirm/text, gated onVP_EMIT_MILESTONES=1, byte-identical to the vite-task protocol; render output unchanged when disabled.CLI snapshot testjobs for Linux, macOS, and Windows (Windows runs a cross-compiled nextest archive, no Rust toolchain on the runner); snapshot-baseline-only changes still trigger CI despite the markdown filters.just snapshot-test [filter],just snapshot-test-global(no JS build needed),pnpm snapshot-test. Contributor docs:crates/vite_cli_snapshots/tests/cli_snapshots/README.md.Validation: 14 snapshot trials plus redaction/migrator/prompt unit suites pass in compare mode on all three CI platforms.
Follow-ups per the RFC phasing:
local-registrycase support, remaining prompt components (multiselect, password, spinner), migration batches, legacy runner removal.