Refactor waspc e2e tests from shell-script DSL to native Haskell steps - #4294
Open
cprecioso wants to merge 21 commits into
Open
Refactor waspc e2e tests from shell-script DSL to native Haskell steps#4294cprecioso wants to merge 21 commits into
cprecioso wants to merge 21 commits into
Conversation
Previously, each e2e test case was built up as one big bash one-liner (ShellCommandBuilder produced shell command strings folded with ' && ') and executed via 'shell'. Snapshot tests additionally executed their commands at test-tree construction time, before tasty even started. Now each test case is a list of native Haskell steps: - Step (ReaderT context IO) replaces the pure Reader shell-string builder. - Commands are first-class values run via proc (no shell), with explicit cwd from the step context, env var modifiers, stdin support, output captured for assertions and teed to the per-test log, and process-group termination on interruption (now also for regular tests). - File operations and assertions use System.Directory & friends instead of printf/base64/awk/grep/rsync hacks, making steps platform-agnostic. - Snapshot tests are now single directory-level golden tests (tasty-golden Advanced API) that run their steps at test runtime, compare current vs golden snapshot dirs reporting all mismatches at once, and keep the accept-all and --accept workflows working. - Per-step log sections and step-aware failure messages make failures point at the exact failing step. Long-running process support (FIXME'd tests) is not implemented yet, but Command.startCommand/terminate is designed to support it. See #3404 https://claude.ai/code/session_01HLhBbhP5j1tGKcqPLXu8oF
GHC's file locking forbids reading a file the same process has open for writing, so close the log before reading its content into the failure message (a failed test won't log anything further anyway). https://claude.ai/code/session_01HLhBbhP5j1tGKcqPLXu8oF
cprecioso
had a problem deploying
to
railway-deploy-test
June 11, 2026 06:41 — with
GitHub Actions
Error
cprecioso
had a problem deploying
to
fly-deploy-test
June 11, 2026 06:41 — with
GitHub Actions
Error
@wasp.sh/spec
@wasp.sh/wasp-cli
@wasp.sh/wasp-cli-darwin-arm64-unknown
@wasp.sh/wasp-cli-darwin-x64-unknown
@wasp.sh/wasp-cli-linux-arm64-glibc
@wasp.sh/wasp-cli-linux-x64-glibc
@wasp.sh/wasp-cli-linux-x64-musl
commit: |
cprecioso
marked this pull request as draft
June 11, 2026 07:01
cprecioso
temporarily deployed
to
fly-deploy-test
June 11, 2026 07:07 — with
GitHub Actions
Inactive
cprecioso
temporarily deployed
to
railway-deploy-test
June 11, 2026 07:07 — with
GitHub Actions
Inactive
Commit 83734c4 (#4276) added AGENTS.md and CLAUDE.md to examples/kitchen-sink but the golden snapshot was not regenerated, so the kitchen-sink snapshot test has been failing since. Regenerated with --accept. https://claude.ai/code/session_01HLhBbhP5j1tGKcqPLXu8oF
cprecioso
force-pushed
the
claude/session-8f8605
branch
from
June 11, 2026 08:39
3a4ecbc to
35a63c0
Compare
cprecioso
temporarily deployed
to
fly-deploy-test
June 11, 2026 08:52 — with
GitHub Actions
Inactive
cprecioso
temporarily deployed
to
railway-deploy-test
June 11, 2026 08:52 — with
GitHub Actions
Inactive
cprecioso
temporarily deployed
to
railway-deploy-test
June 11, 2026 11:05 — with
GitHub Actions
Inactive
cprecioso
temporarily deployed
to
fly-deploy-test
June 11, 2026 11:05 — with
GitHub Actions
Inactive
cprecioso
had a problem deploying
to
fly-deploy-test
June 11, 2026 11:29 — with
GitHub Actions
Error
cprecioso
had a problem deploying
to
railway-deploy-test
June 11, 2026 11:29 — with
GitHub Actions
Failure
- Build output assertions in ViteBuildTest run a grep command and assert on its exit code again (matching the original tests' semantics), instead of checking file contents natively. The assertAnyFileInDirContainsText/assertNoFileInDirContainsText steps are gone. - Test case steps are a monadic Step value again, mirroring the original ShellCommandBuilder shape: TestCase takes 'sequence [step1, step2, ...]' and makeSnapshotTest sequences its steps internally. https://claude.ai/code/session_01HLhBbhP5j1tGKcqPLXu8oF
cprecioso
force-pushed
the
claude/session-8f8605
branch
from
June 11, 2026 11:34
45412a2 to
4994151
Compare
Deploying wasp-docs-on-main with
|
| Latest commit: |
4901612
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://cc886cb8.wasp-docs-on-main.pages.dev |
| Branch Preview URL: | https://claude-session-8f8605.wasp-docs-on-main.pages.dev |
Since Step is a monad, a test case reads most naturally as a do block:
TestCase "succeed-uncompiled-project" $ do
createTestWaspProject minimalStarterTemplate
inTestWaspProjectDir $ do
runCommand waspCliCompile
assertDirExists ".wasp"
TestCase, makeSnapshotTest and inTestWaspProjectDir/
inSnapshotWaspProjectDir now take a single Step instead of a list,
and helpers like mapM_/forM_ replace manual list building.
https://claude.ai/code/session_01HLhBbhP5j1tGKcqPLXu8oF
cprecioso
temporarily deployed
to
fly-deploy-test
June 11, 2026 12:10 — with
GitHub Actions
Inactive
cprecioso
temporarily deployed
to
railway-deploy-test
June 11, 2026 12:10 — with
GitHub Actions
Inactive
- Rename Step env fields to context/logger - Add HasWaspProjectContext to unify the create/enter WaspProject steps (createWaspProject/inWaspProjectDir) across the test and snapshot contexts - Use waspCliDbReset in the db-reset fail-outside-project case - Specialize the migration-dir assertion and drop the single-use assertDirHasSubdirWithNameContaining - Rename isMigrationDirNameWithDatePrefix to isUnnormalizedMigrationDirName Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Use Test.Tasty.Golden.Advanced.goldenTest the intended way: the golden and tested values are the snapshot contents (each content-checked file's bytes), extracted in the value-producing actions, with the comparator only comparing values. Previously the values were directory paths and the comparator did all the file reading and diffing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cprecioso
temporarily deployed
to
railway-deploy-test
June 30, 2026 15:16 — with
GitHub Actions
Inactive
cprecioso
temporarily deployed
to
fly-deploy-test
June 30, 2026 15:16 — with
GitHub Actions
Inactive
- Shell-quote env var values and args in showCommand - Extract assertCommandSucceeded/assertCommandFailed helpers - Simplify e2eTests to a pure TestTree - Use FilePath.</> for golden snapshot paths - Remove unused loggerLogFile export Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cprecioso
temporarily deployed
to
railway-deploy-test
June 30, 2026 16:41 — with
GitHub Actions
Inactive
cprecioso
temporarily deployed
to
fly-deploy-test
June 30, 2026 16:41 — with
GitHub Actions
Inactive
cprecioso
temporarily deployed
to
railway-deploy-test
July 2, 2026 09:54 — with
GitHub Actions
Inactive
Contributor
|
This pull request is now stale. Next steps if the PR is still relevant, and you are able to devote time to it:
If no action is taken, this PR will be automatically closed in 7 days. |
Contributor
|
This pull request is now closed. Next steps if the PR is still relevant:
|
Conflicts were in the two e2e test files this branch rewrote: - `e2e-tests/Tests/ViteBuildTest.hs`: main only reflowed two `TestCase` expressions in the old shell-command DSL, which this branch replaced with step-based test cases. Kept ours. - `e2e-tests/Main.hs`: kept our pure `e2eTests :: TestTree` (snapshot tests now run under tasty instead of being prepared eagerly in IO), and ported main's concurrency cap from #4449 onto tasty's `NumThreads` so we don't lose the fix for CPU oversubscription. `WASP_E2E_TEST_MAX_JOBS` still overrides it, now for all test cases rather than just snapshot preparation. Follow-ups needed after the textual merge: - Restored `async` in `waspc-e2e-tests` build-depends (main dropped it when it stopped using `mapConcurrently`, but our `Command.hs` uses it) and dropped `unliftio`, which main added for `pooledMapConcurrentlyN` and is now unused. - Reformatted `waspc.cabal` and `e2e-tests/SnapshotTest.hs` with the tool versions main pins (cabal-gild 1.8.4.1, ormolu 0.8.1.1).
cprecioso
temporarily deployed
to
railway-deploy-test
July 28, 2026 08:50 — with
GitHub Actions
Inactive
cprecioso
had a problem deploying
to
fly-deploy-test
July 28, 2026 08:50 — with
GitHub Actions
Failure
- Start command output drains before writing stdin in startCommand, so a large stdin write can't deadlock against a child blocked on full, undrained output pipes. - Extract commonIgnoredFilePathFilters shared by the snapshot manifest (existence) check and the content check, so the two lists can't drift. - Extract makeWaspProjectContext into Context, keeping the "wasp-app" project naming convention in one place.
cprecioso
temporarily deployed
to
fly-deploy-test
July 28, 2026 13:10 — with
GitHub Actions
Inactive
cprecioso
temporarily deployed
to
railway-deploy-test
July 28, 2026 13:10 — with
GitHub Actions
Inactive
cprecioso
temporarily deployed
to
fly-deploy-test
July 29, 2026 11:44 — with
GitHub Actions
Inactive
cprecioso
temporarily deployed
to
railway-deploy-test
July 29, 2026 11:44 — with
GitHub Actions
Inactive
cprecioso
temporarily deployed
to
fly-deploy-test
July 29, 2026 14:34 — with
GitHub Actions
Inactive
cprecioso
temporarily deployed
to
railway-deploy-test
July 29, 2026 14:34 — with
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously, each e2e test case was built up as one big bash one-liner
(ShellCommandBuilder produced shell command strings folded with ' && ')
and executed via 'shell'. Snapshot tests additionally executed their
commands at test-tree construction time, before tasty even started.
Now each test case is a list of native Haskell steps:
cwd from the step context, env var modifiers, stdin support, output
captured for assertions and teed to the per-test log, and process-group
termination on interruption (now also for regular tests).
of printf/base64/awk/grep/rsync hacks, making steps platform-agnostic.
(tasty-golden Advanced API) that run their steps at test runtime,
compare current vs golden snapshot dirs reporting all mismatches at
once, and keep the accept-all and --accept workflows working.
point at the exact failing step.
Long-running process support (FIXME'd tests) is not implemented yet,
but Command.startCommand/terminate is designed to support it.
See #3404
https://claude.ai/code/session_01HLhBbhP5j1tGKcqPLXu8oF