Skip to content

Refactor waspc e2e tests from shell-script DSL to native Haskell steps - #4294

Open
cprecioso wants to merge 21 commits into
mainfrom
claude/session-8f8605
Open

Refactor waspc e2e tests from shell-script DSL to native Haskell steps#4294
cprecioso wants to merge 21 commits into
mainfrom
claude/session-8f8605

Conversation

@cprecioso

Copy link
Copy Markdown
Member

Previously, each e2e test case was built up as one big bash one-liner
(ShellCommandBuilder produced shell command strings folded with ' && ')
and executed via 'shell'. Snapshot tests additionally executed their
commands at test-tree construction time, before tasty even started.

Now each test case is a list of native Haskell steps:

  • Step (ReaderT context IO) replaces the pure Reader shell-string builder.
  • Commands are first-class values run via proc (no shell), with explicit
    cwd from the step context, env var modifiers, stdin support, output
    captured for assertions and teed to the per-test log, and process-group
    termination on interruption (now also for regular tests).
  • File operations and assertions use System.Directory & friends instead
    of printf/base64/awk/grep/rsync hacks, making steps platform-agnostic.
  • Snapshot tests are now single directory-level golden tests
    (tasty-golden Advanced API) that run their steps at test runtime,
    compare current vs golden snapshot dirs reporting all mismatches at
    once, and keep the accept-all and --accept workflows working.
  • Per-step log sections and step-aware failure messages make failures
    point at the exact failing step.

Long-running process support (FIXME'd tests) is not implemented yet,
but Command.startCommand/terminate is designed to support it.

See #3404

https://claude.ai/code/session_01HLhBbhP5j1tGKcqPLXu8oF

claude added 3 commits June 10, 2026 14:42
Previously, each e2e test case was built up as one big bash one-liner
(ShellCommandBuilder produced shell command strings folded with ' && ')
and executed via 'shell'. Snapshot tests additionally executed their
commands at test-tree construction time, before tasty even started.

Now each test case is a list of native Haskell steps:
- Step (ReaderT context IO) replaces the pure Reader shell-string builder.
- Commands are first-class values run via proc (no shell), with explicit
  cwd from the step context, env var modifiers, stdin support, output
  captured for assertions and teed to the per-test log, and process-group
  termination on interruption (now also for regular tests).
- File operations and assertions use System.Directory & friends instead
  of printf/base64/awk/grep/rsync hacks, making steps platform-agnostic.
- Snapshot tests are now single directory-level golden tests
  (tasty-golden Advanced API) that run their steps at test runtime,
  compare current vs golden snapshot dirs reporting all mismatches at
  once, and keep the accept-all and --accept workflows working.
- Per-step log sections and step-aware failure messages make failures
  point at the exact failing step.

Long-running process support (FIXME'd tests) is not implemented yet,
but Command.startCommand/terminate is designed to support it.

See #3404

https://claude.ai/code/session_01HLhBbhP5j1tGKcqPLXu8oF
GHC's file locking forbids reading a file the same process has open for
writing, so close the log before reading its content into the failure
message (a failed test won't log anything further anyway).

https://claude.ai/code/session_01HLhBbhP5j1tGKcqPLXu8oF
@cprecioso
cprecioso requested a review from a team as a code owner June 11, 2026 06:30
@cprecioso
cprecioso requested review from infomiho and removed request for a team June 11, 2026 06:30
@cprecioso cprecioso self-assigned this Jun 11, 2026
@cprecioso
cprecioso removed the request for review from infomiho June 11, 2026 06:33
@pkg-pr-new

pkg-pr-new Bot commented Jun 11, 2026

Copy link
Copy Markdown

Open in StackBlitz

@wasp.sh/spec

npx https://pkg.pr.new/wasp-lang/wasp/@wasp.sh/spec@4294

@wasp.sh/wasp-cli

npx https://pkg.pr.new/wasp-lang/wasp/@wasp.sh/wasp-cli@4294

@wasp.sh/wasp-cli-darwin-arm64-unknown

npx https://pkg.pr.new/wasp-lang/wasp/@wasp.sh/wasp-cli-darwin-arm64-unknown@4294

@wasp.sh/wasp-cli-darwin-x64-unknown

npx https://pkg.pr.new/wasp-lang/wasp/@wasp.sh/wasp-cli-darwin-x64-unknown@4294

@wasp.sh/wasp-cli-linux-arm64-glibc

npx https://pkg.pr.new/wasp-lang/wasp/@wasp.sh/wasp-cli-linux-arm64-glibc@4294

@wasp.sh/wasp-cli-linux-x64-glibc

npx https://pkg.pr.new/wasp-lang/wasp/@wasp.sh/wasp-cli-linux-x64-glibc@4294

@wasp.sh/wasp-cli-linux-x64-musl

npx https://pkg.pr.new/wasp-lang/wasp/@wasp.sh/wasp-cli-linux-x64-musl@4294

commit: 7107c19

@cprecioso
cprecioso marked this pull request as draft June 11, 2026 07:01
@cprecioso
cprecioso temporarily deployed to fly-deploy-test June 11, 2026 07:07 — with GitHub Actions Inactive
@cprecioso
cprecioso temporarily deployed to railway-deploy-test June 11, 2026 07:07 — with GitHub Actions Inactive
Commit 83734c4 (#4276) added AGENTS.md and CLAUDE.md to
examples/kitchen-sink but the golden snapshot was not regenerated,
so the kitchen-sink snapshot test has been failing since.
Regenerated with --accept.

https://claude.ai/code/session_01HLhBbhP5j1tGKcqPLXu8oF
@cprecioso
cprecioso force-pushed the claude/session-8f8605 branch from 3a4ecbc to 35a63c0 Compare June 11, 2026 08:39
@cprecioso
cprecioso temporarily deployed to fly-deploy-test June 11, 2026 08:52 — with GitHub Actions Inactive
@cprecioso
cprecioso temporarily deployed to railway-deploy-test June 11, 2026 08:52 — with GitHub Actions Inactive
@cprecioso
cprecioso temporarily deployed to railway-deploy-test June 11, 2026 11:05 — with GitHub Actions Inactive
@cprecioso
cprecioso temporarily deployed to fly-deploy-test June 11, 2026 11:05 — with GitHub Actions Inactive
- Build output assertions in ViteBuildTest run a grep command and
  assert on its exit code again (matching the original tests'
  semantics), instead of checking file contents natively. The
  assertAnyFileInDirContainsText/assertNoFileInDirContainsText steps
  are gone.
- Test case steps are a monadic Step value again, mirroring the
  original ShellCommandBuilder shape: TestCase takes
  'sequence [step1, step2, ...]' and makeSnapshotTest sequences its
  steps internally.

https://claude.ai/code/session_01HLhBbhP5j1tGKcqPLXu8oF
@cprecioso
cprecioso force-pushed the claude/session-8f8605 branch from 45412a2 to 4994151 Compare June 11, 2026 11:34
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jun 11, 2026

Copy link
Copy Markdown

Deploying wasp-docs-on-main with  Cloudflare Pages  Cloudflare Pages

Latest commit: 4901612
Status: ✅  Deploy successful!
Preview URL: https://cc886cb8.wasp-docs-on-main.pages.dev
Branch Preview URL: https://claude-session-8f8605.wasp-docs-on-main.pages.dev

View logs

Since Step is a monad, a test case reads most naturally as a do block:

  TestCase "succeed-uncompiled-project" $ do
    createTestWaspProject minimalStarterTemplate
    inTestWaspProjectDir $ do
      runCommand waspCliCompile
      assertDirExists ".wasp"

TestCase, makeSnapshotTest and inTestWaspProjectDir/
inSnapshotWaspProjectDir now take a single Step instead of a list,
and helpers like mapM_/forM_ replace manual list building.

https://claude.ai/code/session_01HLhBbhP5j1tGKcqPLXu8oF
@cprecioso
cprecioso temporarily deployed to fly-deploy-test June 11, 2026 12:10 — with GitHub Actions Inactive
@cprecioso
cprecioso temporarily deployed to railway-deploy-test June 11, 2026 12:10 — with GitHub Actions Inactive
cprecioso and others added 2 commits June 30, 2026 17:04
- Rename Step env fields to context/logger
- Add HasWaspProjectContext to unify the create/enter WaspProject steps (createWaspProject/inWaspProjectDir) across the test and snapshot contexts
- Use waspCliDbReset in the db-reset fail-outside-project case
- Specialize the migration-dir assertion and drop the single-use assertDirHasSubdirWithNameContaining
- Rename isMigrationDirNameWithDatePrefix to isUnnormalizedMigrationDirName

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Use Test.Tasty.Golden.Advanced.goldenTest the intended way: the golden and tested values are the snapshot contents (each content-checked file's bytes), extracted in the value-producing actions, with the comparator only comparing values. Previously the values were directory paths and the comparator did all the file reading and diffing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@cprecioso
cprecioso temporarily deployed to railway-deploy-test June 30, 2026 15:16 — with GitHub Actions Inactive
@cprecioso
cprecioso temporarily deployed to fly-deploy-test June 30, 2026 15:16 — with GitHub Actions Inactive
cprecioso and others added 2 commits June 30, 2026 17:54
- Shell-quote env var values and args in showCommand
- Extract assertCommandSucceeded/assertCommandFailed helpers
- Simplify e2eTests to a pure TestTree
- Use FilePath.</> for golden snapshot paths
- Remove unused loggerLogFile export

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@cprecioso
cprecioso temporarily deployed to railway-deploy-test June 30, 2026 16:41 — with GitHub Actions Inactive
@cprecioso
cprecioso temporarily deployed to fly-deploy-test June 30, 2026 16:41 — with GitHub Actions Inactive
@cprecioso
cprecioso temporarily deployed to railway-deploy-test July 2, 2026 09:54 — with GitHub Actions Inactive
@cprecioso
cprecioso temporarily deployed to fly-deploy-test July 2, 2026 09:54 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

This pull request is now stale.
That means that it hasn't seen any activity for a while, and needs to be updated or addressed before it can be merged.

Next steps if the PR is still relevant, and you are able to devote time to it:

  • If you received any questions or feedback, please address them.
  • Merge the latest main into your branch to ensure that you are up to date.
  • If you are still working on the changes, please leave a comment telling us so.
  • Otherwise, just leave a comment explaining why the PR is still relevant.

If no action is taken, this PR will be automatically closed in 7 days.

@github-actions github-actions Bot added the Stale label Jul 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This pull request is now closed.
This doesn't mean that we don't care about it anymore, but rather that it has been inactive for a while and needs some attention to keep it relevant.

Next steps if the PR is still relevant:

  • Follow the next steps in the message above.
  • Create a new PR, linking to this one.

@github-actions github-actions Bot closed this Jul 18, 2026
@cprecioso cprecioso reopened this Jul 28, 2026
@cprecioso cprecioso removed the Stale label Jul 28, 2026
Conflicts were in the two e2e test files this branch rewrote:

- `e2e-tests/Tests/ViteBuildTest.hs`: main only reflowed two `TestCase`
  expressions in the old shell-command DSL, which this branch replaced with
  step-based test cases. Kept ours.

- `e2e-tests/Main.hs`: kept our pure `e2eTests :: TestTree` (snapshot tests now
  run under tasty instead of being prepared eagerly in IO), and ported main's
  concurrency cap from #4449 onto tasty's `NumThreads` so we don't lose the
  fix for CPU oversubscription. `WASP_E2E_TEST_MAX_JOBS` still overrides it,
  now for all test cases rather than just snapshot preparation.

Follow-ups needed after the textual merge:

- Restored `async` in `waspc-e2e-tests` build-depends (main dropped it when it
  stopped using `mapConcurrently`, but our `Command.hs` uses it) and dropped
  `unliftio`, which main added for `pooledMapConcurrentlyN` and is now unused.
- Reformatted `waspc.cabal` and `e2e-tests/SnapshotTest.hs` with the tool
  versions main pins (cabal-gild 1.8.4.1, ormolu 0.8.1.1).
@cprecioso
cprecioso temporarily deployed to railway-deploy-test July 28, 2026 08:50 — with GitHub Actions Inactive
- Start command output drains before writing stdin in startCommand, so a
  large stdin write can't deadlock against a child blocked on full,
  undrained output pipes.
- Extract commonIgnoredFilePathFilters shared by the snapshot manifest
  (existence) check and the content check, so the two lists can't drift.
- Extract makeWaspProjectContext into Context, keeping the "wasp-app"
  project naming convention in one place.
@cprecioso
cprecioso temporarily deployed to fly-deploy-test July 28, 2026 13:10 — with GitHub Actions Inactive
@cprecioso
cprecioso temporarily deployed to railway-deploy-test July 28, 2026 13:10 — with GitHub Actions Inactive
@cprecioso
cprecioso temporarily deployed to fly-deploy-test July 29, 2026 11:44 — with GitHub Actions Inactive
@cprecioso
cprecioso temporarily deployed to railway-deploy-test July 29, 2026 11:44 — with GitHub Actions Inactive
@cprecioso
cprecioso temporarily deployed to fly-deploy-test July 29, 2026 14:34 — with GitHub Actions Inactive
@cprecioso
cprecioso temporarily deployed to railway-deploy-test July 29, 2026 14:34 — with GitHub Actions Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants