-
Notifications
You must be signed in to change notification settings - Fork 46
Convert test infrastructure to tasty-golden based testsuite #450
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
jespercockx
merged 12 commits into
master
from
copilot/improve-agda2hs-test-infrastructure
Nov 29, 2025
Merged
Changes from 6 commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
650442c
Initial plan
Copilot c876ce9
Convert test infrastructure to tasty-golden based testsuite
Copilot 34e2251
Fix test runner for module discovery and UTF-8 encoding
Copilot 40d205d
Convert test infrastructure to tasty-golden based testsuite
Copilot 526e782
Address PR review feedback
Copilot 1bafe15
Remove unused UTCTime import
Copilot f6c7d62
Fix Makefile TAB characters and simplify sortByModTime
Copilot ecacb2d
Fix Makefile for CI
andreasabel 883e123
Attempt to fix nix CI
andreasabel 14b7595
Fix nix CI: add diffutils parameter to agda2hs.nix
Copilot 9df9f43
Disable tests in nix build to fix circular dependency
Copilot 6ce23e5
Address code style review comments
Copilot File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| # Developer Documentation | ||
|
|
||
| ## Running the Test Suite | ||
|
|
||
| The test suite uses `tasty-golden` for golden value testing. Tests are organized into two categories: | ||
|
|
||
| - **Succeed tests** (`test/Succeed/`): Agda files that should compile successfully with agda2hs. The generated `.hs` files are the golden values. | ||
| - **Fail tests** (`test/Fail/`): Agda files that should fail to compile. The error messages (in `.err` files) are the golden values. | ||
|
|
||
| ### Running All Tests | ||
|
|
||
| ```bash | ||
| # Run all tests (includes whitespace check and container tests) | ||
| make test | ||
|
|
||
| # Or run just the agda2hs tests | ||
| cabal test agda2hs-test | ||
| ``` | ||
|
|
||
| ### Running Specific Test Categories | ||
|
|
||
| ```bash | ||
| # Run only successful tests | ||
| make succeed | ||
|
|
||
| # Run only failing tests | ||
| make fail | ||
|
|
||
| # Or using cabal directly with pattern matching | ||
| cabal test agda2hs-test --test-options='-p Succeed' | ||
| cabal test agda2hs-test --test-options='-p Fail' | ||
| ``` | ||
|
|
||
| ### Running Individual Tests | ||
|
|
||
| ```bash | ||
| # Run a specific test by name | ||
| cabal test agda2hs-test --test-options='-p /Fixities/' | ||
|
|
||
| # Run tests matching a pattern | ||
| cabal test agda2hs-test --test-options='-p /Issue/' | ||
| ``` | ||
|
|
||
| ## Updating Golden Values | ||
|
|
||
| When you make changes that intentionally affect the test output, you need to update the golden values: | ||
|
|
||
| ```bash | ||
| # Update all golden values | ||
| make golden | ||
|
|
||
| # Update only successful test golden values | ||
| make golden-succeed | ||
|
|
||
| # Update only failing test golden values | ||
| make golden-fail | ||
|
|
||
| # Or using cabal directly | ||
| cabal test agda2hs-test --test-options='--accept' | ||
| ``` | ||
|
|
||
| After updating golden values, review the changes with `git diff` to ensure they are correct before committing. | ||
|
|
||
| ## Adding New Tests | ||
|
|
||
| ### Adding a Succeed Test | ||
|
|
||
| 1. Create a new `.agda` file in `test/Succeed/` | ||
| 2. Run `make golden-succeed` to generate the golden `.hs` file | ||
| 3. Verify the generated Haskell code is correct | ||
| 4. Commit both the `.agda` and `.hs` files | ||
|
|
||
| ### Adding a Fail Test | ||
|
|
||
| 1. Create a new `.agda` file in `test/Fail/` | ||
| 2. Run `make golden-fail` to generate the golden `.err` file | ||
| 3. Verify the error message is correct | ||
| 4. Commit both the `.agda` and `.err` files | ||
|
|
||
| ## Test Ordering | ||
|
|
||
| Tests are ordered by: | ||
| 1. Modification date of the `.agda` file (newest first) | ||
| 2. Modification date of the golden value file (newest first) | ||
| 3. Alphabetically | ||
|
|
||
| This ordering ensures that recently modified tests appear first in the output, making it easier to focus on tests you're actively working on. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,55 +1,63 @@ | ||
| .PHONY : install agda repl libHtml testContainers test test-on-CI testHtml golden clean docs | ||
| .PHONY : build install agda repl libHtml testContainers test succeed fail golden golden-succeed golden-fail clean docs fixWhitespace checkWhitespace | ||
|
|
||
| FILES = $(shell find src -type f) | ||
|
|
||
| build : | ||
| cabal build | ||
| cabal build | ||
|
|
||
| install : | ||
| cabal install --overwrite-policy=always | ||
| cabal install --overwrite-policy=always | ||
|
|
||
| agda : | ||
| cabal install Agda --program-suffix=-erased --overwrite-policy=always | ||
| cabal install Agda --program-suffix=-erased --overwrite-policy=always | ||
|
|
||
| repl : | ||
| cabal repl # e.g. `:set args -itest -otest/build test/AllTests.agda ... main ... :r ... main` | ||
| cabal repl | ||
|
|
||
| libHtml : | ||
| cabal run agda2hs -- --html --include-path lib/base lib/base/Haskell/Prelude.agda | ||
| cp html/Haskell.Prelude.html html/index.html | ||
|
|
||
| test/agda2hs : $(FILES) | ||
| cabal install agda2hs --overwrite-policy=always --installdir=test --install-method=copy | ||
| cabal run agda2hs -- --html --include-path lib/base lib/base/Haskell/Prelude.agda | ||
| cp html/Haskell.Prelude.html html/index.html | ||
|
|
||
| testContainers: | ||
| cd ./lib/containers && ./generate-haskell.sh && cabal build containers-prop | ||
| cd ./lib/containers && ./generate-haskell.sh && cabal build containers-prop | ||
|
|
||
| # Run all tests | ||
| test : checkWhitespace succeed fail testContainers | ||
|
|
||
| # Run only successful tests | ||
| succeed : | ||
| cabal test agda2hs-test --test-options='-p Succeed' | ||
|
|
||
| test : checkWhitespace test-on-CI | ||
| # Run only failing tests | ||
| fail : | ||
| cabal test agda2hs-test --test-options='-p Fail' | ||
|
|
||
| test-on-CI : test/agda2hs testContainers | ||
| make -C test | ||
| # Update all golden values | ||
| golden : golden-succeed golden-fail | ||
|
|
||
| testHtml : test/agda2hs | ||
| make -C test html | ||
| # Update golden values for successful tests | ||
| golden-succeed : | ||
| cabal test agda2hs-test --test-options='-p Succeed --accept' | ||
|
|
||
| golden : | ||
| make -C test golden | ||
| # Update golden values for failing tests | ||
| golden-fail : | ||
| cabal test agda2hs-test --test-options='-p Fail --accept' | ||
|
|
||
| clean : | ||
| make -C test clean | ||
| cabal clean | ||
| rm -rf test/_build/ | ||
|
|
||
| docs : | ||
| make -C docs html | ||
| make -C docs html | ||
|
|
||
| FIXW_BIN = fix-whitespace | ||
|
|
||
| .PHONY : fixWhitespace ## Fix the whitespace issue. | ||
| fixWhitespace : have-bin-$(FIXW_BIN) fix-whitespace.yaml | ||
jespercockx marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| $(FIXW_BIN) | ||
| $(FIXW_BIN) | ||
|
|
||
| .PHONY : checkWhitespace ## Check the whitespace issue without fixing it. | ||
| checkWhitespace : have-bin-$(FIXW_BIN) fix-whitespace.yaml | ||
| $(FIXW_BIN) --check | ||
| $(FIXW_BIN) --check | ||
|
|
||
| .PHONY : have-bin-% ## Installing binaries for developer services | ||
| .PHONY : have-bin-% | ||
jespercockx marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| have-bin-% : | ||
| @($* --help > /dev/null) || $(CABAL) install --ignore-project $* | ||
| @($* --help > /dev/null) || $(CABAL) install --ignore-project $* | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.