Open
Conversation
Randomizes test dispatch order to catch hidden dependencies between tests. Without --shuffle, tests dispatch in registration order as before (buffered dispatch is always used, but the order is preserved). Uses a _TestOrdering union type (_InOrder | _Shuffled) to represent the ordering mode, with the seed resolved at parse time. The shuffle itself uses Rand.from_u64(seed) with Fisher-Yates via the stdlib Random.shuffle method. --list --shuffle=SEED shows the shuffled order so users can preview what a given seed produces without running anything. Closes #5075
SeanTAllen
added
the
changelog - added
ponylang-main
added
the
discuss during sync
SeanTAllen
added
do not merge
Tests now exercise the actual _Shuffled.apply method that PonyTest calls, rather than testing Rand.shuffle directly (which the random package already covers).
The previous tests only exercised Rand.shuffle through a thin wrapper. These tests create PonyTest instances with controlled args and a capture OutStream, verifying that --list --shuffle=SEED produces the correct shuffled output through the entire code path: argument parsing, buffered name collection, shuffle application, and seed output formatting.
Matches the convention used by every other package in the stdlib aggregator.
Both now start with (h, args, ...) so the shared prefix is the same.
Each test now passes its own TestList to _RunList/_RunListWith, so the test input is visible at each call site and changing one test's input doesn't break the others.
Each test now creates its own object literal implementing TestList, so changing one test's input can't break the others.
Instead of testing individual hardcoded seeds, run 10 seeds through the full PonyTest --list flow and verify that the resulting test orderings vary across seeds. Each seed exercises the complete code path: argument parsing, buffered name collection, shuffle, and output. Keep the seed-zero and preserves-order tests as specific edge case and baseline tests respectively.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements RFC #82. PonyTest gets a
--shuffle[=SEED]option that randomizes test dispatch order to catch hidden test coupling.--shufflegenerates a random seed fromTime.cycles()--shuffle=SEEDuses a specific U64 seed for reproducibilityTest seed: Nbefore any test output--list --shuffle=SEEDpreviews the shuffled order without running tests--sequential, and exclusion groupsTest dispatch is now always buffered (tests are collected during
apply()and dispatched in_all_tests_applied()). Without--shuffle, dispatch order matches registration order. The ordering mode is represented as a_TestOrderingunion type (_InOrder | _Shuffled) rather than boolean flags.Adds 5 unit tests for shuffle behavior (determinism, order change, element preservation, seed differentiation, seed 0 validity) and aggregates them into the stdlib test suite.
Closes #5075