Skip to content

Commit d8d63e2

Browse files
authored
Reduce Test Suite test concurrency to reduce test flakes (#3979)
# Description of Changes <!-- Please describe your change, mention any related tickets, and so on here. --> This PR started as an experiment to see if restricting the amount of test job threads would reduce the flakiness in the `Test Suite` CI job. I have tested several values here (`1`, `2` and `10`) and I believe 2 is very stable. I'm also very confused with this change because it seems like somehow the `Test Suite` CI job is now faster than it was before: Before: <img width="502" height="57" alt="image" src="https://github.com/user-attachments/assets/d944041c-8ccc-4382-a8a7-06e7ff5cba12" /> After: <img width="481" height="46" alt="image" src="https://github.com/user-attachments/assets/49b89143-b201-4a1a-820c-13e5b3feda76" /> It's possible that having less test threads means less cpu contention on the github runners, but I'm not completely sure. It could also somehow be less lock contention in SpacetimeDB. I have looked at the output of the `Test Suite` job and it does appear that all tests are running. # API and ABI breaking changes <!-- If this is an API or ABI breaking change, please apply the corresponding GitHub label. --> None # Expected complexity level and risk 0 - this only updates our workflow and has no impact on the code. <!-- How complicated do you think these changes are? Grade on a scale from 1 to 5, where 1 is a trivial change, and 5 is a deep-reaching and complex change. This complexity rating applies not only to the complexity apparent in the diff, but also to its interactions with existing and future code. If you answered more than a 2, explain what is complex about the PR, and what other components it interacts with in potentially concerning ways. --> # Testing <!-- Describe any testing you've done, and any testing you'd like your reviewers to do, so that you're confident that all the changes work as expected! --> - [x] CI is passing and `Test Suite` is less flakey --------- Signed-off-by: John Detter <4099508+jdetter@users.noreply.github.com>
1 parent af4d3f3 commit d8d63e2

1 file changed

Lines changed: 2 additions & 1 deletion

File tree

tools/ci/src/main.rs

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -237,7 +237,8 @@ fn main() -> Result<()> {
237237
Some(CiCmd::Test) => {
238238
// TODO: This doesn't work on at least user Linux machines, because something here apparently uses `sudo`?
239239

240-
cmd!("cargo", "test", "--all", "--", "--skip", "unreal").run()?;
240+
// cmd!("cargo", "test", "--all", "--", "--skip", "unreal").run()?;
241+
cmd!("cargo", "test", "--all", "--", "--test-threads=2", "--skip", "unreal").run()?;
241242
// TODO: This should check for a diff at the start. If there is one, we should alert the user
242243
// that we're disabling diff checks because they have a dirty git repo, and to re-run in a clean one
243244
// if they want those checks.

0 commit comments

Comments
 (0)