Fix non-deterministic test execution order in run-tests.sh

[Copilot]

Shell glob expansion order is filesystem-dependent, causing scripts/unit-tests/test-run-tests.sh to fail intermittently in CI when tests run in unexpected order (observed in Actions job 54569245475).

Changes

• Deterministic iteration: Replace glob iteration with sorted mapfile to guarantee alphabetical test order

  # Before: non-deterministic
  for test in scripts/lib/playground-tests/*.sh; do
  
  # After: deterministic (LC_ALL=C ensures consistent collation)
  mapfile -t tests < <(printf '%s\n' scripts/lib/playground-tests/*.sh | LC_ALL=C sort)
  for test in "${tests[@]}"; do

• Quote variables: Harden script against word-splitting ("$test", "$test_name", "$log_file")

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• wordpress.org
  • Triggering command: node /home/REDACTED/work/playground-tester/playground-tester/node_modules/@wp-playground/cli/cli.js build-snapshot --quiet --port=9400 --wp=REDACTED --outfile=./temp/wordpress.zip (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

Issue

The unit test scripts/unit-tests/test-run-tests.sh expects tests to run in a specific order (ast-sqlite-boot, asyncify-boot, jspi-boot). On some runners the shell glob scripts/lib/playground-tests/*.sh expands in a non-deterministic way causing the actual output order to differ and the test to fail (Observed failure in Actions job 54569245475 at ref 73fb63a).

Goal

Make test execution order deterministic and harden run-tests.sh by quoting variables and invoking test scripts safely. Create a branch and PR that updates scripts/run-tests.sh accordingly.

Changes to make

1. Replace the non-deterministic glob iteration:
   for test in scripts/lib/playground-tests/*.sh; do

   with a deterministic, sorted iteration using mapfile and sort:

   mapfile -t tests < <(printf '%s\n' scripts/lib/playground-tests/*.sh | LC_ALL=C sort)
   for test in "${tests[@]}"; do

   This ensures consistent ordering across environments.

2. Quote variables to avoid word-splitting issues and call test files safely. Replace instances such as:
   item_name=$(basename $item_path)
   test_name=$(basename $test .sh)
   result=$(./$test --blueprint "$blueprint_path" --wordpress "$wordpress_path" || true)
   parse_raw_logs --test-name $test_name --"$test_type" --item-name "$item_name" --input $log_file --output "$log_folder/error.json"

   with quoted equivalents:
   item_name=$(basename "$item_path")
   test_name=$(basename "$test" .sh)
   result=$(./"$test" --blueprint "$blueprint_path" --wordpress "$wordpress_path" || true)
   parse_raw_logs --test-name "$test_name" --"$test_type" --item-name "$item_name" --input "$log_file" --output "$log_folder/error.json"

3. Add a brief comment explaining why sorting is needed.

Patch (edit scripts/run-tests.sh)

--- a/scripts/run-tests.sh
+++ b/scripts/run-tests.sh
@@
-for test in scripts/lib/playground-tests/.sh; do
+# Iterate over tests in a deterministic, sorted order to ensure stable output
+# Sorting guarantees the same order across CI runners and prevents test failures
+mapfile -t tests < <(printf '%s\n' scripts/lib/playground-tests/.sh | LC_ALL=C sort)
+for test in "${tests[@]}"; do

• item_name=$(basename $item_path)
• test_name=$(basename $test .sh)
• log_folder="$item_path/$test_name"
• log_file="$log_folder/error.log"

• item_name=$(basename "$item_path")
• test_name=$(basename "$test" .sh)
• log_folder="$item_path/$test_name"
• log_file="$log_folder/error.log"
  @@

• result=$(./$test --blueprint "$blueprint_path" --wordpress "$wordpress_path" || true)

• result=$(./"$test" --blueprint "$blueprint_path" --wordpress "$wordpress_path" || true)
  @@

•    echo "$result" > "$log_file"
  

•    parse_raw_logs --test-name $test_name --"$test_type" --item-name "$item_name" --input $log_file --output "$log_folder/error.json"
  

•    echo "$result" > "$log_file"
  

•    parse_raw_logs --test-name "$test_name" --"$test_type" --item-name "$item_name" --input "$log_file" --output "$log_folder/error.json"
  

@@

• fatal_errors=$(get_number_of_errors_by_level "FATAL" "$log_folder/error.json")

• fatal_errors=$(get_number_of_errors_by_level "FATAL" "$log_folder/error.json")
  @@

• rm -rf "$temp_folder"

• rm -rf "$temp_folder"
  done

Rationale

The change fixes the nondeterministic ordering of playground-tests scripts which caused the unit test expecting a specific order to fail in CI (see job 54569245475 at ref 73fb63a). Quoting variables is additional hardening to prevent subtle failures on different environments.

Testing

Run the unit tests locally: scripts/unit-tests/test-run-tests.sh

Please create a pull request with these changes and a commit message: "Make playground test execution order deterministic by sorting tests".

References

• Failing job logs (Actions job id 54569245475): ref 73fb63a
• File to change: scripts/run-tests.sh at ref 73fb63a

This pull request was created as a result of the following prompt from Copilot chat.

Issue

The unit test scripts/unit-tests/test-run-tests.sh expects tests to run in a specific order (ast-sqlite-boot, asyncify-boot, jspi-boot). On some runners the shell glob scripts/lib/playground-tests/*.sh expands in a non-deterministic way causing the actual output order to differ and the test to fail (Observed failure in Actions job 54569245475 at ref 73fb63a).

Goal

Make test execution order deterministic and harden run-tests.sh by quoting variables and invoking test scripts safely. Create a branch and PR that updates scripts/run-tests.sh accordingly.

Changes to make

1. Replace the non-deterministic glob iteration:
   for test in scripts/lib/playground-tests/*.sh; do

   with a deterministic, sorted iteration using mapfile and sort:

   mapfile -t tests < <(printf '%s\n' scripts/lib/playground-tests/*.sh | LC_ALL=C sort)
   for test in "${tests[@]}"; do

   This ensures consistent ordering across environments.

2. Quote variables to avoid word-splitting issues and call test files safely. Replace instances such as:
   item_name=$(basename $item_path)
   test_name=$(basename $test .sh)
   result=$(./$test --blueprint "$blueprint_path" --wordpress "$wordpress_path" || true)
   parse_raw_logs --test-name $test_name --"$test_type" --item-name "$item_name" --input $log_file --output "$log_folder/error.json"

   with quoted equivalents:
   item_name=$(basename "$item_path")
   test_name=$(basename "$test" .sh)
   result=$(./"$test" --blueprint "$blueprint_path" --wordpress "$wordpress_path" || true)
   parse_raw_logs --test-name "$test_name" --"$test_type" --item-name "$item_name" --input "$log_file" --output "$log_folder/error.json"

3. Add a brief comment explaining why sorting is needed.

Patch (edit scripts/run-tests.sh)

--- a/scripts/run-tests.sh
+++ b/scripts/run-tests.sh
@@
-for test in scripts/lib/playground-tests/.sh; do
+# Iterate over tests in a deterministic, sorted order to ensure stable output
+# Sorting guarantees the same order across CI runners and prevents test failures
+mapfile -t tests < <(printf '%s\n' scripts/lib/playground-tests/.sh | LC_ALL=C sort)
+for test in "${tests[@]}"; do

• item_name=$(basename $item_path)
• test_name=$(basename $test .sh)
• log_folder="$item_path/$test_name"
• log_file="$log_folder/error.log"

• item_name=$(basename "$item_path")
• test_name=$(basename "$test" .sh)
• log_folder="$item_path/$test_name"
• log_file="$log_folder/error.log"
  @@

• result=$(./$test --blueprint "$blueprint_path" --wordpress "$wordpress_path" || true)

• result=$(./"$test" --blueprint "$blueprint_path" --wordpress "$wordpress_path" || true)
  @@

•    echo "$result" > "$log_file"
  

•    parse_raw_logs --test-name $test_name --"$test_type" --item-name "$item_name" --input $log_file --output "$log_folder/error.json"
  

•    echo "$result" > "$log_file"
  

•    parse_raw_logs --test-name "$test_name" --"$test_type" --item-name "$item_name" --input "$log_file" --output "$log_folder/error.json"
  

@@

• fatal_errors=$(get_number_of_errors_by_level "FATAL" "$log_folder/error.json")

• fatal_errors=$(get_number_of_errors_by_level "FATAL" "$log_folder/error.json")
  @@

• rm -rf "$temp_folder"

• rm -rf "$temp_folder"
  done

Rationale

The change fixes the nondeterministic ordering of playground-tests scripts which caused the unit test expecting a specific order to fail in CI (see job 54569245475 at ref 73fb63a). Quoting variables is additional hardening to prevent subtle failures on different environments.

Testing

Run the unit tests locally: scripts/unit-tests/test-run-tests.sh

Please create a pull request with these changes and a commit message: "Make playground test execution order deterministic by sorting tests".

References

• Failing job logs (Actions job id 54569245475): ref 73fb63a
• File to change: scripts/run-tests.sh at ref 73fb63a

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.