Skip to content

Commit 5013033

Browse files
aaronskyluispadron
andauthored
[testing] expose the xcresult bundle to post action binaries for processing (#2711)
Allows for customization of xcresult handling outside of the test runner. --------- Co-authored-by: Luis Padron <[email protected]>
1 parent 7cb3c8e commit 5013033

File tree

6 files changed

+31
-9
lines changed

6 files changed

+31
-9
lines changed

apple/testing/default_runner/ios_test_runner.template.sh

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -269,6 +269,7 @@ else
269269
)
270270
fi
271271

272+
# Run a pre-action binary, if provided.
272273
pre_action_binary=%(pre_action_binary)s
273274
"$pre_action_binary"
274275

@@ -279,6 +280,7 @@ cmd=("%(testrunner_binary)s"
279280
"${custom_xctestrunner_args[@]}")
280281
"${cmd[@]}" 2>&1 || test_exit_code=$?
281282

283+
# Run a post-action binary, if provided.
282284
post_action_binary=%(post_action_binary)s
283285
TEST_EXIT_CODE=$test_exit_code \
284286
"$post_action_binary"

apple/testing/default_runner/ios_xctestrun_runner.bzl

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -190,7 +190,7 @@ A binary to run prior to test execution. Runs after simulator creation. Sets the
190190
executable = True,
191191
cfg = "exec",
192192
doc = """
193-
A binary to run following test execution. Runs after testing but before test result handling and coverage processing. Sets the `$TEST_EXIT_CODE`, `$TEST_LOG_FILE`, and `$SIMULATOR_UDID` environment variables, in addition to any other variables available to the test runner.
193+
A binary to run following test execution. Runs after testing but before test result handling and coverage processing. Sets the `$TEST_EXIT_CODE`, `$TEST_LOG_FILE`, and `$SIMULATOR_UDID` environment variables, the `$TEST_XCRESULT_BUNDLE_PATH` environment variable if the test run produces an XCResult bundle, and any other variables available to the test runner.
194194
""",
195195
),
196196
"_simulator_creator": attr.label(

apple/testing/default_runner/ios_xctestrun_runner.template.sh

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -446,6 +446,7 @@ if (( ${#custom_xcodebuild_args[@]} )); then
446446
should_use_xcodebuild=true
447447
fi
448448

449+
# Run a pre-action binary, if provided.
449450
pre_action_binary=%(pre_action_binary)s
450451
SIMULATOR_UDID="$simulator_id" \
451452
"$pre_action_binary"
@@ -547,11 +548,20 @@ else
547548
|| test_exit_code=$?
548549
fi
549550

551+
# Run a post-action binary, if provided.
550552
post_action_binary=%(post_action_binary)s
551-
TEST_EXIT_CODE=$test_exit_code \
552-
TEST_LOG_FILE="$testlog" \
553-
SIMULATOR_UDID="$simulator_id" \
554-
"$post_action_binary"
553+
if [[ -n "${result_bundle_path:-}" ]]; then
554+
TEST_EXIT_CODE=$test_exit_code \
555+
TEST_LOG_FILE="$testlog" \
556+
SIMULATOR_UDID="$simulator_id" \
557+
TEST_XCRESULT_BUNDLE_PATH="$result_bundle_path" \
558+
"$post_action_binary"
559+
else
560+
TEST_EXIT_CODE=$test_exit_code \
561+
TEST_LOG_FILE="$testlog" \
562+
SIMULATOR_UDID="$simulator_id" \
563+
"$post_action_binary"
564+
fi
555565

556566
if [[
557567
"$test_exit_code" -eq 0 &&

apple/testing/default_runner/macos_test_runner.bzl

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -138,7 +138,7 @@ A binary to run prior to test execution. Sets any environment variables availabl
138138
executable = True,
139139
cfg = "exec",
140140
doc = """
141-
A binary to run following test execution. Runs after testing but before test result handling and coverage processing. Sets the `$TEST_EXIT_CODE` environment variable, in addition to any other variables available to the test runner.
141+
A binary to run following test execution. Runs after testing but before test result handling and coverage processing. Sets the `$TEST_EXIT_CODE`, `$TEST_LOG_FILE`, and `$TEST_XCRESULT_BUNDLE_PATH` environment variables, in addition to any other variables available to the test runner.
142142
""",
143143
),
144144
"_test_template": attr.label(

apple/testing/default_runner/macos_test_runner.template.sh

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -134,22 +134,32 @@ if [[ "$XML_OUTPUT_FILE" != /* ]]; then
134134
export XML_OUTPUT_FILE="$PWD/$XML_OUTPUT_FILE"
135135
fi
136136

137+
138+
# Run a pre-action binary, if provided.
137139
pre_action_binary=%(pre_action_binary)s
138140
"$pre_action_binary"
139141

142+
readonly result_bundle_path="$TEST_UNDECLARED_OUTPUTS_DIR/tests.xcresult"
143+
# TEST_UNDECLARED_OUTPUTS_DIR isn't cleaned up with multiple retries of flaky tests
144+
rm -rf "$result_bundle_path"
145+
140146
test_exit_code=0
147+
readonly testlog="$TEST_TMP_DIR/test.log"
141148
# Run xcodebuild with the xctestrun file just created. If the test failed, this
142149
# command will return non-zero, which is enough to tell bazel that the test
143150
# failed.
144-
rm -rf "$TEST_UNDECLARED_OUTPUTS_DIR/tests.xcresult"
145151
xcodebuild test-without-building \
146152
-destination "platform=macOS" \
147-
-resultBundlePath "$TEST_UNDECLARED_OUTPUTS_DIR/tests.xcresult" \
153+
-resultBundlePath "$result_bundle_path" \
148154
-xctestrun "$XCTESTRUN" \
155+
2>&1 | tee -i "$testlog" \
149156
|| test_exit_code=$?
150157

158+
# Run a post-action binary, if provided.
151159
post_action_binary=%(post_action_binary)s
152160
TEST_EXIT_CODE=$test_exit_code \
161+
TEST_LOG_FILE="$testlog" \
162+
TEST_XCRESULT_BUNDLE_PATH="$result_bundle_path" \
153163
"$post_action_binary"
154164

155165
if [[ "$test_exit_code" -ne 0 ]]; then

doc/rules-ios.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -718,7 +718,7 @@ in Xcode.
718718
| <a id="ios_xctestrun_runner-destination_timeout"></a>destination_timeout | Use the specified timeout when searching for a destination device. The default is 30 seconds. | Integer | optional | `0` |
719719
| <a id="ios_xctestrun_runner-device_type"></a>device_type | The device type of the iOS simulator to run test. The supported types correspond to the output of `xcrun simctl list devicetypes`. E.g., iPhone X, iPad Air. By default, it reads from --ios_simulator_device or falls back to some device. | String | optional | `""` |
720720
| <a id="ios_xctestrun_runner-os_version"></a>os_version | The os version of the iOS simulator to run test. The supported os versions correspond to the output of `xcrun simctl list runtimes`. E.g., 15.5. By default, it reads --ios_simulator_version and then falls back to the latest supported version. | String | optional | `""` |
721-
| <a id="ios_xctestrun_runner-post_action"></a>post_action | A binary to run following test execution. Runs after testing but before test result handling and coverage processing. Sets the `$TEST_EXIT_CODE`, `$TEST_LOG_FILE`, and `$SIMULATOR_UDID` environment variables, in addition to any other variables available to the test runner. | <a href="https://bazel.build/concepts/labels">Label</a> | optional | `None` |
721+
| <a id="ios_xctestrun_runner-post_action"></a>post_action | A binary to run following test execution. Runs after testing but before test result handling and coverage processing. Sets the `$TEST_EXIT_CODE`, `$TEST_LOG_FILE`, and `$SIMULATOR_UDID` environment variables, the `$TEST_XCRESULT_BUNDLE_PATH` environment variable if the test run produces an XCResult bundle, and any other variables available to the test runner. | <a href="https://bazel.build/concepts/labels">Label</a> | optional | `None` |
722722
| <a id="ios_xctestrun_runner-pre_action"></a>pre_action | A binary to run prior to test execution. Runs after simulator creation. Sets the `$SIMULATOR_UDID` environment variable, in addition to any other variables available to the test runner. | <a href="https://bazel.build/concepts/labels">Label</a> | optional | `None` |
723723
| <a id="ios_xctestrun_runner-random"></a>random | Whether to run the tests in random order to identify unintended state dependencies. | Boolean | optional | `False` |
724724
| <a id="ios_xctestrun_runner-reuse_simulator"></a>reuse_simulator | Toggle simulator reuse. The default behavior is to reuse an existing device of the same type and OS version. When disabled, a new simulator is created before testing starts and shutdown when the runner completes. | Boolean | optional | `True` |

0 commit comments

Comments
 (0)