Skip to content

Commit 0518d72

Browse files
cipolleschimeta-codesync[bot]
cipolleschi
authored and
meta-codesync[bot]
committed
Add E2E test retry jobs for PRs to handle flakiness (#56581)
Summary: - Adds retry jobs (up to 2 retries) for all 4 E2E test jobs on PRs to handle flakiness - Original E2E jobs use `continue-on-error` on PRs so failures don't block the workflow; on `main`, behavior is unchanged and the existing `rerun-failed-jobs` mechanism continues to work - Each retry runs on a fresh runner (addressing environment-level flakes) and is triggered via step-level outcome captured as a job output - Added `overwrite: true` to artifact uploads in maestro composite actions so retry jobs don't conflict on artifact names ### How it works - **On PRs:** E2E job fails → `retry_1` triggers → if that fails → `retry_2` triggers. All have `continue-on-error` so the workflow stays green. - **On `main`:** `continue-on-error` is `false`, retry jobs are skipped (PR-only), and `rerun-failed-jobs` handles retries as before. ### Known limitation Since these are matrix jobs (Debug/Release), the job output uses the last-to-complete matrix combination's value. If only one flavor fails and the passing one finishes last, the retry may not trigger. In the common flakiness pattern (environment-level issues), both flavors tend to be affected, so this works well in practice. ## Changelog: [Internal] - Add E2E test retry jobs for PRs Pull Request resolved: #56581 Test Plan: - CI will validate the workflow syntax - On a PR where E2E tests pass: retry jobs should be skipped - On a PR where E2E tests fail due to flakiness: retry jobs should trigger and (ideally) pass on a fresh runner - On pushes to `main`: existing `rerun-failed-jobs` behavior is preserved (retry jobs are skipped) Reviewed By: cortinico Differential Revision: D102327900 Pulled By: cipolleschi fbshipit-source-id: e257e012357fdba442c302085ceed0667355272e
1 parent 3c64581 commit 0518d72

8 files changed

Lines changed: 519 additions & 254 deletions

File tree

‎.github/actions/maestro-android/action.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,7 @@ runs:
7777
if: always()
7878
with:
7979
name: e2e_android_${{ steps.normalize-app-id.outputs.app-id }}_report_${{ inputs.flavor }}_${{ inputs.emulator-arch }}_NewArch
80+
overwrite: true
8081
path: |
8182
report.xml
8283
screen.mp4
@@ -85,4 +86,5 @@ runs:
8586
uses: actions/upload-artifact@v6
8687
with:
8788
name: maestro-logs-android-${{ steps.normalize-app-id.outputs.app-id }}-${{ inputs.flavor }}-${{ inputs.emulator-arch }}-NewArch
89+
overwrite: true
8890
path: /tmp/MaestroLogs

‎.github/actions/maestro-ios/action.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ runs:
6969
uses: actions/upload-artifact@v6
7070
with:
7171
name: e2e_ios_${{ inputs.app-id }}_report_${{ inputs.flavor }}_NewArch
72+
overwrite: true
7273
path: |
7374
video_record_1.mov
7475
video_record_2.mov
@@ -81,4 +82,5 @@ runs:
8182
uses: actions/upload-artifact@v6
8283
with:
8384
name: maestro-logs-${{ inputs.app-id }}-${{ inputs.flavor }}-NewArch
85+
overwrite: true
8486
path: /tmp/MaestroLogs

‎.github/workflows/e2e-android-rntester.yml‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
name: E2E Android RNTester
2+
3+
permissions:
4+
contents: read
5+
6+
on:
7+
workflow_call:
8+
inputs:
9+
fail-on-error:
10+
type: boolean
11+
default: false
12+
outputs:
13+
status:
14+
description: "The result of the E2E tests (success or failure)"
15+
value: ${{ jobs.report.outputs.status }}
16+
17+
jobs:
18+
test:
19+
runs-on: 4-core-ubuntu
20+
outputs:
21+
status: ${{ steps.report-status.outputs.status }}
22+
strategy:
23+
fail-fast: false
24+
matrix:
25+
flavor: [debug, release]
26+
steps:
27+
- name: Checkout
28+
uses: actions/checkout@v6
29+
- name: Setup node.js
30+
uses: ./.github/actions/setup-node
31+
- name: Install node dependencies
32+
uses: ./.github/actions/yarn-install
33+
- name: Download APK
34+
uses: actions/download-artifact@v7
35+
with:
36+
name: rntester-${{ matrix.flavor }}
37+
path: ./packages/rn-tester/android/app/build/outputs/apk/${{ matrix.flavor }}/
38+
- name: Print folder structure
39+
run: ls -lR ./packages/rn-tester/android/app/build/outputs/apk/${{ matrix.flavor }}/
40+
- name: Run E2E Tests
41+
id: run-tests
42+
continue-on-error: true
43+
uses: ./.github/actions/maestro-android
44+
timeout-minutes: 60
45+
with:
46+
app-path: ./packages/rn-tester/android/app/build/outputs/apk/${{ matrix.flavor }}/app-x86-${{ matrix.flavor }}.apk
47+
app-id: com.facebook.react.uiapp
48+
maestro-flow: ./packages/rn-tester/.maestro
49+
flavor: ${{ matrix.flavor }}
50+
- name: Report status
51+
id: report-status
52+
if: ${{ always() && steps.run-tests.outcome == 'failure' }}
53+
run: echo "status=failure" >> $GITHUB_OUTPUT
54+
55+
report:
56+
needs: test
57+
if: always()
58+
runs-on: ubuntu-latest
59+
outputs:
60+
status: ${{ steps.check.outputs.status }}
61+
steps:
62+
- id: check
63+
run: |
64+
if [[ "${{ needs.test.outputs.status }}" == "failure" ]]; then
65+
echo "status=failure" >> $GITHUB_OUTPUT
66+
else
67+
echo "status=success" >> $GITHUB_OUTPUT
68+
fi
69+
- name: Fail if needed
70+
if: ${{ inputs.fail-on-error && steps.check.outputs.status == 'failure' }}
71+
run: exit 1

‎.github/workflows/e2e-android-templateapp.yml‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
name: E2E Android Template App
2+
3+
permissions:
4+
contents: read
5+
6+
on:
7+
workflow_call:
8+
inputs:
9+
fail-on-error:
10+
type: boolean
11+
default: false
12+
outputs:
13+
status:
14+
description: "The result of the E2E tests (success or failure)"
15+
value: ${{ jobs.report.outputs.status }}
16+
17+
jobs:
18+
test:
19+
runs-on: 4-core-ubuntu
20+
outputs:
21+
status: ${{ steps.report-status.outputs.status }}
22+
strategy:
23+
fail-fast: false
24+
matrix:
25+
flavor: [debug, release]
26+
steps:
27+
- name: Checkout
28+
uses: actions/checkout@v6
29+
- name: Setup node.js
30+
uses: ./.github/actions/setup-node
31+
- name: Run yarn
32+
uses: ./.github/actions/yarn-install
33+
- name: Set up JDK 17
34+
uses: actions/setup-java@v5
35+
with:
36+
java-version: '17'
37+
distribution: 'zulu'
38+
- name: Download Maven Local
39+
uses: actions/download-artifact@v7
40+
with:
41+
name: maven-local
42+
path: /tmp/react-native-tmp/maven-local
43+
- name: Download React Native Package
44+
uses: actions/download-artifact@v7
45+
with:
46+
name: react-native-package
47+
path: /tmp/react-native-tmp
48+
- name: Print /tmp folder
49+
run: ls -lR /tmp/react-native-tmp
50+
- name: Prepare artifacts
51+
id: prepare-artifacts
52+
run: |
53+
REACT_NATIVE_PKG=$(find /tmp/react-native-tmp -type f -name "*.tgz")
54+
echo "React Native tgs is $REACT_NATIVE_PKG"
55+
56+
MAVEN_LOCAL=/tmp/react-native-tmp/maven-local
57+
echo "Maven local path is $MAVEN_LOCAL"
58+
59+
# For stable branches, we want to use the stable branch of the template
60+
# In all the other cases, we want to use "main"
61+
BRANCH=${{ github.ref_name }}
62+
if ! [[ $BRANCH == *-stable* ]]; then
63+
BRANCH=main
64+
fi
65+
node ./scripts/e2e/init-project-e2e.js --projectName RNTestProject --currentBranch $BRANCH --directory /tmp/RNTestProject --pathToLocalReactNative $REACT_NATIVE_PKG
66+
67+
echo "Feed maven local to gradle.properties"
68+
cd /tmp/RNTestProject
69+
echo "react.internal.mavenLocalRepo=$MAVEN_LOCAL" >> android/gradle.properties
70+
71+
# Build
72+
cd android
73+
CAPITALIZED_FLAVOR=$(echo "${{ matrix.flavor }}" | awk '{print toupper(substr($0, 1, 1)) substr($0, 2)}')
74+
./gradlew assemble$CAPITALIZED_FLAVOR --no-daemon -PreactNativeArchitectures=x86
75+
76+
- name: Run E2E Tests
77+
id: run-tests
78+
continue-on-error: true
79+
uses: ./.github/actions/maestro-android
80+
timeout-minutes: 60
81+
with:
82+
app-path: /tmp/RNTestProject/android/app/build/outputs/apk/${{ matrix.flavor }}/app-${{ matrix.flavor }}.apk
83+
app-id: com.rntestproject
84+
maestro-flow: ./scripts/e2e/.maestro/
85+
install-java: 'false'
86+
flavor: ${{ matrix.flavor }}
87+
working-directory: /tmp/RNTestProject
88+
- name: Report status
89+
id: report-status
90+
if: ${{ always() && steps.run-tests.outcome == 'failure' }}
91+
run: echo "status=failure" >> $GITHUB_OUTPUT
92+
93+
report:
94+
needs: test
95+
if: always()
96+
runs-on: ubuntu-latest
97+
outputs:
98+
status: ${{ steps.check.outputs.status }}
99+
steps:
100+
- id: check
101+
run: |
102+
if [[ "${{ needs.test.outputs.status }}" == "failure" ]]; then
103+
echo "status=failure" >> $GITHUB_OUTPUT
104+
else
105+
echo "status=success" >> $GITHUB_OUTPUT
106+
fi
107+
- name: Fail if needed
108+
if: ${{ inputs.fail-on-error && steps.check.outputs.status == 'failure' }}
109+
run: exit 1

‎.github/workflows/e2e-ios-rntester.yml‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
name: E2E iOS RNTester
2+
3+
permissions:
4+
contents: read
5+
6+
on:
7+
workflow_call:
8+
inputs:
9+
fail-on-error:
10+
type: boolean
11+
default: false
12+
outputs:
13+
status:
14+
description: "The result of the E2E tests (success or failure)"
15+
value: ${{ jobs.report.outputs.status }}
16+
17+
jobs:
18+
test:
19+
runs-on: macos-15-large
20+
outputs:
21+
status: ${{ steps.report-status.outputs.status }}
22+
strategy:
23+
fail-fast: false
24+
matrix:
25+
flavor: [Debug, Release]
26+
steps:
27+
- name: Checkout
28+
uses: actions/checkout@v6
29+
- name: Setup Node.js
30+
uses: ./.github/actions/setup-node
31+
- name: Download App
32+
uses: actions/download-artifact@v7
33+
with:
34+
name: RNTesterApp-NewArch-${{ matrix.flavor }}
35+
path: /tmp/RNTesterBuild/RNTester.app
36+
- name: Check downloaded folder content
37+
run: ls -lR /tmp/RNTesterBuild
38+
- name: Setup xcode
39+
uses: ./.github/actions/setup-xcode
40+
- name: Run E2E Tests
41+
id: run-tests
42+
continue-on-error: true
43+
uses: ./.github/actions/maestro-ios
44+
with:
45+
app-path: "/tmp/RNTesterBuild/RNTester.app"
46+
app-id: com.meta.RNTester.localDevelopment
47+
maestro-flow: ./packages/rn-tester/.maestro/
48+
flavor: ${{ matrix.flavor }}
49+
- name: Report status
50+
id: report-status
51+
if: ${{ always() && steps.run-tests.outcome == 'failure' }}
52+
run: echo "status=failure" >> $GITHUB_OUTPUT
53+
54+
report:
55+
needs: test
56+
if: always()
57+
runs-on: ubuntu-latest
58+
outputs:
59+
status: ${{ steps.check.outputs.status }}
60+
steps:
61+
- id: check
62+
run: |
63+
if [[ "${{ needs.test.outputs.status }}" == "failure" ]]; then
64+
echo "status=failure" >> $GITHUB_OUTPUT
65+
else
66+
echo "status=success" >> $GITHUB_OUTPUT
67+
fi
68+
- name: Fail if needed
69+
if: ${{ inputs.fail-on-error && steps.check.outputs.status == 'failure' }}
70+
run: exit 1

0 commit comments

Comments
 (0)