Skip to content

Use workflow_run to trigger emulator.wtf tests #6864

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
TimoPtr wants to merge 3 commits into
base: main
Choose a base branch
from
+151 −57
Open

Use workflow_run to trigger emulator.wtf tests #6864

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1815460
Use workflow_run to trigger emulator.wtf tests
TimoPtr May 19, 2026
1377053
Add concurrency and avoid running on failure
TimoPtr May 19, 2026
feb2e59
Tight version validation from 23 to 40
TimoPtr May 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 137 additions & 0 deletions .github/workflows/pr-emulator-wtf.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
name: PR Emulator.wtf

on: # yamllint disable-line rule:truthy
workflow_run:
workflows: ['Pull Request']
types: [completed]

concurrency:
group: ${{ github.workflow }}-${{ github.event.workflow_run.head_repository.full_name }}-${{ github.event.workflow_run.head_branch }}
cancel-in-progress: true

permissions: {}

jobs:
emulator_wtf:
name: "Instrumentation Test app"
runs-on: ubuntu-latest
environment: ui-test
if: >
github.event.workflow_run.event == 'pull_request'
&& github.event.workflow_run.conclusion != 'cancelled'
&& github.event.workflow_run.conclusion != 'skipped'
permissions:
id-token: write # Needed for OIDC authentication
actions: read # Needed for retrieving artifacts
steps:
# Halt before spending any emulator.wtf credits if the triggering Pull Request
# workflow did not succeed (publish_test_results still runs afterward to
# aggregate whatever results pr.yml produced).
- name: Halt if triggering workflow did not succeed
if: github.event.workflow_run.conclusion != 'success'
env:
CONCLUSION: ${{ github.event.workflow_run.conclusion }}
run: |
echo "::error::Triggering 'Pull Request' workflow concluded as '$CONCLUSION'; skipping emulator.wtf to save resources"
exit 1

Comment thread
TimoPtr marked this conversation as resolved.
- name: Download emulator.wtf inputs
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: emulator-wtf-inputs
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
path: emulator-wtf-inputs

- name: Load device list
id: devices
# The artifact comes from a PR build that may originate from a fork; validate
# every line strictly before piping the contents into GITHUB_OUTPUT to prevent
# heredoc-terminator injection.
run: |
if [ ! -s emulator-wtf-inputs/devices.txt ]; then
echo "::error::devices.txt is empty or missing"
exit 1
fi
while IFS= read -r line; do
if [[ ! "$line" =~ ^model=Pixel7,version=[0-9]+$ ]]; then
echo "::error::Invalid line in devices.txt"
exit 1
fi
Comment on lines +57 to +60
Copy link
Copy Markdown
Member Author

@TimoPtr TimoPtr May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jpelgrom I didn't know what to do there that's why I kept an acceptable range [23-40]

Copy link
Copy Markdown
Member

@jpelgrom jpelgrom May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fine for now, we have some time until reached and it's better than nothing

done < emulator-wtf-inputs/devices.txt
{
echo 'list<<DEVICES_EOF'
cat emulator-wtf-inputs/devices.txt
echo 'DEVICES_EOF'
} >> "$GITHUB_OUTPUT"

- uses: emulator-wtf/actions/configure-credentials@42753a2a6bda2e657741a7385131bf61db464899 # v1.0.0
with:
oidc-configuration-id: ${{ vars.EMULATOR_WTF_OIDC_CONFIGURATION_ID }}

- name: Run tests full
uses: emulator-wtf/actions/run-tests@42753a2a6bda2e657741a7385131bf61db464899 # v1.0.0
with:
devices: ${{ steps.devices.outputs.list }}
app: emulator-wtf-inputs/app/build/outputs/apk/full/debug/app-full-debug.apk
test: emulator-wtf-inputs/app/build/outputs/apk/androidTest/full/debug/app-full-debug-androidTest.apk
outputs-dir: build/test-results/full

- name: Run tests minimal
uses: emulator-wtf/actions/run-tests@42753a2a6bda2e657741a7385131bf61db464899 # v1.0.0
with:
devices: ${{ steps.devices.outputs.list }}
app: emulator-wtf-inputs/app/build/outputs/apk/minimal/debug/app-minimal-debug.apk
test: emulator-wtf-inputs/app/build/outputs/apk/androidTest/minimal/debug/app-minimal-debug-androidTest.apk
outputs-dir: build/test-results/minimal

- name: Upload test results
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: Instrumentation Test app results
path: |
**/build/test-results/**/TEST-*.xml
**/build/test-results/**/results.xml

- name: Upload test reports
if: failure()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: Instrumentation Test app reports
path: build/test-results/**

publish_test_results:
name: "Publish Tests Results"
needs: [emulator_wtf]
if: always()
runs-on: ubuntu-latest
permissions:
checks: write
pull-requests: write
actions: read
contents: read
steps:
- name: Download artifacts from pr.yml run
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
path: pr-artifacts

- name: Download emulator.wtf results from this run
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: Instrumentation Test app results
path: pr-artifacts/Instrumentation Test app results

- name: Publish Test Results
uses: EnricoMi/publish-unit-test-result-action@c950f6fb443cb5af20a377fd0dfaa78838901040 # v2.23.0
with:
commit: ${{ github.event.workflow_run.head_sha }}
event_name: ${{ github.event.workflow_run.event }}
comment_mode: "failures"
action_fail: true
files: |
pr-artifacts/**/TEST-*.xml
pr-artifacts/**/results.xml
71 changes: 14 additions & 57 deletions .github/workflows/pr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -319,13 +319,10 @@ jobs:
with:
test-name: Unit test

instrumentation_test_app:
name: "Instrumentation Test app"
build_emulator_wtf_apks:
name: "Build Emulator.wtf APKs"
needs: [lint, lockfiles, ktlint]
runs-on: ubuntu-latest
environment: ui-test
permissions:
id-token: write # Needed for OIDC authentication
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
Expand All @@ -340,42 +337,27 @@ jobs:
run: ./gradlew :app:assembleDebug :app:assembleAndroidTest -Pabis=x86,x86_64

- name: Build device list
Copy link
Copy Markdown
Member

@jpelgrom jpelgrom May 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not move this step to the new workflow? There is no actual building happening in this step, and then we have less upload/download and untrusted input to deal with.

Copy link
Copy Markdown
Member Author

@TimoPtr TimoPtr May 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It allow us to build the list of devices in a PR, otherwise we need to make a first PR to update the list since we execute the code from main not from the PR.

Copy link
Copy Markdown
Member

@jpelgrom jpelgrom May 19, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could use actions/checkout with a reference to the PR branch, no? That also works for forks based on what I can find. We don't actually use the files in the new workflow so no conflict.

Snippet I found, ref/repository should be changed to use github.event.workflow_run:

- name: Clone repository
  uses: actions/checkout
  with:
    fetch-depth: 0
    ref: ${{github.event.pull_request.head.ref}}
    repository: ${{github.event.pull_request.head.repo.full_name}}

Copy link
Copy Markdown
Member

@jpelgrom jpelgrom May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think this could work if moved and is a cleaner approach. You've already added the number validation in the other workflow, not too different from validating what is in the version file...

id: devices
run: |
MIN=$(grep '^androidSdk-min' gradle/libs.versions.toml | cut -d'"' -f2)
MAX=$(grep '^androidSdk-target' gradle/libs.versions.toml | cut -d'"' -f2)
{
echo 'list<<DEVICES_EOF'
for v in $(seq "$MIN" "$MAX"); do
echo "model=Pixel7,version=$v"
done
echo 'DEVICES_EOF'
} >> "$GITHUB_OUTPUT"

- uses: emulator-wtf/actions/configure-credentials@42753a2a6bda2e657741a7385131bf61db464899 # v1.0.0
with:
oidc-configuration-id: ${{ vars.EMULATOR_WTF_OIDC_CONFIGURATION_ID }} # This is retrieved from the emulator.wtf console

- name: Run tests full
uses: emulator-wtf/actions/run-tests@42753a2a6bda2e657741a7385131bf61db464899 # v1.0.0
with:
devices: ${{ steps.devices.outputs.list }}
app: app/build/outputs/apk/full/debug/app-full-debug.apk
test: app/build/outputs/apk/androidTest/full/debug/app-full-debug-androidTest.apk
outputs-dir: build/test-results/full
} > devices.txt

- name: Run tests minimal
uses: emulator-wtf/actions/run-tests@42753a2a6bda2e657741a7385131bf61db464899 # v1.0.0
with:
devices: ${{ steps.devices.outputs.list }}
app: app/build/outputs/apk/minimal/debug/app-minimal-debug.apk
test: app/build/outputs/apk/androidTest/minimal/debug/app-minimal-debug-androidTest.apk
outputs-dir: build/test-results/minimal

- uses: ./.github/actions/upload-test-results
if: always()
- name: Upload emulator.wtf inputs
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
test-name: Instrumentation Test app
name: emulator-wtf-inputs
retention-days: 1
if-no-files-found: error
path: |
app/build/outputs/apk/full/debug/app-full-debug.apk
app/build/outputs/apk/minimal/debug/app-minimal-debug.apk
app/build/outputs/apk/androidTest/full/debug/app-full-debug-androidTest.apk
app/build/outputs/apk/androidTest/minimal/debug/app-minimal-debug-androidTest.apk
devices.txt

instrumentation_test:
name: "Instrumentation Tests"
Expand Down Expand Up @@ -451,28 +433,3 @@ jobs:
if: always()
with:
test-name: Instrumentation test (API${{ matrix.configuration.api-level }}) ${{ matrix.configuration.target }}

publish_test_results:
name: "Publish Tests Results"
needs: [instrumentation_test, unit_tests, screenshot_test, instrumentation_test_app]
runs-on: ubuntu-latest
permissions:
checks: write
pull-requests: write
if: always()

steps:
- name: Download Artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1

- name: Publish Test Results EnricoMi
uses: EnricoMi/publish-unit-test-result-action@c950f6fb443cb5af20a377fd0dfaa78838901040 # v2.23.0
if: always()
with:
comment_mode: "failures"
# We want to fail the build to block the merge if any test fail it allows us simplify the setup of the protected branch.
action_fail: true
files: |
**/androidTest-results/**/TEST-*.xml
**/build/test-results/**/TEST-*.xml
**/build/test-results/**/results.xml