[iOS Screenshots CI] Fix screenshots generation GitHub Action #15822

jaclync · 2025-06-26T13:46:49Z

Why

The iOS screenshots generation CI job via the generate-screenshots GitHub label in a PR has been broken for a while since it was added in 2021. The team has been relying on manually generating the raw and promo screenshots and upload them to the App Store. This PR fixes the CI job which is a GitHub Action that public repos have unlimited access for free. We might still want to migrate to Buildkite for better performance in the future, but since this is a job required rather infrequently and without cost, the slow time (3h6m) and 403MB for just the promo screenshots for 17 locales are fine for now.

How

Replace S3 upload/download with GHA artifacts
Remove custom font download and use Helvetica Neue instead, since Proxima Nova doesn't seem to be available in CI macOS image even though it is supposed to be included in macOS Sonoma+

Known issue

When running the CI job with all locales, there were some flaky locales that didn't pass testScreenshots for the first time. I also noticed the last screenshot, showing a push notification, is missing the push notification for a subset of locales from the promo-screenshots artifact. For example, the it 5th screenshot in both tablet and phone does not have the notification. I hadn't seen this flakiness locally, and will look into this more separately in WOOMOB-722.

Test plan

Verify screenshot generation jobs upload artifacts correctly
Compare generated screenshots with the system font have little visual difference from App Store screenshots

Example screenshots

Feel free to download en-US screenshots here from GHA artifacts and compare them with the App Store screenshots.

locale	App Store (Proxima Nova)	CI generated (Helvetica Neue)
en-US
en-US
ar-SA

locale	App Store (Proxima Nova)	CI generated (Helvetica Neue)
es-ES
ja

I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

…ine upload rejected: `step` may only contain alphanumeric characters, slashes, dashes or underscores`.

…tifacts: GET ...: 400 Bad Request: Multiple artifacts were found for query: `screenshot-artifacts.tar`. Try scoping by the job ID or name.`.

… every CI run.

🚨 Missing path option in the cache plugin to restore 🚨 Error: running "plugin cache post-checkout" shell hook: The plugin cache post-checkout hook exited with status 1`.

…/Products/Debug-iphonesimulator/'`.

… SCREENSHOT_LANGUAGE: unbound variable`.

WooCommerceScreenshots-Runner (3506) encountered an error (Failed to load the test bundle. (Underlying Error: The bundle “WooCommerceScreenshots” couldn’t be loaded. The bundle couldn’t be loaded. Try reinstalling the bundle.`.

… S3_BUCKET: unbound variable` error.

…ials from GHA.

…e in macOS Sonoma and above.

… generate-screenshots` error.

… error (Failed to load the test bundle. (Underlying Error: The bundle “WooCommerceScreenshots” couldn’t be loaded` error.

…eenshots. Run `git lfs install && git lfs fetch && git lfs pull` to fix this.` error.

…drawText` tool.

jaclync · 2025-06-27T12:19:16Z

.github/workflows/screenshots.yml

-        name: screenshot-runner
-        path: fastlane/DerivedData/Build/Products/Debug-iphonesimulator/WooCommerceScreenshots-Runner.app
+        name: screenshot-build-products
+        path: fastlane/DerivedData/Build/Products/Debug-iphonesimulator/


🗒️ There are more files required for later steps than just the WooCommerce.app and WooCommerceScreenshots-Runner.app. Otherwise, errors like WooCommerceScreenshots-Runner (19200) encountered an error (Failed to load the test bundle. (Underlying Error: The bundle “WooCommerceScreenshots” couldn’t be loaded are thrown.

… available in CI macOS images.

…s look good for all locales.

wpmobilebot · 2025-06-27T18:15:30Z

📲 You can test the changes from this Pull Request in WooCommerce iOS Prototype by scanning the QR code below to install the corresponding build.

	App Name	WooCommerce iOS Prototype
	Build Number	`pr15822-f02111c`
	Version	`22.8`
	Bundle ID	`com.automattic.alpha.woocommerce`
	Commit	`f02111c`
	Installation URL	1ighn86jhhmeg

Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

iangmaia · 2025-07-01T16:29:13Z

.github/workflows/screenshots.yml

-    runs-on: macos-latest
+    runs-on: macos-15

    steps:


It's not part of this PR, but an improvement idea:

we should be able to remove the Install Screenshot Gems step and use this env var at the job level instead (which will make the ruby/setup-ruby@v1 step to already use the screenshots group without a separate step).

env: BUNDLE_WITH: screenshots

Updated in 6f118d2, I hope this is what you meant. I triggered the screenshots generation workflow to verify that it works.

iangmaia · 2025-07-01T16:50:10Z

.github/workflows/screenshots.yml

-        name: screenshot-runner
-        path: fastlane/DerivedData/Build/Products/Debug-iphonesimulator/WooCommerceScreenshots-Runner.app
+        name: screenshot-build-products
+        path: fastlane/DerivedData/Build/Products/Debug-iphonesimulator/


Probably not worth it for the storage reclaim per se, but maybe worth it to make it clearer that these are discardable adding something like:

Suggested change

path: fastlane/DerivedData/Build/Products/Debug-iphonesimulator/

path: fastlane/DerivedData/Build/Products/Debug-iphonesimulator/

retention-days: 1

Good idea, updated in f02111c.

iangmaia

Congrats on getting it up and working again! 👍
Approving to unblock; just added a couple of minor comments.

And I agree it's best to at some point migrate this to Buildkite and integrate it with the rest of the release automation.

wpmobilebot · 2025-07-11T08:16:49Z

Version 22.8 has now entered code-freeze, so the milestone of this PR has been updated to 22.9.

…p and use an env var at the job level instead. https://github.com/woocommerce/woocommerce-ios/pull/15822/files#r2178045214

…e workflow.

jaclync · 2025-07-21T12:37:09Z

Merging this PR now, as the latest screenshots job passed with the promo screenshots generated. There is a flaky issue in the UI test itself with push notification, I will look into it separately in WOOMOB-722. I will also look into how we integrate promo screenshots upload to the App Store in WOOMOB-723.

jaclync added 18 commits June 26, 2025 09:48

[WIP] Try migrating screenshots github action to buildkite.

06476ef

[WIP] Only run screenshots commands in CI for testing.

2caeee9

[WIP] Try fixing `fatal: Failed to upload and process pipeline: Pipel…

9b2e082

…ine upload rejected: `step` may only contain alphanumeric characters, slashes, dashes or underscores`.

[WIP] Try fixing canceled take-screenshots tasks.

afd89c0

[WIP] Try fixing error `buildkite-agent: fatal: failed to download ar…

7816c79

…tifacts: GET ...: 400 Bad Request: Multiple artifacts were found for query: `screenshot-artifacts.tar`. Try scoping by the job ID or name.`.

[WIP] Try caching build-screenshots to avoid unnecessary 8m execution…

f719660

… every CI run.

[WIP] Try fixing download artifacts issue again.

4038f98

[WIP] Try fixing error `Running plugin cache post-checkout hook

444a6e1

🚨 Missing path option in the cache plugin to restore 🚨 Error: running "plugin cache post-checkout" shell hook: The plugin cache post-checkout hook exited with status 1`.

[WIP] Try fixing cache again.

a4f18d0

[WIP] Try fixing Uploading artifacts error.

d9674d6

[WIP] Another attempt.

dd81581

[WIP] Try fixing `tar: could not chdir to 'fastlane/DerivedData/Build…

ebc89b3

…/Products/Debug-iphonesimulator/'`.

[WIP] Try fixing `./.buildkite/commands/take-screenshots.sh: line 18:…

9f6eb2e

… SCREENSHOT_LANGUAGE: unbound variable`.

[WIP] Try fixing `Testing failed:

52aa932

WooCommerceScreenshots-Runner (3506) encountered an error (Failed to load the test bundle. (Underlying Error: The bundle “WooCommerceScreenshots” couldn’t be loaded. The bundle couldn’t be loaded. Try reinstalling the bundle.`.

[WIP] Try fixing `./.buildkite/commands/take-screenshots.sh: line 32:…

040dd35

… S3_BUCKET: unbound variable` error.

[WIP] Try fixing env variables not set again.

b1a0451

Use Buildkite artifacts instead of S3 since we don't know the credent…

83acffe

…ials from GHA.

Remove custom font download and installation as it should be availabl…

91cbef9

…e in macOS Sonoma and above.

jaclync force-pushed the feat/WOOMOB-687-fix-ios-screenshots-gen-ci branch from 9b6b3c5 to 91cbef9 Compare June 26, 2025 13:48

jaclync added the generate screenshots Triggers a GitHub Action to generate screenshots for this PR. label Jun 26, 2025

jaclync added 5 commits June 26, 2025 09:54

Remove S3 variables in pipeline.

3408a99

Try fixing GHA Xcode version error by running on macOS 15 image.

97d7b69

Fix `The following step dependencies were not found during the build:…

ad1db4f

… generate-screenshots` error.

GHA: try fixing `WooCommerceScreenshots-Runner (19200) encountered an…

d34e8b1

… error (Failed to load the test bundle. (Underlying Error: The bundle “WooCommerceScreenshots” couldn’t be loaded` error.

Buildkite: try fixing `LFS not enabled – unable to generate promo scr…

b733d22

…eenshots. Run `git lfs install && git lfs fetch && git lfs pull` to fix this.` error.

jaclync added generate screenshots Triggers a GitHub Action to generate screenshots for this PR. and removed generate screenshots Triggers a GitHub Action to generate screenshots for this PR. labels Jun 26, 2025

jaclync added 3 commits June 26, 2025 10:50

Buildkite: temporarily remove PR label requirement.

2d980bc

Buildkite: try fixing remaining LFS issue.

79a93a2

Buildkite: try fixing errors in process-screenshots.

3e30316

jaclync changed the title ~~[iOS Screenshots CI] Replace S3 storage with Buildkite artifacts for screenshot workflow~~ [iOS Screenshots CI] Fix screenshots generation GitHub Action Jun 26, 2025

jaclync marked this pull request as draft June 26, 2025 21:04

jaclync mentioned this pull request Jun 27, 2025

[POS as a tab i1] Fix issues from raw & promo screenshots generation #15805

Merged

1 task

Base automatically changed from feat/WOOMOB-673-generate-screenshots-for-pos-tab to trunk June 27, 2025 12:13

Just specify one font name since the fallback did not work with the `…

dfa74a8

…drawText` tool.

jaclync commented Jun 27, 2025

View reviewed changes

jaclync added 4 commits June 27, 2025 08:21

Remove Proxima Nova font now that we are using Helvetica Neue that is…

580a40f

… available in CI macOS images.

Re-enable all locales to measure the time and ensure promo screenshot…

f90287c

…s look good for all locales.

Try fixing ar issue by matching IOS_LOCALES in Fastfile.

602c99c

Merge branch 'trunk' into feat/WOOMOB-687-fix-ios-screenshots-gen-ci

01278a7

jaclync removed the generate screenshots Triggers a GitHub Action to generate screenshots for this PR. label Jun 27, 2025

Uncomment tasks pipeline steps.

172b470

jaclync marked this pull request as ready for review June 27, 2025 20:06

jaclync requested a review from a team June 27, 2025 20:06

jaclync added this to the 22.8 milestone Jun 27, 2025

jaclync added the category: screenshots Related to generating screenshots for the App Store. label Jun 27, 2025

iangmaia reviewed Jul 1, 2025

View reviewed changes

iangmaia approved these changes Jul 1, 2025

View reviewed changes

wpmobilebot modified the milestones: 22.8, 22.9 Jul 11, 2025

jaclync added 3 commits July 18, 2025 16:03

Merge branch 'trunk' into feat/WOOMOB-687-fix-ios-screenshots-gen-ci

5112f63

Try addressing suggestion to remove the Install Screenshot Gems ste…

6f118d2

…p and use an env var at the job level instead. https://github.com/woocommerce/woocommerce-ios/pull/15822/files#r2178045214

Make build files discardable after 1 day as they are only used for th…

f02111c

…e workflow.

jaclync added the generate screenshots Triggers a GitHub Action to generate screenshots for this PR. label Jul 18, 2025

jaclync merged commit 57650fa into trunk Jul 21, 2025
50 checks passed

jaclync deleted the feat/WOOMOB-687-fix-ios-screenshots-gen-ci branch July 21, 2025 12:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[iOS Screenshots CI] Fix screenshots generation GitHub Action #15822

[iOS Screenshots CI] Fix screenshots generation GitHub Action #15822

jaclync commented Jun 26, 2025 •

edited

Loading

Uh oh!

jaclync Jun 27, 2025

Uh oh!

wpmobilebot commented Jun 27, 2025 •

edited

Loading

Uh oh!

iangmaia Jul 1, 2025

Uh oh!

jaclync Jul 18, 2025

Uh oh!

iangmaia Jul 1, 2025

Uh oh!

jaclync Jul 18, 2025

Uh oh!

iangmaia left a comment

Uh oh!

wpmobilebot commented Jul 11, 2025

Uh oh!

jaclync commented Jul 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	path: fastlane/DerivedData/Build/Products/Debug-iphonesimulator/
	path: fastlane/DerivedData/Build/Products/Debug-iphonesimulator/
	retention-days: 1

[iOS Screenshots CI] Fix screenshots generation GitHub Action #15822

[iOS Screenshots CI] Fix screenshots generation GitHub Action #15822

Conversation

jaclync commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

How

Known issue

Test plan

Example screenshots

Uh oh!

jaclync Jun 27, 2025

Choose a reason for hiding this comment

Uh oh!

wpmobilebot commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iangmaia Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

jaclync Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

iangmaia Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

jaclync Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

iangmaia left a comment

Choose a reason for hiding this comment

Uh oh!

wpmobilebot commented Jul 11, 2025

Uh oh!

jaclync commented Jul 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jaclync commented Jun 26, 2025 •

edited

Loading

wpmobilebot commented Jun 27, 2025 •

edited

Loading