Skip to content

feat(test): local Android Game Loop harness with golden screenshots#2445

Open
leanmendoza wants to merge 3 commits into
mainfrom
feat/add-in-device-test
Open

feat(test): local Android Game Loop harness with golden screenshots#2445
leanmendoza wants to merge 3 commits into
mainfrom
feat/add-in-device-test

Conversation

@leanmendoza

Copy link
Copy Markdown
Collaborator

What

Adds an on-device E2E test harness driven by the Firebase Test Lab Android Game Loop contract (com.google.intent.action.TEST_LOOP + integer scenario extra), runnable locally over adb with a single command — no Test Lab / Blaze required:

cargo run -- game-loop --scenario all           # run every scenario, compare vs goldens
cargo run -- game-loop --scenario 3 --bless      # capture this run's shots as new goldens
cargo run -- game-loop --scenario 5 --guest-seed 42 --install --clear

The device stays fixed, so golden screenshots are pixel-stable run to run (the whole reason to use a real device instead of Test Lab's rotating farm for regression goldens).

How it's structured

xtask driver (src/game_loop/) — reproduces exactly what Test Lab does over adb: install + pm clear, fire the TEST_LOOP intent, tail logcat for the RESULT line, pull user://gameloop_*.png, compare vs tests/snapshots/gameloop/ (or --bless). Split by responsibility — options / adb / launch / screenshots / golden / scenario, each < 200 lines. --scenario all fans out over the registry.

GDScript runner (godot/src/test/game_loop/) — an autoload detects the Game Loop launch, runs the requested scenario, prints a machine-parseable RESULT to logcat, and finishes via the native gameLoopFinish. Typed result classes (GameLoopResult / CaptureResult), a GameLoopScenario base, and a shared LoginFlowScenario.

# Scenario
1 guest login
2 screenshot smoke
3 login flow (COMEBACK)
4 fresh-guest onboarding — random seed → AVATAR_CREATE / NAMING / DISCOVER_FTUE
5 in-world load of aesironline.dcl.eth with async_wait_until_scene_loaded(30s)

Deterministic captures (capture_mode.gd) — a momentary global freeze: scene_runner pause + avatars to canonical idle + time_scale=0 + canonicalized AnimationPlayer phase. Optional live-world hide so the non-deterministic 3D scene / SDK UI never draws, and manual per-golden binary masks (<golden>_mask.png) to exclude dynamic regions (place cards, random avatar/name) from the compare.

Golden mode — a per-run guest-seed forces a deterministic guest via a seed-derived anchor through the standard Play-as-Guest flow, reachable only through this harness (no deeplink/UI path, so a production install can't be driven into a forced identity).

Comparator fix

compare_images_similarity normalized per-pixel distance by sqrt(255*3) (~18× too small), which produced negative similarities. Fixed to 255*sqrt(channels) + clamp to [0,1], added compare_images_similarity_masked, and re-mapped caller thresholds via an exact monotonic transform so existing snapshot tests keep their strictness.

Verified

Developed and verified end-to-end on a physical device (Samsung A54). Scenarios 1–5 run, pull shots, and compare; scenario 3 goldens committed.

Follow-ups (not in this PR)

  • Author _mask.png files for scenario 4's dynamic screens (random avatar/name) and bless its goldens.
  • Decide scenario 5's in-world golden: it runs ~98.7% run-to-run (avatar/spawn/lighting), so it needs either an avatar _mask.png or --threshold 0.98 — not the default 0.995.

🤖 Generated with Claude Code

Adds an on-device E2E test harness driven by the Firebase Test Lab
Android Game Loop contract (com.google.intent.action.TEST_LOOP + integer
scenario extra), runnable locally over adb with `cargo run -- game-loop`
against a USB-connected device — no Test Lab / Blaze needed.

xtask driver (src/game_loop/): install+pm clear, fire the TEST_LOOP
intent, tail logcat for the RESULT line, pull user://gameloop_*.png, and
compare vs goldens (tests/snapshots/gameloop/) or --bless them. Split by
responsibility: options/adb/launch/screenshots/golden/scenario, each
<200 lines. `--scenario all` fans out over the registry.

GDScript runner (godot/src/test/game_loop/): autoload detects the
Game Loop launch, runs the requested scenario, prints a machine-parseable
RESULT to logcat, finishes via native gameLoopFinish. Typed result
classes (GameLoopResult/CaptureResult), a GameLoopScenario base, and a
shared LoginFlowScenario. Scenarios: 1 guest login, 2 screenshot,
3 login flow, 4 fresh-guest onboarding (random seed -> AVATAR_CREATE/
NAMING/DISCOVER_FTUE), 5 in-world load of aesironline.dcl.eth with
async_wait_until_scene_loaded(30s).

Deterministic captures (capture_mode.gd): momentary global freeze
(scene_runner pause + avatars to canonical idle + time_scale=0 +
canonicalized AnimationPlayer phase), optional live-world hide so the
non-deterministic 3D scene/UI never draws, and manual per-golden binary
masks (<golden>_mask.png) to exclude dynamic regions from the compare.

Golden mode: a per-run guest-seed forces a deterministic guest via a
seed-derived anchor through the STANDARD Play-as-Guest flow, reachable
ONLY through this harness (no deeplink/UI path).

Fix image_comparison: normalize per-pixel distance by 255*sqrt(channels)
(was sqrt(255*3), ~18x too small, produced negative similarities) and
clamp to [0,1]; add compare_images_similarity_masked; re-map caller
thresholds via an exact monotonic transform to preserve test strictness.
@leanmendoza
leanmendoza requested a review from a team as a code owner July 6, 2026 14:37
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

📦 Build Report

🤖 Android

Artifact Status
APK 📱 Download APK
AAB main/release builds only
Debug Symbols main/release builds only

Build Status: ✅ Success

🍏 iOS

The signed APK is uploaded to R2 on every build (link above). Add the build label (alias: build-ios) to build & ship iOS to TestFlight and post a Slack "Android build ready" notification.


🔗 Workflow Run: View logs

🔄 Updated: 2026-07-15 13:44:11 UTC

@kuruk-mm
kuruk-mm self-requested a review July 14, 2026 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants