fix: stop stamping recovered iOS captures truncated; confirm Android alert dismissal - #2315
Conversation
…alert dismissal Two CI flake families on main and PRs since 2026-09-03. iOS Smoke, `is absent ... capture was truncated` (7 of 13 failures): the runner's stampedSnapshotPayload set `truncated: true` on every non-healthy capture, so a complete private-AX tree taken while the XCTest channel was penalized as slow (the normal state on a loaded CI host) was reported as truncated. Nothing consumed that until the strict absence assertion (#2245) refused truncated captures. `truncated` now tracks completeness only: payload truncation, a depth-limited capture, or a sparse terminal payload. The E2E conformance helper asserted the old conflation and now asserts `truncated === false`; a runner unit test pins the new contract and joins the targeted list in ios.yml. Android Smoke, `get text id="automation-alert-result"` selector miss (5 of 5 failures): #2260 replaced a polling wait with a one-shot read right after `alert dismiss`, and Android's `alert accept|dismiss` returned as soon as the button was pressed, while the dialog window was still the only thing in the accessibility tree. They now poll until the same dialog is gone (a different alert taking its place counts as dismissed), bounded by the existing action budget, else fail with "did not dismiss the visible alert" like the iOS runner already does.
Size Report
Startup median (7 runs, lower is better):
|
…lert action The scripted Android alert scenarios served the same dialog to every capture, which encoded the old return-after-press behavior; alert accept/dismiss now confirm the dialog is gone, so a dialog that never leaves is the failure it should be (covered by a new scenario). The fixtures now hide the dialog once its button is tapped or Back is sent, the way the ANR recovery scenario already did.
|
Independent Terra high sentinel review of aa8f5a7: No confirmed production defect in the iOS completeness change or Android post-dismiss polling route. The reported local simulator/emulator evidence is consistent with the traced behavior. Blocked on deterministic provider-integration fixture drift: job 101372400747 fails 4 Android alert cases because each static Remaining CI is still pending; this head is not merge-ready. |
thymikee
added
the
ready-for-human
|
Superseding my prior sentinel note for updated head d4a4f83: Clean. The provider scenarios now reuse the existing stateful world callbacks to remove a dialog after tap/Back, explicitly restore it before the sequential second action, and add the persistent-dialog negative case. That addresses the deterministic 4-case provider-integration failure without changing production behavior. I found no confirmed production defect in the iOS completeness or Android dismissal-polling routes. The reported local simulator/emulator evidence is consistent with the unchanged production diff. Current-head CI remains pending, so this is not yet merge-ready. |
…e for any failed step The Android smoke still missed `id="automation-alert-result"` on CI right after a confirmed dismissal: the daemon opened a fresh helper session for that read and its 2s capture had no such node, while the same one-shot read passes locally in 150ms. The fixture's re-render after the button callback is app timing, so the scenario waits for the outcome text (the polling landmark #2260 removed) and then pins it to the canary element. The harness kept only a screenshot, and only for wait timeouts, so the tree that produced a selector miss was never in the artifacts. Every unexpected step failure now writes failed-step-N.png and failed-step-N-snapshot.json next to failed-step.txt.
…out of android-lifecycle The test-file size ratchet rejects growth in android-lifecycle.test.ts (1,597 lines at the merge-base), and the dialog re-check work added a scenario there. The alert scenarios now live in android-alert.test.ts and the scripted dialog surfaces they share with the ANR scenarios in android-dialog-fixtures.ts; the lifecycle file drops to 1,260 lines.
thymikee
removed
the
ready-for-human
|
Sentinel delta review at b783ab3: clean coordinator + independent Terra/high review. Alert integration cases/fixtures remain intact after relocation; the E2E waits for the app callback result then pins it to the specific result element. Failure artifacts retain the original failed-step evidence and add best-effort screenshot/snapshot capture. Previous production review and reported local simulator/emulator proof remain applicable. Current Android, Coverage, Integration and core gates pass; iOS smoke is still pending, with no current failing checks or conflicts. Ready for human review, not yet confirmed merge-ready. Supersedes the old-head label/verdict. |
thymikee
added
the
ready-for-human
|
1 participant
Summary
Two CI flake families hit main and several PRs from 2026-09-03 to 2026-09-05. Both are product defects that a test change from that week exposed, not environment noise.
is absent id="automation-close-sheet"→capture was truncatedget text id="automation-alert-result"→Selector did not matchiOS:
truncatedmeant provenance, not completenessstampedSnapshotPayload(RunnerTests+SnapshotCapturePlan.swift) settruncated: trueon every non-healthy capture since #783. On a loaded macOS CI host the fixture app's XCTest tree/query captures time out or exceed the 3s slow threshold, the runner penalizes that channel for 120s, and every capture becomesSNAPSHOT_RECOVERED backend=private-ax reason=deferred— a whole tree reported as truncated. Nothing consumed the flag until the strict absence assertion (#2245, merged 09-03) refused truncated captures, which is when the flake started. The E2E conformance helper even asserted the conflation (truncated === (state !== 'healthy')).Fix:
truncated = payload.truncated || state == "sparse" || effectiveDepth != nil. New runner unit testtestStampedPayloadTruncationTracksCompletenessNotRecoveryProvenance(added to the ios.yml targeted list); the conformance helper now assertstruncated === false.Android:
alert dismissreturned before the dialog left#2260 swapped a polling
wait textfor a one-shotget textright afteralert dismiss. Android'shandleAndroidAlertActionpressed the button and returned immediately, while the iOS runner already re-checksalertStillVisible. In the failing CI artifact the dismiss step took 4.9s and the following read 2.3s and still saw only the dialog window.Fix:
alert accept|dismisspoll until the same dialog (title + buttons) is gone, bounded by the existingALERT_ACTION_RETRY_MSbudget, else fail withalert <action> did not dismiss the visible alert. A different alert taking its place counts as dismissed.Validation
is absentwith the private-AX backend pinned through the daemon flag the conformance transport uses reproduced the CI error before the fix (observation: "truncated", qualityrecovered/deferred,truncated: true) and passes after (truncated: false, quality unchanged). Runner unit tests (new + 4 neighbors) pass; the plain tree path passes before and after.smoke:automation-systempasses on a private emulator with the rebuilt dist;alert dismissnow 2.6s, the following read 0.8s. platform-android suite 656 tests green; the new tests are revert-sensitive (read count and the budget failure).Not addressed here
The remaining iOS Smoke failures (4–5 of 13) are
waittimeouts on an overloaded host:xcrun simctl terminatetiming out at 2s, tree captures over the 8s slice, the Settings replay failing attempt 1 with a capture stall even in green runs, and a hot runner reused from the replay step becoming unresponsive so a 15–20s restart consumes the wait's 10s budget. Whether a mid-wait runner restart should get its own budget, or the fixture E2E should stop reusing the replay step's runner, is a separate decision.