ci: consolidate e2e to one job per variant#395
Open
douglance wants to merge 6 commits into
Open
Conversation
Each of the ~54 spec-x-variant jobs paid ~190s of identical fixed cost (setup steps, app start, synpress chain seeding) around 44s-10min of actual test. One job per variant pays it 5x instead of 54x, cutting e2e compute ~43% (~360 to ~205 runner-min) and eliminating runner-pool contention. Specs run sequentially per variant, smoke (login) first. Also fixes the clean-up job's cache key (grepped for a -run_attempt suffix the build never writes, so the delete was a no-op).
The build cache key is scoped to run_id, so the standalone run built into a cache no other workflow could read - 2 wasted minutes per PR push. test.yml already builds via workflow_call within its own run.
Fails in CI regardless of timeout (token-list search never resolves) and forces manual rerun cycles on every PR. Skip until fixed.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
There was a problem hiding this comment.
Pull request overview
Optimizes the repository’s CI E2E pipeline by reducing duplicated setup work and fixing build-cache lifecycle issues, trading parallelism for significantly lower runner compute and less queue contention.
Changes:
- Consolidates E2E CI execution to one job per testnode variant (sequential specs per variant) and updates job/artifact naming accordingly.
- Disables the known-flaky
importTokenE2E suite to unblock CI while the underlying token-list search issue is unresolved. - Removes the standalone
build.ymlpull_request trigger and corrects build-cache deletion logic to target the actual cache key.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| packages/arb-token-bridge-ui/tests/e2e/specs/importToken.cy.ts | Skips the import-token suite in CI to avoid a known failure mode. |
| .github/workflows/formatSpecfiles.js | Generates a reduced E2E matrix (one entry per variant) instead of per-spec fanout. |
| .github/workflows/e2e-tests.yml | Runs full spec set per variant, adjusts naming, and fixes cache deletion targeting. |
| .github/workflows/build.yml | Removes redundant PR-triggered runs and documents why workflow_call-only is required. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
17
to
24
| const testType = process.argv[2]; | ||
| switch (testType) { | ||
| case 'regular': { | ||
| specFiles.forEach((spec) => { | ||
| tests.push({ | ||
| ...spec, | ||
| type: 'regular', | ||
| typeName: '', | ||
| tag: tagMap['regular'], | ||
| }); | ||
| tests.push({ | ||
| ...spec, | ||
| type: 'orbit-eth', | ||
| typeName: 'with L3 (ETH)', | ||
| tag: tagMap['orbit-eth'], | ||
| }); | ||
| tests.push({ | ||
| ...spec, | ||
| type: 'orbit-custom-6dec', | ||
| typeName: 'with L3 (6 decimals custom)', | ||
| tag: tagMap['orbit-custom-6dec'], | ||
| }); | ||
| tests.push({ | ||
| ...spec, | ||
| type: 'orbit-custom-18dec', | ||
| typeName: 'with L3 (18 decimals custom)', | ||
| tag: tagMap['orbit-custom-18dec'], | ||
| }); | ||
| tests.push({ | ||
| ...spec, | ||
| type: 'orbit-custom-20dec', | ||
| typeName: 'with L3 (20 decimals custom)', | ||
| tag: tagMap['orbit-custom-20dec'], | ||
| }); | ||
| variants.forEach((variant) => { | ||
| tests.push({ ...variant, recordVideo: false }); | ||
| }); | ||
| break; | ||
| } |
Comment on lines
235
to
239
| run: | | ||
| if gh actions-cache list | grep build-artifacts-${{ github.run_id }}-${{ github.run_attempt }} | ||
| if gh actions-cache list | grep build-artifacts-${{ github.run_id }} | ||
| then | ||
| gh actions-cache delete build-artifacts-${{ github.run_id }}-${{ github.run_attempt }} --confirm | ||
| gh actions-cache delete build-artifacts-${{ github.run_id }} --confirm | ||
| fi |
approveToken asserts the one-time approval dialog appears for a MAX deposit, which requires the user wallet's allowance to still be the 1-token setup amount. With specs sharing one chain per variant job, a later position let earlier specs raise the allowance and the dialog never showed (and its mid-flow failure left a dangling MetaMask window that broke changeMetamaskNetwork in the next two specs).
Reruns share the run_id and do not re-run the succeeded build job, so deleting the cache in clean-up after a red run made every rerun fail at restore. Delete only on green runs.
Healthy jobs finish in ~35m; a testnode that boots with a stalled validator (a v0.2.2 image flake, ~1 in 5 boots) makes withdrawal specs burn their full timeouts and the job run 90m+. Fail fast and rerun.
dewanshparashar
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #357, optimizing CI for value-per-second based on measured job data from the last 6 Test runs (E2E = ~360 runner-min/run, 96% of all compute; ~190s of every job was duplicated fixed cost around 44s–10min of actual test).
-run_attemptsuffix that's never written — it had never deleted anything) and only delete on green runs, since reruns share the run_id and need the cache.Measured result (run 29045845494, fully green)
Known remaining flake: the v0.2.2 image's stalled-validator boot (~1 in 5). Real fix is bumping to ≥v0.2.9 (validator fix upstream), blocked on the custom-fee token-bridge export bug and the build-time-baked network config — tracked for the composite-action follow-up branch.
Test plan
formatSpecfiles.js regularemits 5 variant entries; YAML validatesTest E2E Successgreengh run rerun --failedafter a red run restores the build from cache (the pre-fix failure mode was observed on run 29031121473 attempt 2; the guard prevents the cache deletion that caused it, but no red run has occurred since to exercise it end-to-end)