Skip to content

QVAC-18790 infra: scope e2e bootstrap to deps required by filtered tests#1991

Merged
Victor-Rodzko merged 4 commits into
mainfrom
feat/producer-driven-bootstrap-deps
May 13, 2026
Merged

QVAC-18790 infra: scope e2e bootstrap to deps required by filtered tests#1991
Victor-Rodzko merged 4 commits into
mainfrom
feat/producer-driven-bootstrap-deps

Conversation

@Victor-Rodzko
Copy link
Copy Markdown
Contributor

@Victor-Rodzko Victor-Rodzko commented May 12, 2026

🎯 What problem does this PR solve?

  • SDK e2e consumers (packages/sdk/tests-qvac) call ResourceManager.downloadAllOnce on bootstrap and warm every registered model (~24 desktop / ~22 mobile) regardless of which tests will actually run.
  • On Device Farm this is the dominant cost of each CI job — Android bootstrap regularly hits ~20–30 min, and one iOS run failed with ❌ No consumers connected within 1200s timeout while the device was still pre-loading models.
  • A typical filtered run only needs a subset of those deps (e.g. --filter registry needs 0, --filter translation needs 8).

See QVAC-18790 for the full problem statement and CI evidence.

📝 How does it solve it?

  • Companion framework change in tetherto/qvac-test-suite#70 delivers the producer's post-filter TestDefinition[] to the consumer's bootstrap callback over MQTT (register-ack.filteredTestIds).
  • ResourceManager.downloadAllOnce gains an optional allowedDeps: ReadonlySet<string> — when provided, downloads + pre-loads are scoped to that subset; when omitted, the legacy "warm everything" path is preserved (used by the CI cache-warmer step and any older framework).
  • New shared/collect-test-deps.ts helper extracts metadata.dependency / metadata.dependencies from the filtered tests (drops the "none" sentinel, dedupes).
  • Both desktop/consumer.ts and mobile/consumer.ts bootstrap callbacks now accept filteredTests?: TestDefinition[], derive allowedDeps via collectTestDeps, and pass it through. undefined keeps today's behaviour, so older framework versions and the unfiltered CI cache-warmer continue to populate the full cache.

🧪 How was it tested?

Local

  • Desktop, npx qvac-test run:local:desktop --filter transcription,parakeet🎯 Bootstrap dep-filter active: keeping 4 dep(s); 20 of 24 defined excluded, bootstrap 9.97 s (was minutes).
  • iOS on a real iPhone, npx qvac-test run:local:ios --filter transcription,parakeetkeeping 4 dep(s); 18 of 22 defined excluded, bootstrap 20.2 s (was ~25 min). Full batch 62.86 s, 13 passed / 18 skipped (known QVAC-18460 iOS skips) / 0 failed.
  • No-filter path (older framework / cache-warmer) verified by typecheck + bootstrap() called without args: allowedDeps stays undefined, manager warms everything as before.

CI

Two full QVAC Tests (sdk) runs against this PR (using the matching preview build of tetherto/qvac-test-suite#70), both fully green:

Job (mac-mini-m4-gpu) Run 1 --filter translation (8 deps) Run 2 --filter registry (0 deps)
Filter marker keeping 8 dep(s); 16 of 24 defined excluded keeping 0 dep(s); 24 of 24 defined excluded
Consumer-side Bootstrap completed in 4 098 ms 1 ms
Test results 52 / 52 pass 11 / 11 pass

Mobile, same two runs:

  • Android run-producer: 22m12s → 2m31s (test set 52 → 11). 0/52 → 1/52 completed transition shows bootstrap ~18.5 min for 8 deps on S25 Ultra (down from ~30 min full-registry warm seen in run 25672716968). For the 0-dep run the device starts the first test ~10 s after register.
  • iOS run-producer: 6m40s → 3m04s, no No consumers connected timeouts. 49 pass / 3 skip vs 11 / 0.
  • CI runs: 25720111911 and 25722580067.

📌 Follow-ups before merge

  • @tetherto/qvac-test-suite is currently pinned to 0.6.3-tmp.pr-70.runid-25717564206 (preview build of tetherto/qvac-test-suite#70). Once that PR lands and 0.6.3 is published, this dep moves to ^0.6.3.

@Victor-Rodzko Victor-Rodzko requested review from a team as code owners May 12, 2026 06:52
@Victor-Rodzko Victor-Rodzko marked this pull request as draft May 12, 2026 06:54
Comment thread packages/sdk/tests-qvac/package.json Outdated
@Victor-Rodzko Victor-Rodzko marked this pull request as ready for review May 12, 2026 09:08
@Victor-Rodzko Victor-Rodzko changed the title infra[notask]: scope e2e bootstrap to deps required by filtered tests QVAC-18790 infra: scope e2e bootstrap to deps required by filtered tests May 12, 2026
@Victor-Rodzko Victor-Rodzko force-pushed the feat/producer-driven-bootstrap-deps branch from c891ea9 to 1d8cb13 Compare May 12, 2026 14:59
NamelsKing
NamelsKing previously approved these changes May 12, 2026
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 12, 2026

Tier-based Approval Status

**PR Tier:** TIER1

**Current Status:** ✅ APPROVED

**Requirements:**
- 1 Team Member approval ✅ (1/1)
- 1 Team Lead OR Management approval ✅ (1/1)



---
*This comment is automatically updated when reviews change.*

lauripiisang
lauripiisang previously approved these changes May 12, 2026
opaninakuffo
opaninakuffo previously approved these changes May 12, 2026
Victor-Rodzko and others added 3 commits May 13, 2026 09:21
Wire up the new producer-driven bootstrap contract from
@tetherto/qvac-test-suite 0.6.3 so the SDK e2e harness stops
pre-loading every registered model when a --filter / --suite is
in play.

- ResourceManager.downloadAllOnce now accepts an optional
  `allowedDeps: ReadonlySet<string>` to scope downloads + pre-loads
  to a subset of registered models; without it the manager keeps
  the legacy "warm everything" behavior.
- New `collect-test-deps` helper extracts the
  metadata.dependency / metadata.dependencies keys from a
  TestDefinition[] and dedupes them.
- desktop/mobile bootstrap callbacks now take the
  `filteredTests?: TestDefinition[]` argument the framework supplies
  via register-ack, fed through collectTestDeps into downloadAllOnce.

Locally cuts iOS bootstrap from ~25min to ~20s and desktop from
~10min (warm cache) to ~10s for `--filter transcription,parakeet`.

Bumps @tetherto/qvac-test-suite to a temporary preview build of
qvac-test-suite#70 (0.6.3-tmp.pr-70.runid-...); will be moved to
^0.6.3 once that PR lands and a real release is cut.

Co-authored-by: Cursor <cursoragent@cursor.com>
@Victor-Rodzko Victor-Rodzko force-pushed the feat/producer-driven-bootstrap-deps branch from 1d8cb13 to b9d4a0c Compare May 13, 2026 06:21
@Victor-Rodzko
Copy link
Copy Markdown
Contributor Author

/review

@Victor-Rodzko Victor-Rodzko merged commit 4cea285 into main May 13, 2026
25 checks passed
@Victor-Rodzko Victor-Rodzko deleted the feat/producer-driven-bootstrap-deps branch May 13, 2026 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants