test(tools-pack): give packaging suites a realistic timeout - #6215
Open
PerishCode wants to merge 1 commit into
Open
test(tools-pack): give packaging suites a realistic timeout#6215PerishCode wants to merge 1 commit into
PerishCode wants to merge 1 commit into
Conversation
tools/pack had no vitest config, so its suites ran on vitest's 5s default. That package shells out to real packaging tools — 7z archive and extract, pnpm pack, installer assembly — against temp directories, so 5s is close enough to their actual cost that runner speed alone decides pass/fail. The Windows launcher payload archive test failed merge-queue CI at 5007ms while passing the same tree on the pull-request run and locally. Add a package vitest config with testTimeout 20_000, matching apps/daemon, which is subprocess-heavy for the same reason. The include pattern matches the 36 existing suites exactly; test counts are unchanged.
mrcfps
approved these changes
Jul 29, 2026
mrcfps
left a comment
Contributor
There was a problem hiding this comment.
@PerishCode I reviewed the new tools/pack/vitest.config.ts against the package test inventory and the live CI invocation. The config preserves Node execution, includes all 36 existing .test.ts suites, and raises only the per-test ceiling to 20 seconds. On this head, the full tools-pack suite passed with 248 tests passing and 8 skipped, and the Windows launcher-payload lane passed with 5 tests passing and 1 skipped. Thanks for documenting the original flake evidence so carefully and keeping the fix focused—nice work. 🙌
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
I hit this landing an unrelated docs PR (#6207): its merge-queue run failed and
the PR was dequeued, on a tree whose only change was one markdown file and two
documentation lines. The failure was
tools/pack/tests/launcher-payload.test.ts→ "creates a Windows payload 7z with bootstrap-readable contents", with
Test timed out in 5000msat 5007ms. The same job passed on that PR's ownrun, on the same tree.
The pain is not that one test is slow — measurement says the opposite. On this
branch's own CI run the whole file completed in 657ms for all 6 tests. In
the failing merge-queue run the non-7z tests in that file summed to ~720ms
while the 7z test alone hit 5007ms. So the test normally costs well under a
second and occasionally stalls past 5s on a Windows runner: roughly an 8x
outlier, not a marginal overshoot.
tools/packhas no vitest config at all, so the whole package runs on vitest's5s default — a value nobody chose for it. That package shells out to real
packaging tools (7z archive and extract,
pnpm pack, installer assembly)against temp directories, which is exactly the workload where an occasional
subprocess or filesystem stall is expected. With no headroom, such a stall
fails a merge-queue gate and dequeues an unrelated PR.
This change raises the ceiling so a stall stops being a merge-gate failure. It
deliberately does not diagnose the stall — see Adjacent issues.
Recent history for that job: 10 success, 1 cancelled, 1 failure (this one).
Rare, but it lands on a merge-queue gate, so it costs a full re-queue cycle.
What users will see
No user-facing or behavioral change. Contributors touching
tools/packget amerge-queue gate that is not decided by Windows runner speed.
Surface area
Screenshots
Not applicable.
Bug fix verification
A red spec is not available for this one: the failing test is
it.skipIf(process.platform !== "win32"), so it cannot execute on macOS orLinux, and the symptom is a timing boundary on a Windows CI runner rather than
a falsifiable behavior. Reproducing it deterministically would mean pinning
runner speed, which the harness cannot do.
Verification done instead:
5007msagainst a5000mslimit, no assertion output.tools/packhas no vitest config, so the 5s value was vitest'sdefault rather than a deliberate budget, and that
apps/daemon(20_000) andapps/desktop(10_000) already raise it for the same reason.includecould silently stoprunning suites outside the pattern. All 36 tools/pack suites live in
tests/and are.test.ts; zero files sit outside it.Adjacent issues
Not addressed here, deliberately:
same tree points at the environment — 7z subprocess startup contention,
antivirus scanning the temp archive, or cold filesystem — not at the test
logic. Raising the timeout stops it from failing the gate; it does not
explain it. Worth revisiting if the job's failure rate rises.
use a smaller fixture, is a test-shape question with its own tradeoffs.
docs/testing/test-efficiency.zh-CN.md, duration budgets should follow arepresentative baseline rather than lead it. This PR removes a false failure
signal; it does not introduce a duration guard.
Validation
Baseline vs. change, both on this branch:
pnpm --filter @open-design/tools-pack testwithout the config:36 files, 248 passed, 8 skipped (256)
pnpm --filter @open-design/tools-pack testwith the config:36 files, 248 passed, 8 skipped (256) — identical, so the include pattern
drops nothing
pnpm guard— passedpnpm typecheck— passedpnpm --filter @open-design/tools-pack typecheck— passedgit diff --check— cleanMeasured on this branch's CI run (Windows job):
launcher-payload.test.tscompleted in 657ms for all 6 tests — comfortably inside the old 5s limit too.
This run therefore does not by itself demonstrate the fix working; it
establishes the test's normal cost, which is what makes the 5007ms failure
identifiable as an outlier rather than a marginal overshoot.
Note: the affected test is Windows-only and skips on this machine, so the
timeout headroom itself can only be exercised by the Windows CI job.