fix(ci): bound apt so a stalled mirror cannot hang the snapshot refresh

[CybotTM]

Closes #99, this time with the cause rather than a bound on the symptom.

The cause

apt-get. Log of the hung run 32269308335, last two lines before twenty minutes of silence:

Installing dependencies...
Get:1 file:/etc/apt/apt-mirrors.txt Mirrorlist [144 B]

playwright install --with-deps shells out to apt-get update for the browser system libraries, and it stalled on the first mirror fetch. The identical signature stalled a completely unrelated step in another repository the same afternoon — sudo apt-get update -qq … && sudo apt-get install -y -qq shellcheck in the shared validate workflow, 37 minutes, different repo, different tool.

apt applies no timeout to package acquisition by default, so a mirror that accepts the connection and then stops responding blocks forever instead of failing. That is also the only hypothesis that explains the intermittency: the same command finished in 73 seconds on one run and never on the next, which is mirror state rather than invocation shape.

Three earlier hypotheses are retired by this: npx resolution, Playwright flakiness, and the missing harden-runner step. None of them touch apt, and the second hang happened in a job that has harden-runner.

The fix

/etc/apt/apt.conf.d/99-ci-timeouts sets Acquire::Retries and per-scheme timeouts. Playwright owns the apt-get invocation, so bounding apt through its own configuration is what reaches the child process — passing flags is not an option here.

A timeout 300 plus three attempts wraps the step for whatever survives that, and the step no longer discards its output. Worst case is 15 minutes, inside the job's 20-minute bound from #101. A retry emits ::warning:: so the flake stays visible instead of being silently absorbed.

What this does not claim

An intermittent fault cannot be proven gone by one green run. What is verifiable here is that the failure mode is now bounded at every level — apt fails instead of hanging, the step ends instead of the job, the job ends instead of the runner — and that the diagnosis rests on log evidence from two independent hangs rather than on elimination.

Assisted by claude-code:claude-opus-5 — Session

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[CybotTM]

Verification dispatch on this branch: 32289956430, job green end to end, Snapshots already current — nothing to refresh (baselines on main are current, so it correctly opened no PR).

The interesting part is that this run hit the stall and recovered from it, so it is evidence rather than a lucky pass:

18:54:13  Installing dependencies...
18:54:14  Get:1 file:/etc/apt/apt-mirrors.txt Mirrorlist [144 B]
18:56:10  Get:1 file:/etc/apt/apt-mirrors.txt Mirrorlist [144 B]

The mirror fetch is issued, goes quiet for nearly two minutes, and is issued again — that second Get:1 is Acquire::Retries firing after Acquire::http::Timeout cut the dead connection. Under the old configuration that same silence had no end: run 32269308335 sat on this exact line for twenty minutes.

The install step took 4m15s against a 73–80 second baseline, so the stall cost real time — and then finished. Zero attempt N did not complete warnings, meaning the step-level timeout 300 never had to fire: apt bounded itself, which is the layer the fix targets.

To be precise about what this shows: one run that stalled and recovered demonstrates the bound works under the failure condition. It does not prove every future stall stays under 300s — that is what the retry and the job timeout are for.

Assisted by claude-code:claude-opus-5 — Session