Skip to content

CI hygiene: job timeouts, fail-fast gating, committed shard timings #1370

Description

@pinin4fjords

From the July 2026 codebase assessment.

Evidence

  • No workflow in .github/workflows/ sets timeout-minutes on any job, so every job inherits GitHub's 6-hour default. A hung render or test shard burns 6 hours of runner time before being killed.
  • There is no needs: ordering: the 9-job test matrix (3 Python versions x 3 splits) and the pr-renders double-corpus render all launch in parallel with lint/format, so a formatting-only failure still pays for the full matrix.
  • pr-renders.yml renders the full 224-fixture corpus twice (base SHA + head SHA) on any non-.mmd change. The base-render cache is keyed purely on base SHA, so a rebased PR misses cache and re-renders everything.
  • pytest-split sharding is blind: no committed timing data, no --durations output, no slow markers, so nothing shows which of the 1,706 tests dominate wall-clock and whether the 3-way shard is balanced.
  • The gate-coverage ratchet only enforces on the CPython 3.11 lane (BASELINE_PYTHON = (3, 11) in scripts/routing_gate_coverage.py:55), which is by design but worth stating in the workflow so a 3.11 lane removal doesn't silently drop the ratchet.

Proposal

  1. Set timeout-minutes on every job (generous but finite, e.g. 30-45 for tests, 60 for renders).
  2. Gate the test matrix and pr-renders behind quick lint/format jobs via needs:.
  3. Commit a pytest-split timings file (or --durations snapshot) and refresh it periodically so sharding is balanced and slow tests are visible.
  4. Consider keying the base-render cache on the base tree hash rather than base SHA so rebases can still hit.

Metadata

Metadata

Assignees

No one assigned

    Labels

    assessment-2026-07Follow-ups from the July 2026 whole-codebase assessmentpriority:mediumModerate leverage / visual defect

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions