Skip to content

CI: run the suite in parallel, trace coverage on 3.12 only - #314

Merged
ashwin-agami merged 3 commits into
mainfrom
issue-296-ci-speed
Sep 14, 2026
Merged

CI: run the suite in parallel, trace coverage on 3.12 only#314
ashwin-agami merged 3 commits into
mainfrom
issue-296-ci-speed

Conversation

@ashwin-agami

@ashwin-agami ashwin-agami commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Closes #296.

lint + test spent ~600s per Python version in the test step, one test at a time, under coverage tracing, on all three legs.

What the measurement said

pytest tests/ --durations=40, serial, no coverage, py3.12, 12-core laptop:

  • 5m27s for 5535 passed / 12 skipped.
  • The slowest test is 4.25s; the 40 slowest together are ~80s of the 327s.

So there was no handful of slow tests to fix — the time is the long tail, and the lever is running it in parallel.

What this changes

before after
py3.10, py3.11 full suite, serial, --cov + floor full suite, -n auto, no coverage
py3.12 full suite, serial, --cov + floor full suite, -n auto, --cov + --cov-fail-under=85, COVERAGE_CORE=sysmon
  • The floor stays at 85 and is enforced on 3.12. The three legs measured the same coverage when the floor was set, and there is no [tool.coverage] branch setting, so sysmon runs without falling back.
  • Nothing is deselected or skipped on any leg. The two safety-corpus jobs are untouched and stay serial — their collection sentinels count per session.
  • The job and check names are unchanged, so branch protection is unaffected. tests/e2e/test_suite_integrity.py (which reads this workflow) still passes.
  • dev.py test / dev.py cover, the pre-push hook and CONTRIBUTING.md get the same -n auto, so local runs match CI.

Parallel safety

Measured locally, same machine, same commit:

run wall result
serial 5m27s 39 failed, 5535 passed
-n auto 1m31s 39 failed, 5535 passed — identical failure set
-n auto + sysmon coverage + floor (CI's 3.12 leg) 1m33s 39 failed, 5535 passed — identical failure set; 88.96%, floor of 85 reached

With sysmon, coverage added about 2 seconds to the parallel run. The floor is left at 85 — ratcheting it toward 89 is a separate change.

The 39 failures are #293 (a ~/.config/agami/path pointer on this machine leaking a real artifacts dir into the suite) — the same ids with and without -n auto, and CI has no such pointer. Before running it I also scanned the suite for fixed ports, shared /tmp paths, and session-scoped state that two workers could collide on, and found none that a worker process shares.

The extra warnings under -n auto (29 vs 7) are the same three deprecation warnings, reported once per worker.

On CI (this PR's first run, whole job including setup)

leg before (#296) after
py3.10 ~10m 3m53s
py3.11 ~10m 3m34s
py3.12 (coverage + floor) ~10m 2m55s

Guarding the split

tests/e2e/test_suite_integrity.py::test_exactly_one_leg_enforces_the_coverage_floor fails if no step, or more than one, carries --cov-fail-under, or if that step's condition names a version the matrix doesn't run — so the floor can't be switched off by an if: edit while every leg stays green. Mutation-checked against a flipped condition, an absent version, and a doubled floor. CLAUDE.md and pyproject.toml no longer say the floor doesn't exist yet.

🤖 Generated with Claude Code

ashwin-agami and others added 2 commits September 13, 2026 17:29
lint + test spent ~600s per Python version in pytest, serially, under
coverage tracing. --durations showed no handful of slow tests — the
slowest is ~4s — so the time is the long tail.

- all legs run pytest-xdist `-n auto`; nothing is deselected
- 3.12 alone collects coverage, with COVERAGE_CORE=sysmon, and keeps
  --cov-fail-under=85; 3.10/3.11 run the full suite without tracing
- dev.py test/cover, the pre-push hook and CONTRIBUTING match
- ignore per-worker .coverage.* files left by an interrupted run

Measured locally on py3.12: serial 5m27s; -n auto 1m31s; -n auto with
sysmon coverage 1m33s at 88.96%. Same failure set in all three (the
unrelated #293 home-pointer failures).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 14, 2026 00:31

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Add workflow contract assertions and update the stale coverage documentation.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR parallelizes local and CI test execution while keeping coverage enforcement on Python 3.12.

Changes:

  • Adds pytest-xdist with -n auto.
  • Uses sysmon coverage and an 85% floor on Python 3.12.
  • Updates contributor and pre-commit workflows.
  • Ignores per-worker coverage artifacts.
File summaries
File Summary
dev.py Enables parallel test execution.
CONTRIBUTING.md Documents parallel test commands.
.pre-commit-config.yaml Parallelizes pre-push tests.
.gitignore Ignores per-worker coverage files.
.github/workflows/ci.yml Splits parallel testing and 3.12 coverage enforcement.
Review details

Suppressed comments (1)

.github/workflows/ci.yml:77

  • Adding --cov-fail-under=85 here makes the repository's stated coverage contract stale: CLAUDE.md:60-62 and pyproject.toml:27-29 still say that no floor exists and that it will be added later. Please update those statements to describe the new 3.12-only CLI floor (without moving it into pyproject.toml, which would change local behavior).
          --cov-fail-under=85
  • Files reviewed: 4/5 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/ci.yml
Comment on lines 68 to +71
- name: pytest --cov
if: matrix.python-version == '3.12'
env:
COVERAGE_CORE: sysmon

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in de753ce. Added tests/e2e/test_suite_integrity.py::test_exactly_one_leg_enforces_the_coverage_floor: exactly one lint-and-test step may carry --cov-fail-under, and its if: must name a version the matrix actually runs. Mutation-checked — it fails on a flipped condition, on a version not in the matrix, and on the floor appearing in both steps.

I deliberately didn't assert -n auto or COVERAGE_CORE=sysmon: dropping either makes CI slower, not wrong, and a red build over a speed flag is a guard nobody wants. The floor is the correctness contract, so that's what's pinned.

…oor docs

Gating the floor to py3.12 makes it one `if:` edit away from gating no
leg at all while every leg stays green. The new suite-integrity test
fails if zero or several steps carry --cov-fail-under, or if the step's
condition names a version the matrix doesn't run. Mutation-checked: it
fails on a flipped condition, an absent version, and a doubled floor.

CLAUDE.md and pyproject.toml still said no floor existed; they now
point at the CI flag as the single source.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ashwin-agami

Copy link
Copy Markdown
Contributor Author

Re Copilot's suppressed comment on ci.yml:77 (stale floor docs): fixed in de753ce. CLAUDE.md and the [tool.pytest.ini_options] comment in pyproject.toml said no floor existed yet — already stale before this PR, since the floor has been in ci.yml since July. Both now point at --cov-fail-under on the py3.12 leg as the single source, and neither moves it into pyproject.toml, so a bare local pytest stays un-floored.

@ashwin-agami
ashwin-agami enabled auto-merge (squash) September 14, 2026 00:40
@ashwin-agami
ashwin-agami merged commit cdc34ac into main Sep 14, 2026
9 checks passed
@ashwin-agami
ashwin-agami deleted the issue-296-ci-speed branch September 14, 2026 00:42
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 14, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: lint + test takes ~10 min per Python version; the test step is almost all of it

2 participants