Skip to content

Improve Coverage Accuracy and Speed Up Tests#144

Open
Darshan808 wants to merge 10 commits into
jupyterlab:mainfrom
Darshan808:increase-coverage
Open

Improve Coverage Accuracy and Speed Up Tests#144
Darshan808 wants to merge 10 commits into
jupyterlab:mainfrom
Darshan808:increase-coverage

Conversation

@Darshan808

@Darshan808 Darshan808 commented Jul 2, 2026

Copy link
Copy Markdown
Member

Fixes #95

Description

Our coverage number was misleading. It reported ~16%, but that was because almost all of the real testing happens through subprocess.run(["jupyter-builder", ...]) calls, and coverage does not track subprocesses by default. The code was being tested, just not measured.

  • Use coverage.py 7.10's patch = ["subprocess"], so the jupyter-builder and jlpm CLIs collect coverage themselves and pytest-cov merges everything into a single report.

Result: Coverage increased from ~16% → ~70% without changing any tests.

Faster tests

  • Move the template clone, jlpm install, and TypeScript build into session-scoped fixtures (tests/conftest.py). Each test now gets a cheap local copy instead of repeating the expensive setup.
    • Before: 6 clones / 6 installs
    • After: 1 clone / 3 installs (one per template variant)
  • Replace fixed sleeps in watch tests with polling for build output.
  • Mark all tests in test_tpl.py as slow, so pytest -m "not slow" provides a much faster local test run.

@Darshan808 Darshan808 added the maintenance Change related to maintenance of the repository label Jul 2, 2026
@Darshan808

Copy link
Copy Markdown
Member Author

Some progress here.
On ubuntu and MacOS runners, I see 70% coverage. But on windows runners I see 62% coverage.

@Darshan808 Darshan808 changed the title Increase coverage Improve Coverage Accuracy and Speed Up Tests Jul 4, 2026
@Darshan808

Copy link
Copy Markdown
Member Author

The version mismatch test started to act flaky.

@Darshan808

Copy link
Copy Markdown
Member Author

Ah, Test is being flaky as it is hitting rate limits when trying to pull core-meta from github.

(HTTP Error 403: rate limit exceeded)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to (1) make CI coverage reporting accurate for code exercised via subprocess-driven CLI tests and (2) reduce test runtime by moving expensive template setup/build work into shared pytest fixtures.

Changes:

  • Enable subprocess-aware coverage collection via coverage.py config (patch = ["subprocess"]) and re-enable the CI coverage gate.
  • Speed up tests by introducing session-scoped fixtures that render/build the template once and provide per-test copies.
  • Reduce watch-test flakiness by replacing fixed sleeps with polling, and mark template tests as slow.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
tests/test_tpl.py Refactors template/watch tests to use shared fixtures, adds polling for watch tests, and marks all tests as slow.
tests/conftest.py Adds session-scoped fixtures to render/build/install the template once and provide isolated per-test copies; sets COVERAGE_FILE.
pyproject.toml Updates test dependencies and adds coverage + pytest marker configuration to support subprocess coverage and slow marker.
jupyter_builder/debug_log_file_mixin.py Fixes empty-path directory handling for debug logs and broadens best-effort unlink suppression for Windows.
.github/workflows/builder-tests.yml Re-enables the coverage job to enforce a minimum coverage threshold in CI.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/test_tpl.py
Comment thread tests/test_tpl.py
Comment thread jupyter_builder/debug_log_file_mixin.py
Comment thread tests/test_tpl.py
Darshan808 and others added 2 commits July 5, 2026 14:12
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@Darshan808 Darshan808 marked this pull request as ready for review July 5, 2026 08:28
@Darshan808 Darshan808 requested a review from krassowski July 5, 2026 08:28
@Darshan808 Darshan808 self-assigned this Jul 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintenance Change related to maintenance of the repository

Projects

Status: Todo
Status: No status

Development

Successfully merging this pull request may close these issues.

Run coverage on CI

2 participants