Skip to content

feat(ci): add BATS coverage reporting artifacts - #911

Merged
castrojo merged 4 commits into
projectbluefin:testingfrom
castrojo:fix/issue-518-bats-coverage
Aug 7, 2026
Merged

feat(ci): add BATS coverage reporting artifacts#911
castrojo merged 4 commits into
projectbluefin:testingfrom
castrojo:fix/issue-518-bats-coverage

Conversation

@castrojo

Copy link
Copy Markdown
Contributor

Summary

  • install kcov in PR validation unit-test job
  • run BATS tests with coverage instrumentation for build_files/ and system_files/
  • upload coverage output as a bats-kcov-report artifact
  • keep existing TAP artifact upload and document this behavior in the CI skill

Why

Issue #518 reports that Bluefin has BATS unit tests but no shell coverage reporting. This adds CI-visible coverage output without requiring external secrets or token setup.

Fixes #518

Validation

  • python3 -m compileall tests
  • just check (blocked locally: just not installed in this environment)
  • pre-commit run --all-files (blocked locally because just/toolchain prerequisites are unavailable)

@castrojo
castrojo requested a review from a team as a code owner July 28, 2026 16:58
@castrojo castrojo added the 4-review A pull request is awaiting review. label Jul 28, 2026

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The required unit-test workflow is red because sudo apt-get install -y bats kcov fails on the Ubuntu 24.04 runner (E: Unable to locate package kcov). Please install kcov from an available, pinned source (or use a runner/image that provides it) before enabling the coverage step.

@bketelsen

Copy link
Copy Markdown

I opened castrojo#1 against this PR's head branch. It fixes the unavailable Ubuntu 24.04 kcov package by building v43 from a checksum-verified pinned commit.

Validation exposed a second issue: wrapping only the top-level bats process produces a 0% report because the scripts under test run in child shells. The stacked PR source-maps those child runs back to the original repository files and emits merged HTML/Cobertura output.

castrojo and others added 3 commits August 6, 2026 23:43
Run the existing BATS unit tests under kcov and upload the generated report alongside TAP output. Document the new artifact in the CI skill.

Assisted-by: goose
Build kcov v43 from a checksum-verified pinned source because Ubuntu 24.04 does not package it. Instrument child Bash processes, map patched test sandboxes back to repository sources, and emit merged HTML and Cobertura reports that fail when coverage is empty.

Assisted-by: GPT-5.6 Sol via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Streaming the full BATS TAP log through kcov's captured stdout pipe
deadlocks the job: kcov stops draining the pipe once its buffer fills, so
the suite stalls indefinitely. Redirect the coverage run's BATS output to
a file instead, which also removes the pipe backpressure that made the
instrumented run take tens of minutes (now ~17s locally).

Use the documented --bash-parse-files-in-dir flag, cache the pinned kcov
v43 build so repeat runs skip the compile, make the child-shell wrapper
fall back to the real bash when kcov is not on PATH, and raise the job
timeout to cover a cold kcov build.

Verified in an Ubuntu 24.04 container: 269/564 shell lines (47.7%)
attributed to real build_files/ and system_files/ sources.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@castrojo
castrojo force-pushed the fix/issue-518-bats-coverage branch from f8e84ca to 3a90f74 Compare August 7, 2026 03:43
@castrojo

castrojo commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

@hanthor thanks — your blocking review was correct, and validation turned up a second (worse) defect. Both are fixed, and this branch now also carries @bketelsen's work from castrojo#1.

1. kcov is not installable on Ubuntu 24.04 (your review). Confirmed: apt-get install -y kcov in an ubuntu:24.04 container gives E: Unable to locate package kcov — kcov was dropped from noble. Fixed by building kcov v43 from a pinned source commit (a39874f938ce13f7a65f253120d1ec946b349ffe) whose release archive SHA-256 is verified with sha256sum --check before it is unpacked. No PPA, no floating URL, no unverified binary. The build output is cached (actions/cache, key kcov-v43-a39874f-...) so only the first run pays the compile cost; job timeout raised to 25 minutes to cover a cold build.

2. The report was 0%. Wrapping only the top-level bats process produces a valid-looking but empty report, because every test executes the script under test in a child shell (run bash "${PATCHED_SCRIPT}") against a patched sandbox copy. Fixed by routing those child bash <script> calls through tests/coverage/bin/bash, which starts a nested kcov run and records the sandbox copy's original build_files//system_files/ path; tests/coverage/merge_kcov.py then merges those child hits into kcov's pre-parsed source inventory and emits HTML + Cobertura + a step-summary table. It hard-fails if zero lines were executed, so a silent 0% report can never ship again.

3. (found during validation) the coverage step deadlocked. With the instrumented run streaming the full TAP log through kcov's captured stdout pipe, kcov stops draining once the pipe buffer fills and the job hangs forever — I reproduced this twice locally (hung at ~96 traced child runs, no progress for 15+ min). Fixed by redirecting the coverage run's BATS output to a file. That also removed the backpressure: the instrumented suite now finishes in ~17 seconds instead of stalling.

Evidence (Ubuntu 24.04 container, pinned kcov v43, exact CI command):

BATS shell coverage: 269/564 lines (47.7%)

| File | Covered | Valid | Coverage |
| build_files/base/00-image-info.sh          |  39 |  40 |  97.5% |
| build_files/base/03-packages.sh            |  26 |  26 | 100.0% |
| build_files/base/05-override-install.sh    |  21 |  21 | 100.0% |
| build_files/base/17-cleanup.sh             |  28 |  29 |  96.6% |
| build_files/shared/build-gnome-extensions.sh | 22 | 22 | 100.0% |
| system_files/.../11-framework-ucsi-workaround.sh | 17 | 23 | 73.9% |
| ... 27 files total

Verified identical BATS pass/fail results with and without kcov, so instrumentation does not change test outcomes. Known limitation, documented in docs/skills/ci/SKILL.md: tests that source a library directly into the BATS process (e.g. package-lib, validate-repos) are not traced.

Also closed #974, which was a byte-identical resubmit of this PR targeting main in violation of the testing-first flow.

The instrumented rerun exists to collect coverage; `Run unit tests` owns
pass/fail. Ubuntu runners currently have 18 pre-existing BATS failures in
`04-install-kernel-akmods_test.bats` and friends that the `| tee` pipeline
in that step masks, so aborting the coverage step on a non-zero BATS exit
made the new step the only place those failures surfaced. Record the exit
status instead and keep failing when kcov collected no executed lines.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

4-review A pull request is awaiting review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants