feat(ci): add BATS coverage reporting artifacts - #911
Conversation
hanthor
left a comment
There was a problem hiding this comment.
The required unit-test workflow is red because sudo apt-get install -y bats kcov fails on the Ubuntu 24.04 runner (E: Unable to locate package kcov). Please install kcov from an available, pinned source (or use a runner/image that provides it) before enabling the coverage step.
b8f2245 to
9e56872
Compare
|
I opened castrojo#1 against this PR's head branch. It fixes the unavailable Ubuntu 24.04 Validation exposed a second issue: wrapping only the top-level |
Run the existing BATS unit tests under kcov and upload the generated report alongside TAP output. Document the new artifact in the CI skill. Assisted-by: goose
Build kcov v43 from a checksum-verified pinned source because Ubuntu 24.04 does not package it. Instrument child Bash processes, map patched test sandboxes back to repository sources, and emit merged HTML and Cobertura reports that fail when coverage is empty. Assisted-by: GPT-5.6 Sol via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Streaming the full BATS TAP log through kcov's captured stdout pipe deadlocks the job: kcov stops draining the pipe once its buffer fills, so the suite stalls indefinitely. Redirect the coverage run's BATS output to a file instead, which also removes the pipe backpressure that made the instrumented run take tens of minutes (now ~17s locally). Use the documented --bash-parse-files-in-dir flag, cache the pinned kcov v43 build so repeat runs skip the compile, make the child-shell wrapper fall back to the real bash when kcov is not on PATH, and raise the job timeout to cover a cold kcov build. Verified in an Ubuntu 24.04 container: 269/564 shell lines (47.7%) attributed to real build_files/ and system_files/ sources. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
f8e84ca to
3a90f74
Compare
|
@hanthor thanks — your blocking review was correct, and validation turned up a second (worse) defect. Both are fixed, and this branch now also carries @bketelsen's work from castrojo#1. 1. 2. The report was 0%. Wrapping only the top-level 3. (found during validation) the coverage step deadlocked. With the instrumented run streaming the full TAP log through kcov's captured stdout pipe, kcov stops draining once the pipe buffer fills and the job hangs forever — I reproduced this twice locally (hung at ~96 traced child runs, no progress for 15+ min). Fixed by redirecting the coverage run's BATS output to a file. That also removed the backpressure: the instrumented suite now finishes in ~17 seconds instead of stalling. Evidence (Ubuntu 24.04 container, pinned kcov v43, exact CI command): Verified identical BATS pass/fail results with and without kcov, so instrumentation does not change test outcomes. Known limitation, documented in Also closed #974, which was a byte-identical resubmit of this PR targeting |
The instrumented rerun exists to collect coverage; `Run unit tests` owns pass/fail. Ubuntu runners currently have 18 pre-existing BATS failures in `04-install-kernel-akmods_test.bats` and friends that the `| tee` pipeline in that step masks, so aborting the coverage step on a non-zero BATS exit made the new step the only place those failures surfaced. Record the exit status instead and keep failing when kcov collected no executed lines. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Summary
kcovin PR validation unit-test jobbuild_files/andsystem_files/bats-kcov-reportartifactWhy
Issue #518 reports that Bluefin has BATS unit tests but no shell coverage reporting. This adds CI-visible coverage output without requiring external secrets or token setup.
Fixes #518
Validation
python3 -m compileall testsjust check(blocked locally:justnot installed in this environment)pre-commit run --all-files(blocked locally becausejust/toolchain prerequisites are unavailable)