Coverage not captured for Bash functions tested by Bats

[KlfJoat]

I'm running kcov on a few different repos of Bash scripts where I have extensive Bats test suites. I'd like to use it with the VSC extension Coverage Gutters.

In these repos, I make heavy use of Bash functions. But I'm running into a problem where it seems like my extensive use of functions somehow hides the code from kcov.

I have uploaded a runnable example at the following repo: https://github.com/KlfJoat/gh-issue-kcov-nofuncsbats

Versions of relevant programs

• kcov - v43-21-g30d3 (self-compiled)
• bash - 5.2.32(1)-release (Ubuntu 24.10)
• Bats - 1.12.0 (latest)

I've tried with both default (which I think uses PS4) and --bash-method=DEBUG. DEBUG hides all results from me and never shows anything on the screen.

What I run

kcov --cobertura-only --include-path=. --exclude-path=test,coverage --clean coverage test/bats/bin/bats --verbose-run test/*.bats

Expected Output

A cov.xml file that shows any code in the function has been executed at least once.

Actual Output

On stdout

1..9
ok 1 bash_versionnewerthan noarg fail
ok 2 bash_versionnewerthan 1 blankarg fail
ok 3 bash_versionnewerthan major alpha fail
ok 4 bash_versionnewerthan minor alpha fail
ok 5 bash_versionnewerthan patch alpha fail
ok 6 bash_versionnewerthan 4.4.0 pass
ok 7 bash_versionnewerthan current pass
Your bash version is  '5.2.32(1)-release' ]]
Your bash version is  '5.2.32(1)-release' != *\T\h\i\s\ \s\c\r\i\p\t\ \r\e\q\u\i\r\e\s\ \a\ \b\a\s\h\ \v\e\r\s\i\o\n\ \8\.\1\.\2\3\ \o\r\ \n\e\w\e\r\.* ]]
ok 8 bash_versionnewerthan 8.1.23 fail
Your bash version is  '5.2.32(1)-release' ]]
Your bash version is  '5.2.32(1)-release' != *\T\h\i\s\ \s\c\r\i\p\t\ \r\e\q\u\i\r\e\s\ \a\ \b\a\s\h\ \v\e\r\s\i\o\n\ \8\0\.\0\.\0\ \o\r\ \n\e\w\e\r\.* ]]
ok 9 bash_versionnewerthan major 80 fail

(the random output besides TAP seems to be Bats interacting with kcov; it's not there when I run straight Bats)

In coverage/cov.xml

No tracking of the execution of the function, despite stdout clearly showing the test executed successfully.

Every line shows 0 hits except for the export -f where I make the function visible to Bats.

[SimonKagstrom]

Thanks for the report, it's very well written!

I don't run bats myself (or much of bash coverage collection in general), but I tried running your repo. For me it doesn't work at all, but it appears I see the trace output in the shell:

**Simons-iMac-7:gh-issue-kcov-nofuncsbats ska$ kcov --bash-method=PS4 --cobertura-only --include-path=. --exclude-path=test,coverage --clean coverage bats test/library.bats  | head
1..9
not ok 1 bash_versionnewerthan noarg fail
# (from function `__assert_stream' in file test/bats_helpers/bats-assert/src/assert_output.bash, line 246,
#  from function `assert_output' in file test/bats_helpers/bats-assert/src/assert_output.bash, line 125,
#  in test file test/library.bats, line 23)
#   `assert_output '[bash_versionnewerthan] You passed no arguments or more than 3 arguments to test the bash version.'' failed
# kkcov@/usr/local/Cellar/bats-core/1.12.0/libexec/bats-core/bats-exec-test@353@bats_debug_trap /usr/local/Cellar/bats-core/1.12.0/libexec/bats-core/bats-exec-test
# kkcov@/usr/local/Cellar/bats-core/1.12.0/lib/bats-core/tracing.bash@278@local NORMALIZED_INPUT
# kkcov@/usr/local/Cellar/bats-core/1.12.0/lib/bats-core/tracing.bash@279@bats_normalize_windows_dir_path NORMALIZED_INPUT /usr/local/Cellar/bats-core/1.12.0/libexec/bats-core**

and no output as far as I can see. If I remove --include-path, I do see collection of the bats infrastructure itself:

				<class name="bats_exec_test__30" filename="usr/local/Cellar/bats-core/1.12.0/libexec/bats-core/bats-exec-test" branch-rate="1.0" complexity="1.0" line-rate="0.274">
				<class name="bats_exec_file__31" filename="usr/local/Cellar/bats-core/1.12.0/libexec/bats-core/bats-exec-file" branch-rate="1.0" complexity="1.0" line-rate="0.528">
				<class name="common_bash__32" filename="usr/local/Cellar/bats-core/1.12.0/lib/bats-core/common.bash" branch-rate="1.0" complexity="1.0" line-rate="0.112">
				<class name="tracing_bash__33" filename="usr/local/Cellar/bats-core/1.12.0/lib/bats-core/tracing.bash" branch-rate="1.0" complexity="1.0" line-rate="0.260">
				<class name="preprocessing_bash__34" filename="usr/local/Cellar/bats-core/1.12.0/lib/bats-core/preprocessing.bash" branch-rate="1.0" complexity="1.0" line-rate="1.000">
				<class name="1_library_bats_src__35" filename="private/var/folders/q0/7smwmwmd4wj6twb4yw1bpz540000gn/T/bats-run-xxjIA3/1-library.bats.src" branch-rate="1.0" complexity="1.0" line-rate="0.229">
				<class name="warnings_bash__36" filename="usr/local/Cellar/bats-core/1.12.0/lib/bats-core/warnings.bash" branch-rate="1.0" complexity="1.0" line-rate="0.087">
				<class name="bats_preprocess__37" filename="usr/local/Cellar/bats-core/1.12.0/libexec/bats-core/bats-preprocess" branch-rate="1.0" complexity="1.0" line-rate="0.625">
				<class name="test_functions_bash__38" filename="usr/local/Cellar/bats-core/1.12.0/lib/bats-core/test_functions.bash" branch-rate="1.0" complexity="1.0" line-rate="0.085">
				<class name="bats_gather_tests__39" filename="usr/local/Cellar/bats-core/1.12.0/libexec/bats-core/bats-gather-tests" branch-rate="1.0" complexity="1.0" line-rate="0.331">
				<class name="bats__40" filename="usr/local/Cellar/bats-core/1.12.0/bin/bats" branch-rate="1.0" complexity="1.0" line-rate="0.297">
				<class name="bats_exec_suite__41" filename="usr/local/Cellar/bats-core/1.12.0/libexec/bats-core/bats-exec-suite" branch-rate="1.0" complexity="1.0" line-rate="0.452">
				<class name="validator_bash__42" filename="usr/local/Cellar/bats-core/1.12.0/lib/bats-core/validator.bash" branch-rate="1.0" complexity="1.0" line-rate="0.929">
				<class name="bats_format_cat__43" filename="usr/local/Cellar/bats-core/1.12.0/libexec/bats-core/bats-format-cat" branch-rate="1.0" complexity="1.0" line-rate="1.000">
				<class name="bats__44" filename="usr/local/Cellar/bats-core/1.12.0/libexec/bats-core/bats" branch-rate="1.0" complexity="1.0" line-rate="0.360">

[KlfJoat]

Hi,

Thanks for looking at my bug report.

In an attempt to recreate your problem with my repo, I went through the following steps.

1. I created a new minimal install Ubuntu 24.10 VM.
2. I installed git, because minimal install.

sudo apt install git

3. I cloned my repo & updated submodules. (the below is entirely copy/pasteable)

git clone https://github.com/KlfJoat/gh-issue-kcov-nofuncsbats.git
cd gh-issue-kcov-nofuncsbats || exit 1
git submodule init && git submodule update

4. I cloned & compiled kcov (you will find PR Add cmake & build-essential to Ubuntu build packages #477 where I found some documentation issues in INSTALL.md related to my minimal Ubuntu install).
5. I ran my example command from my repo root. (the below is entirely copy/pasteable)

if git rev-parse --is-inside-work-tree &>/dev/null; then \
  cd $(git rev-parse --show-toplevel) && kcov --cobertura-only --include-path=. --exclude-path=test,coverage --clean coverage test/bats/bin/bats --verbose-run test/*.bats; \
fi

I received the same output (both stdout and cov.xml) as in my initial report.

Please let me know how I can help you to reproduce my issue.

Thanks!

[SimonKagstrom]

Forgot to say (although perhaps obvious from the printouts) that I'm on MacOS. Bash version 5.2.37.

If you run your script with just PS4 (i.e., without kcov), do you see the traces then?

Another think you can try is to disable the xtrace FD support. Can be done through the --debug-force-bash-stderr flag to kcov.

[KlfJoat]

Just to be clear, are you running the exact kcov command line I have? Your version shows a lot of differences, including just running bats as the executable.

Please run the bats test with test/bats/bin/bats test/*.bats. Otherwise, you're using system bats, not the one you grabbed as a submodule to my repo.

[KlfJoat]

If you run your script with just PS4 (i.e., without kcov), do you see the traces then?

Do you mean run bash -x test/bats/bin/bats test/*.bats? That shows approx 20 lines of Bats running, then it abruptly stops and shows me Bats output. It never shows any of my script's commands.

That makes no sense to me, though. Because otherwise, how does kcov see the parts of my scripts that do run?

Another think you can try is to disable the xtrace FD support. Can be done through the --debug-force-bash-stderr flag to kcov.

That just sprays what appears to be the same text over and over on my screen from Bats. Lines and lines constantly. Like...

# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@14@read -r line
# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@15@printf '%s\n' '# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@15@printf '\''%s\n'\'' '\''# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@15@printf '\''\'\'''\''%s\n'\''\'\'''\'' '\''\'\'''\''# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@4@read -r line'\''\'\'''\'''\'''
# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@14@read -r line
# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@15@printf '%s\n' '# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@14@read -r line'
# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@14@read -r line
# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@15@printf '%s\n' '# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@15@printf '\''%s\n'\'' '\''# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@14@read -r line'\'''
# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@14@read -r line
# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@4@IFS=
# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@15@printf '%s\n' '# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@14@read -r line'
^C# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@4@read -r line
# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@14@read -r line
# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@5@printf '# %s\n' 'kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@14@read -r line'
# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@4@IFS=
# kcov@/home/klfjoat/src/gh-issue-kcov-nofuncsbats/test/bats/lib/bats-core/common.bash@4@read -r line

I hit Ctrl-C after waiting more than 5 minutes, and got this.

# Received SIGINT, aborting ...

# bats warning: Executed 1 instead of expected 9 tests

[jakub-bochenski]

hey @KlfJoat I'm also testing some sourced functions working with BATS/kcov

I see the are covered (mostly) fine.
The one difference I see is that I'm not actually exporting the functions. Instead I'm sourcing them in the test file outside of test cases/setup.

[KlfJoat]

The one difference I see is that I'm not actually exporting the functions. Instead I'm sourcing them in the test file outside of test cases/setup.

Per Bats doc recommendation, I don't execute anything outside of tests/setup_*/load.

In general you should avoid code outside tests, because each test file will be evaluated many times. However, there are situations in which this might be useful, e.g. when you want to check for dependencies and fail immediately if they’re not present.

If the script you're sourcing doesn't have code at the top short-circuiting repeated sourcings, you're fully re-sourcing and re-running your sourced script multiple times for each Bats run. I don't remember how many times--I saw the number once. I think it's not O(n), but it's O(1) or O(log n). Still, all of that is unnecessary and specifically recommended against.

[jakub-bochenski]

Per Bats doc recommendation, I don't execute anything outside of tests/setup_*/load.

I've seen that. I've also seen that the official extensions source directly in code, e.g. https://github.com/bats-core/bats-assert/blob/master/test/assert.bats :)

Sure, reprocessing the file wastes some time. Other than that I don't see how else would that be a problem (assuming I only define functions in the soured file).

I don't remember how many times--I saw the number once. I think it's not O(n), but it's O(1) or O(log n).

It might be O(n^2) actually: bats-core/bats-core#1036