Why doesn't pytest-cov fail with a non-zero exit code when code coverage threshold isn't met, when running on multiple directories?

[samdewr]

Summary

Expected vs actual result

When the average test coverage threshold across multiple modules isn't met, pytest doesn't fail with a non-zero exit code, even though it should.

Reproducer

Versions

OS and hardware

  Model Name:	MacBook Pro
  Model Identifier:	Mac16,5
  Chip:	                 Apple M4 Max
  System Version:	macOS 15.6 (24G84)
  Kernel Version:	Darwin 24.6.0

Pytest versions

pytest = "^8.3.4"
pytest-cov = "^6.0.0"

Config

In pyproject.toml:

[tool.coverage.run]
branch = true

[tool.coverage.report]
fail_under = 75
exclude_lines = ['pragma: no cover', 'raise NotImplementedError', '\s*\.\.\.']

[tool.coverage.html]
directory = './coverage'

[tool.coverage.xml]
output = './coverage/cov.xml'

Code

My command:

❯ pytest --cov module1 --cov module2 --cov module3 --cov-report --cov-fail-under=75

I get the following result:

FAIL Required test coverage of 75.0% not reached. Total coverage: 74.79%

But the exit code remains zero:

❯ echo $?
0

As a result, my CI/CD pipeline (triggered upon a PR to main) passes just fine even though my coverage threshold isn't met. This causes my main branch to have too low test coverage.

I've tried

• Passing --cov=module1,module2,module3 instead of having one pytest call with three different --cov arguments. This causes the coverage report to fail altogether, because it says that CoverageWarning: Module module1,module2,module3 was never imported. I.e., it tries to import module1,module2,module3 as a single module and it fails, since they are separate modules.
• Passing the to-be-included modules in pyproject.toml under [tool.coverage.run] in the include argument. Problem persists: test coverage threshold failure is noted, but exit code is zero.
• Interestingly, when you test on each of the modules individually, you do get a non-zero exit code for the module that has too low coverage.
• On the internet, I can't really seem to find a working solution. Copilot isn't very useful here either.

[andywatkins-ancoris]

Thank you @samdewr for raising this. I also have this issue and found your comment on Stackoverflow, leading to this.

For anyone else coming across this issue, I worked around it in bash as follows, and thought it might be useful.
Put simply, I'm simultaneously sending the output to stdout and to a file using tee, and then looking for the error
message in the file using grep

pytest --cov module1 <other cov parameters> --cov-report --cov-fail-under=75  | tee pytestcov-output.txt
if grep -q 'FAIL Required test coverage of' pytestcov-output.txt; then
    exit 1
fi

[webknjaz]

@andywatkins-ancoris filterwarnings = error would be more elegant, though.

[samdewr]

Hi both, thanks for your replies! I have been able to reproduce the issue (see also this StackOverflow post).

Update including MWE

I've been able to reproduce the bug. See this repo README.

It looks like the rounding of the test coverage percentage is the issue. As e.g. in my example, the test coverage threshold isn't met (66.66% instead of the required 67%). Therefore, I rightfully see the message FAIL Required test coverage of 67% not reached. Total coverage: 66.67%. Nonetheless, the exit code seems to look at the rounded, zero-decimal percentage values of the test coverage vs. test coverage threshold, because the exit code is zero.