test: start tracking test coverage

[anakin87]

Related Issues

• part of https://github.com/deepset-ai/haystack-private/issues/275

We want to start tracking test coverage and add comments on PRs with this info.
Ideally, we want to track unit test coverage, but also secondarily keep track of combined test coverage (for some integrations like Document Stores, most tests are integration tests).

We would also like to avoid relying on hosted tools.

I have investigated several options involving GitHub actions, but I finally chose https://github.com/py-cov-action/python-coverage-comment-action for its ability to show positive or negative variations of test coverage natively.

Proposed Changes:

This is a pilot, so I am just introducing this change for the Kreuzberg integration.

• we now also run tests on pushes to main (just one operating system and one Python version) to populate baseline coverage (stored on a branch)
  • we store both unit tests coverage and combined coverage. Currently, combined coverage is not shown, but I'd plan to show it in the README when all integrations are covered by this change.
    Result should look like this:

• using the mentioned action on PRs, prepare and save a coverage comment as a GitHub artifact
• another workflow takes these comments and publishes them on the PR (this split is needed to make everything work securely when PRs from forks are expected - Docs)

How did you test it?

I tried this extensively on my fork. See for example anakin87#5

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.

[github-actions]

Coverage report (kreuzberg)

This PR does not seem to contain any modification to coverable code.

[davidsbatista]

Looks good to me, seems that the comment posted on PRs will exclude integration test coverage? Is the reason that some community members PR don't run the integration tests?

[davidsbatista]

LGTM - just a question/curiosity

[anakin87]

Looks good to me, seems that the comment posted on PRs will exclude integration test coverage? Is the reason that some community members PR don't run the integration tests?

We don't expose API keys on PRs running from forks for security reasons. So, in this case, integration tests requiring API keys are skipped and combined coverage cannot be computed.