Skip to content

Add Test Analytics #13225

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Apr 28, 2025
Merged

Add Test Analytics #13225

merged 13 commits into from
Apr 28, 2025

Conversation

katia-sentry
Copy link
Contributor

@katia-sentry katia-sentry commented Apr 23, 2025

Hey there! I work over at Sentry on the Codecov team and we've set a goal for adding some of our new features to open source repos already using Codecov. I saw this repo using Codecov, and it seemed like a good fit :)

What is the new features?

Test Analytics is a new Codecov feature providing insights into your test suite's health.

  • Overview of test run times and failure rates across branches.
  • PR comments listing failed tests with stack traces for easier debugging.
  • Identification of flaky tests from main in PR comments, enabling quick retries and fixes.

Looking forward to your feedback!

@katia-sentry katia-sentry requested a review from larsoner as a code owner April 23, 2025 15:08
Copy link

welcome bot commented Apr 23, 2025

Hello! 👋 Thanks for opening your first pull request here! ❤️ We will try to get back to you soon. 🚴

@@ -242,8 +242,10 @@ jobs:
command: |
if [[ $(cat gitlog.txt) == *"[circle front]"* ]] || [[ $(cat build.txt) == "html-memory" ]] ; then
make test-doc;
coverage run -m pytest --junitxml=test-results.xml;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we want to run pytest here. Our test-doc run is:

test-doc: sample_data testing_data
	$(PYTESTS) --doctest-modules --doctest-ignore-import-errors --doctest-glob='*.rst' ./doc/ --ignore=./doc/auto_examples --ignore=./doc/auto_tutorials --ignore=./doc/_build --ignore=./doc/conf.py --ignore=doc/sphinxext --fulltrace

And we already produce a suitable junit.yml I think... we probably just need to upload the one we already create

@larsoner
Copy link
Member

@katia-sentry let me know if you want me to push directly to adapt or if you want to iterate a bit. Feel free to add your name to doc/changes/names.inc plus a changelog entry in doc/changes/13225.other.rst with the :newcontrib: reST directive as well!

@katia-sentry
Copy link
Contributor Author

@larsoner appreciate your review and feedback. I made the requested changes. Let me know if I missed anything, or more iteration is required. Thank you!

@larsoner larsoner merged commit 1d34cd9 into mne-tools:main Apr 28, 2025
8 checks passed
Copy link

welcome bot commented Apr 28, 2025

🎉 Congrats on merging your first pull request! 🥳 Looking forward to seeing more from you in the future! 💪

@larsoner
Copy link
Member

Thanks @katia-sentry !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants