Skip to content

Use skip markers in tests #4188

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 8 commits into
base: develop
Choose a base branch
from
Open

Use skip markers in tests #4188

wants to merge 8 commits into from

Conversation

Flamefire
Copy link
Contributor

@Flamefire Flamefire commented Jan 19, 2023 •
edited by branfosj
Loading

From #3790

We can use the more conventional @unittest.skipIf or self.skipTest markers to skip tests instead of C&P of conditions which might be wrong (some were)
An approach taken from IIRC PyTorch is to define own decorators requiresFoo as a wrapper around a specific skipIf which is reusable and easier to understand.

Only issue I can see is with the change to CI to ignore all skipped tests as now that I look at it previously we were able to selectively ignore some skipped tests.
So maybe we should not allow any skipped tests on CI which usually means installing additional Python packages. And for the tests requiring a github token we then need to resort back to the old approach but maybe we can still use a decorator here

@boegel boegel added the tests label Jan 21, 2023
@boegel boegel added this to the next release (4.7.1?) milestone Jan 21, 2023
boegel
boegel reviewed Jan 21, 2023
View reviewed changes
@boegel
Copy link
Member

boegel commented Feb 1, 2023

Only issue I can see is with the change to CI to ignore all skipped tests as now that I look at it previously we were able to selectively ignore some skipped tests. So maybe we should not allow any skipped tests on CI which usually means installing additional Python packages. And for the tests requiring a github token we then need to resort back to the old approach but maybe we can still use a decorator here

@Flamefire How about only allowing \.s+ in the output when running the test suite in a PR?
Because only then should we allow skipped tests (because the GitHub token is missing)...

So we should conditionally extend IGNORE_PATTERNS with |\.s+ if the test suite is being triggered by a push to develop branch (I think)...

@Flamefire
Copy link
Contributor Author

So we should conditionally extend IGNORE_PATTERNS with |\.s+ if the test suite is being triggered by a push to develop branch (I think)...

I guess you meant to insert a NOT somewhere as no tests should be skipped on push to develop.

How about only allowing \.s+ in the output when running the test suite in a PR?
Because only then should we allow skipped tests (because the GitHub token is missing)...

Makes sense, however we should be more selective here. I hence added a dedicated decorator for that which doesn't produce output for PRs (i.e. silently succeeds) and make use of $FORCE_EB_GITHUB_TESTS to ensure github tests are run when the token is expected to be there.

@Flamefire Flamefire force-pushed the skip-markers branch 5 times, most recently from f76b45c to 65320fb Compare February 2, 2023 12:20
@Flamefire Flamefire mentioned this pull request Feb 2, 2023
Open
2 tasks
@Flamefire Flamefire changed the title Use skip markers and skipTest in tests Use skip markers in tests Feb 2, 2023
@Flamefire Flamefire force-pushed the skip-markers branch 7 times, most recently from 05c14fe to 2a38d1e Compare February 7, 2023 15:08
@easybuilders easybuilders deleted a comment from boegelbot Feb 15, 2023
@easybuilders easybuilders deleted a comment from boegelbot Feb 15, 2023
@easybuilders easybuilders deleted a comment from boegelbot Feb 15, 2023
@Flamefire Flamefire force-pushed the skip-markers branch 2 times, most recently from 4040414 to 149f7ec Compare May 11, 2023 08:56
@boegel boegel added this to the release after 4.8.1 milestone Sep 3, 2023
@boegel boegel modified the milestones: 4.9.1, release after 4.9.1 Apr 3, 2024
@Flamefire Flamefire force-pushed the skip-markers branch 2 times, most recently from 5c5c337 to b6b1f07 Compare April 16, 2024 10:45
@Flamefire Flamefire requested a review from boegel April 25, 2024 12:29
@boegel boegel modified the milestones: 4.9.2, release after 4.9.2 Jun 6, 2024
@boegel boegel modified the milestones: 4.9.3, release after 4.9.3 Aug 28, 2024
Flamefire added 7 commits March 4, 2025 17:29
@Flamefire
Use skipTest than faulty C&P of return conditions
b0087b4
@Flamefire
Silently skip SVN test on CI
54f7ff4 
The pip-installable pysvn is incomplete and the full one is not easy to install.
@Flamefire
Avoid output of expected skip of LMod tests
e664bec 
Simply return if `$TEST_EASYBUILD_MODULES_TOOL` is not set to Lmod.
This avoids any "skip" output on CI
@Flamefire
Move GC3Pie check to skip decorator
cfe5799
@Flamefire
Introduce skip marker for github token-using tests
3757ee8 
For PRs silently skip the tests when no token is available else use a skip.
On GHA force enable the tests when the source repo is 'easybuilds' so a
token is expected to be available.
@Flamefire
Introduce skip_silent* decorators
e27d487 
As that is used a lot use that for better describing the purpose.
@Flamefire
Silently skip GC3PIe test w/ Python 3.11
3340a45 
GC3Pie on Python 3.11 does not work: gc3pie/gc3pie#674
So avoid the skip mark for that.
@boegel boegel modified the milestones: release after 4.9.4, release after 5.0.0 Mar 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@boegel boegel Awaiting requested review from boegel

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
tests
Projects
None yet
Milestone
release after 5.1.0
Development

Successfully merging this pull request may close these issues.
2 participants
@Flamefire @boegel