REP-001: Various rez-tests enhancements

[JeanChristopheMorinPerso]

Proposal to improve rez-test.

• Add run_on field for tests
• Add way to run 'all' tests (regardless of run_on)
• Add rez-test --interactive option
• Add rez-test --inplace option
• Pass custom arguments when running a test
• Add variant iteration options

Note: Some of these features clearly change rez-test into something a bit more general, so the name may have to change (to rez-task for eg). However we will consider this as a separate issue.

1. Add "run_on" field for tests.

This would define when the test gets run. Possible values:

• "default" (the default): Run when rez-test is run with no TEST arg(s).
• "pre-install": Run before a package install (not release); abort the install on failure.
• "pre-release": Run before package release, abort release on failure.
• "explicit": Only run if explicitly specified by rez-test, as a TEST arg.

The param would be a list, so a test can run on multiple of these events.

For pre-install/release, what would happen is:

1. The package source is built;
2. It is then installed to a temp dir.
3. This temp installation is then used to run the test(s);
4. If the tests pass, a second installation is performed to the correct location.

2. Run all tests defined by a package

I feel the need to be able to run all tests defined by a package, no matter what their run_on is set to.

rez-test <package> all

That would introduce a "reserved" target name all, but I think we should still let anyone refine its own all target, hence the quotes around the reserved word ;)

3. Pass custom arguments when running a test

It is common to have to modify the commands of the tests for a package while developing that said package. There is multiple reasons why we do so. For example, you might want to increase the verbosity of your tests only when developing and keep the verbosity lower in your CI tests. The other common reason is to choose a custom target inside a particular test so you can iterate faster and avoid running unnecessary tests.

For example, let's say we have:

tests = {
    'unit': {
        'command': 'pytest {root}/tests/unit_tests'
        'requires': 'pytest'
    }
}

where tests/unit_tests contains multiple test files and each file contains tests with markers (pytest jargon. A marker is basically a way to group the tests so that you can selectively choose which one to run in the command line).

I propose to add a functionality to make rez-test package unit -- -v -m myMarker valid and that would run pytest {root}/tests/unit_tests -v -m myMarker.

The immediate benefit is that it's faster and more intuitive. The second is that it can avoid mistakenly committing the modifications to the tests variable.

Note that passing custom arguments should only be possible if only one test is being run.

4. More execution options

Specifically:

• A rez-test --interactive option. Puts you into an interactive shell for the nominated test. From here you could then use...
• A rez-test --inplace option. Ie, run the tests in the current env, if it provides all the required packages. If not, give an error.

5. Variant iteration options

Many times, it makes sense to run a test on all variants of a package. Currently that doesn't happen (variants aren't iterated over at all) because there's not currently a reliable way to resolve to an explicit variant in rez. However, until such time as we have the required feature to do this, we could get pretty close, and it's probably worth doing (the code needed to do this is minimal).

Some things to consider:

• Some tests won't make sense to run over multiple variants (eg linting).
• Some tests might only make sense to run on certain variants (eg just variants requiring "maya")

How to express this? Perhaps a new variant tests field. For eg, with variant: True, the test will be run on every variant. If False/not present, it will just run on one variant (matching current behaviour). If a dict containing "requires", it will run only on variants overlapping with the given request list; furthermore, if the requires list conflicts with a variant, that variant is skipped.

For example, given:

variants = [
  ["python-3"],
  ["python-3", "maya-2018"],
  ["python-3", "maya-2019"]
]

tests = {
  "linter": {
    "command": "pylint ...",
    "variant": False
  },
  "core": {
    "command": "python {root}/run_core_unit_tests.py",
    "variant": {
      "requires": ["!maya"]
    }
  }
}

• The linter test would run just once, not per variant;
• The 'core' test would run on every variant not containing maya

[nerdvegas]

This all sounds good to me, and in fact I'll add some more rez-test- related feature requests. RE default target list. Yes I agree with this. However, it would also serve a further purpose. Right now, there's no way to configure rez to run the unit tests before performing a package release, but that's something that's been asked for multiple times. This default target list would also be used for the unit tests that get run automatically, pre-release. I think this should be a separate package attribute `default_tests (list of str)`. RE multiple tests on cli - agreed, however this feature would be disallowed in combo with custom test args. RE custom test args - agreed. To that end I see four action items here: * Add default_tests attrib * Add pre-release hook to run default tests and abort release on any failures * Add ability to run multiple tests with one cli call * Add ability to pass args to tests How does that sound? A

…

On Mon, Jul 15, 2019 at 6:17 AM Jean-Christophe Morin < ***@***.***> wrote: Hey everybody, here are some ideas regarding rez-test. At RodeoFX we are heavily relying on it and we found that it's lacking some features. There is multiple proposal in this issue because I didn't want to pollute the issue tracker with too much noise. I basically want to get some input and feedback on my proposals and see if there is any traction. We can split into different issues later if necessary. Default test/target Right now, when no test name is given to rez-test, it will run all tests. I propose to add a new built-in key in the tests dict inside a package definition. The key could be called default or default_test and it could contain a string (the name of a test or all) or a list (a list of test). The reason behind this is that sometimes you might have super slow tests (e.g complex integration tests) that you don't want to run by default, but you still want to run the other tests without having to run them one by one or specify them in the command line. I have to note that this is a little dangerous to add, since someone could already be using the name we will choose. The name of the key would have to be carefully chosen. Or we might even need to introduce a new package attribute to make sure we don't clash with what users of rez already have. Specify multiple tests to run in the command line I basically feel the need to be able to run multiple tests in one shot without running them all. For example, we have package that has 3 tests: test1, test2, test3, I want to be able to run rez-test package test1 test2 or rez-test package all Pass custom arguments when running a test My colleagues and myself always have to modify the commands of the tests while developing a package. There is multiple reasons why we do so, but mainly we like to increase the verbosity of our tests only when developing and keep the verbosity lower in our CI tests. The other common reason is to choose a custom target inside a particular test so we can iterate faster and avoid running unnecessary tests. For example, let's say we have: tests = { 'unit': { 'command': 'pytest {root}/tests/unit_tests' 'requires': 'pytest' } } where tests/unit_tests contains multiple test files and each file contains tests with markers (pytest jargon. A marker is basically a way to group the tests so that you can selectively choose which one to run in the command line). I'd be great if we could do rez-test package unit -- -v -m myMarker that would run pytest {root}/tests/unit_tests -v -m myMarker. The immediate benefit is that it's faster and more intuitive. The second is that it can avoid mistakenly committing the modifications to the tests variable. Doing this could be challenging with the two previous proposals. Because of each test uses a different tool, then we can't easily know if the additional arguments will work or not. We could simply say that we don't care since the developer probably knows what he is doing. Or we just raise if we are running multiple tests and we receive custom tests arguments. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#665>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMOUSSI6DUCN4IWTVVCBJDP7OCUZANCNFSM4IDR35YA> .

[JeanChristopheMorinRodeoFX]

Sounds good to me. Do you have any preference on the order? I would personally go with:

• Add default_tests attrib
• Add ability to pass args to tests
• Add ability to run multiple tests with one cli call
• Add pre-release hook to run default tests and abort release on any
  failures

in this order.

And in how many PRs would you like them? I think I would go with 2, one for all the rez-test inprovements and another for the hook?

[nerdvegas]

What do you think about trying this out as a REP workflow? (ie Rez Enhancement Proposal). We could: * Create the associated issue, eg REP101; * For the sub-tasks outlined above, create issues such as "REP101 - default_tests" * Once all tasks are completed, the REP ticket would be baked off into the Wiki, and the ticket closed. I think they should all be separate PRs - smaller PRs are easier to track and faster to merge. Thoughts? A

…

On Wed, Jul 17, 2019 at 9:25 AM JeanChristopheMorinRodeoFX < ***@***.***> wrote: Sounds good to me. Do you have any preference on the order? I would personally go with: - Add default_tests attrib - Add ability to pass args to tests - Add ability to run multiple tests with one cli call - Add pre-release hook to run default tests and abort release on any failures in this order. And in how many PRs would you like them? I think I would go with 2, one for all the rez-test inprovements and another for the hook? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#665>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMOUSSA74476XFOT63OW6LP7ZKGDANCNFSM4IDR35YA> .

[JeanChristopheMorinRodeoFX]

What do you think about trying this out as a REP workflow? (ie Rez
Enhancement Proposal).

Yep, that's ok for me. Should we start at 001 instead of 101 maybe? And I guess I can simply rename this issue and I'll update the description? We can keep track of the changes in the description now in Github (and same for comments/replies).

As for the PRs, if you prefer to have them separated I'm ok with that.

[nerdvegas]

001 is good. Cheers A

…

On Wed, Jul 17, 2019 at 11:21 AM JeanChristopheMorinRodeoFX < ***@***.***> wrote: What do you think about trying this out as a REP workflow? (ie Rez Enhancement Proposal). Yep, that's ok for me. Should we start at 001 instead of 101 maybe? And I guess I can simply rename this issue and I'll update the description? We can keep track of the changes in the description now in Github (and same for comments/replies). As for the PRs, if you prefer to have them separated I'm ok with that. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#665>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMOUSTFQ3N6PWEIYWCFQJTP7ZXZHANCNFSM4IDR35YA> .

[maxnbk]

Possibly related feature request to add to discussion:

Would be good to add the ability to specify that a given rez-test always runs after rez-build. Doesn't do much good when rez-releasing, but it gives a way for people to efficiently run tests against compiled code reliably/quickly, rather than needing to string together multiple commands. Perhaps for a rez-release scenario, it would be possible for a test failure to cause a release hook not to run, and perhaps attempt to scrub the built package version from disk (Can't be guaranteed to work in all cases, but probably covers all decent package possibilities where someone isn't being hacky with sudo).

[JeanChristopheMorinPerso]

Just updated the description and the title. I rephrased some sentences to make them more generic and remove mentions of RodeoFX.

@maxnbk regarding running the tests at build time, how would you like to tell rez to do this?

Perhaps for a rez-release scenario, it would be possible for a test failure to cause a release hook not to run, and perhaps attempt to scrub the built package version from disk (Can't be guaranteed to work in all cases, but probably covers all decent package possibilities where someone isn't being hacky with sudo).

I'm not too sure what you mean here. Are you asking if it's gonna deploy the code even if the hook make the release to fail or something?

[maxnbk]

test_built_package = True seems sane to me.

I mean that when I run rez-release, if the result of test_built_package (which runs the default tests as spec'd above by you) returns a success after release, then the release hook runs as normal, but in the case of a test failure (though I'm not entirely sure if a single variant failure should result in a whole package failure), the rez-released package path is erased. Basically, a way to help prevent bad releases from going out a door into the wide world.

Let's say that could be controlled by a erase_on_release_failure = True setting. I'm less interested in the second of these two, I'll say, but I have seen its utility regularly in-studio, as someone who routinely clears up a bad rez-release from disk.

[nerdvegas]

I think the pre-release hook would install the package to a temp location first no? rez-test would use that temporary install - thus if anything fails, nothing's been released yet. We should be able to avoid building twice - instead it'd be one build followed by two installs, one temp, one real (if tests pass).

[nerdvegas]

RE build time tests, this I think is actually a more general case of post-build hooks. I don't recall the details just now but I think there were some things to work out to get that to happen. In theory though I agree that hooks should also be able to be specified post-build, pre-release, post-release.

[maxnbk]

I think the pre-release hook would install the package to a temp location first no? rez-test would use that temporary install - thus if anything fails, nothing's been released yet. We should be able to avoid building twice - instead it'd be one build followed by two installs, one temp, one real (if tests pass).

Not always possible when dealing with compiled packages.

[nerdvegas]

Yeah I guess it depends on the build as to whether you can cleanly install the same build into different locations. Perhaps we'd need a new package attrib to describe this - ie whether a full rebuild is necessary for any given package install. In any case, I can't see that we could actually do a release as part of a pre-release hook(!) - even if it's cleaned up after, there's a window of time where that package existed and may get picked up (unless we also update how the `.bulding` marker files work, which is yet more complexity). I don't think we can avoid having to do a temp install in order to test.

…

On Wed, Jul 17, 2019 at 10:28 PM maxnbk ***@***.***> wrote: I think the pre-release hook would install the package to a temp location first no? rez-test would use that temporary install - thus if anything fails, nothing's been released yet. We should be able to avoid building twice - instead it'd be one build followed by two installs, one temp, one real (if tests pass). Not always possible when dealing with compiled packages. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#665>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMOUSRTXH4QERYJ7WPP2BTP74F7BANCNFSM4IDR35YA> .