Attribute coverage script

[caleb-sitton-inl]

---

Pull Request Description

What issue does this change request address?

#402

What are the significant changes in functionality due to this change request?

The main addition is a script that iterates through a list of tests, runs coverage with a regex filter for each individual test, and marks each run with a separate "context", which is shown in the report generated. This script is, admittedly, painfully slow and inefficient, completing a full check_py_coverage run for every test; however, the information it seeks to obtain is, on average, unlikely to change greatly in the short term. The current output of this script is the following report:
coverage_attribution_report.zip (To access report: download, unzip, and open "index" file)

Some changes were also made to the check_py_coverage.sh script. These mainly consisted of command line arguments to support attribute_coverage and debugging to deal properly with Windows filepaths.

Note: attribute_coverage is dependent on raven.developer_utils.get_coverage_tests.py. This script was updated with https://github.com/idaholab/raven/pull/2431 to support attribute_coverage. Future faulty output from attribute_coverage could be due to get_coverage_tests becoming outdated.

---

For Change Control Board: Change Request Review

The following review must be completed by an authorized member of the Change Control Board.

• 1. Review all computer code.
• 2. If any changes occur to the input syntax, there must be an accompanying change to the user manual and xsd schema. If the input syntax change deprecates existing input files, a conversion script needs to be added (see Conversion Scripts).
• 3. Make sure the Python code and commenting standards are respected (camelBack, etc.) - See on the wiki for details.
• 4. Automated Tests should pass.
• 5. If significant functionality is added, there must be tests added to check this. Tests should cover all possible options. Multiple short tests are preferred over one large tes.
• 6. If the change modifies or adds a requirement or a requirement based test case, the Change Control Board's Chair or designee also needs to approve the change. The requirements and the requirements test shall be in sync.
• 7. The merge request must reference an issue. If the issue is closed, the issue close checklist shall be done.
• 8. If an analytic test is changed/added, the the analytic documentation must be updated/added.
• 9. If any test used as a basis for documentation examples have been changed, the associated documentation must be reviewed and assured the text matches the example.

[GabrielSoto-INL]

Can confirm that it takes a while to complete the attribute_coverage script run... Ran it locally in about 2 hours. The new context feature is very cool, really useful addition to the coverage tests. It feels like this is the final step to provide a complete coverage report.

I was wondering though if the scripts could:

• accept the -j argument and feed it to rook which allows multiple tests to run simultaneously
• run heavy tests as well, using the --heavy argument

Do you think this could be implemented? If so, do you think it could be included in this PR or might be best as a separate PR?

[GabrielSoto-INL]

I wonder if additional arguments could be added here for some additional rook args (e.g., --j to specify the number of simultaneous/parallel jobs which might speed up the tests or --heavy to also include heavy tests).

[caleb-sitton-inl]

@GabrielSoto-INL When the command line arguments to check_py_coverage are read in, any unrecognized arguments are placed in the ARGS array, which is expanded and provided as a string of command line arguments to run_tests in line 54. Effectively, any arguments that are not recognized by check_py_coverage are fed unaltered into run_tests. This is the same way that run_tests handles command line arguments intended for rook. I've specifically gotten in the middle of this process for the --re argument in order to create a default if no value is provided by the user, but the rest should just work like they would if put into run_tests directly.

[caleb-sitton-inl]

@GabrielSoto-INL I could easily modify attribute_coverage to forward command line args to check_py_coverage, which should in turn forward them to run_tests. Would that be sufficient for what you're thinking? Since each test is a separate rook run in attribute_coverage, I'm not sure how much faster it would be to run rook with multiple jobs, but both coverage scripts should probably be forwarding command line args anyway.

[GabrielSoto-INL]

I think that might be worth it, at the very least to speed things up. would also allow the option to run the heavy tests as well

[GabrielSoto-INL]

Perfect, with that last change I was able to locally run the attribute coverage script with an additional -f4 command. I did some very crude timings, and it improves the computation time:

attribute_coverage.sh	attribute_coverage.sh -f4
156m 47.616s	58m 9.033s

Changes LGTM, good to merge