Skip to content

Add WCRP compliance check pipeline using cc-plugin-wcrp#67

Merged
ilaflott merged 4 commits intomainfrom
copilot/check-outputs-against-cc-plugin-wcrp
Apr 14, 2026
Merged

Add WCRP compliance check pipeline using cc-plugin-wcrp#67
ilaflott merged 4 commits intomainfrom
copilot/check-outputs-against-cc-plugin-wcrp

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 14, 2026

Describe your changes

Adds a GitHub Actions workflow (wcrp_compliance_check.yml) that runs cc-plugin-wcrp WCRP compliance checks against all NetCDF outputs generated by the test suite.

Key changes

  • Workflow (wcrp_compliance_check.yml): New CI pipeline that runs on PRs and workflow_dispatch. Uses the repository's environment.yaml for conda dependencies. Runs all output-generating tests in a single pytest invocation, gathers the CMOR outputs, then validates them with compliance-checker --test=wcrp_cmip6.
  • Gather script (gather_test_outputs.sh): Reusable script that collects NetCDF test outputs from hardcoded output directories and the pytest --basetemp directory, categorizes them as CMIP6 or CMIP7, and skips uncategorized files (e.g. statics) that are not expected to be compliant.

Pipeline structure

  1. Test execution (most fragile step, runs first): All four output-generating test files run in a single pytest invocation with --basetemp=tmp/pytest_basetemp to preserve temporary artifacts:
    • test_cmor_run_subtool.py — outputs to fremorizer/tests/test_files/outdir
    • test_cmor_run_subtool_further_examples.py — outputs to fremorizer/tests/test_files/outdir_ppan_only
    • test_cmor_yamler_subtool.py::test_cmor_yaml_subtool_dry_run_false — outputs to basetemp
    • test_cli.py — outputs to basetemp (CMIP6 and CMIP7 run cases)
  2. Output gathering: Gather script collects and categorizes all CMOR outputs (14 files: 12 CMIP6, 2 CMIP7), skipping non-compliant statics files.
  3. cc-plugin-wcrp install (deferred behind file_count gate): Only installed if outputs were gathered.
  4. Compliance checks: Runs wcrp_cmip6 checks on CMIP6 and CMIP7 outputs, generating text and JSON reports.
  5. Artifacts: Uploads test outputs and compliance reports.

Bug fixes

  • Fixed bash set -e + ((var++)) from 0 arithmetic gotcha in gather_test_outputs.sh — replaced with var=$((var + 1)).
  • Single pytest invocation prevents --basetemp from being wiped between separate pytest calls.

Checklist before requesting a review

  • I ran my code
  • I tried to make my code readable
  • I tried to comment my code
  • I wrote a new test, if applicable
  • I wrote new instructions/documentation, if applicable
  • I ran pytest and inspected it's output
  • I ran pylint and attempted to implement some of it's feedback
  • No print statements; all user-facing info uses logging module

Copilot AI linked an issue Apr 14, 2026 that may be closed by this pull request
@ilaflott ilaflott force-pushed the copilot/check-outputs-against-cc-plugin-wcrp branch from 7a96095 to 4b85bc8 Compare April 14, 2026 15:22
Copilot AI changed the title [WIP] Check pipeline outputs against WCRP cc-plugin Add WCRP compliance check pipeline using cc-plugin-wcrp Apr 14, 2026
Copilot AI requested a review from ilaflott April 14, 2026 15:23
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Apr 14, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

@@           Coverage Diff           @@
##             main      #67   +/-   ##
=======================================
  Coverage   93.78%   93.78%           
=======================================
  Files           9        9           
  Lines        1206     1206           
=======================================
  Hits         1131     1131           
  Misses         75       75           

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4f590f8...e3d729d. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ilaflott ilaflott force-pushed the copilot/check-outputs-against-cc-plugin-wcrp branch from dfdb851 to 5231a36 Compare April 14, 2026 16:13
@ilaflott ilaflott marked this pull request as ready for review April 14, 2026 16:22
Copy link
Copy Markdown
Member

@ilaflott ilaflott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sure all the testing outputs are being collected, see comments

Comment thread .github/workflows/wcrp_compliance_check.yml
Comment thread .github/workflows/wcrp_compliance_check.yml Outdated
Comment thread .github/workflows/wcrp_compliance_check.yml Outdated
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 14, 2026

make sure all the testing outputs are being collected, see comments

All addressed in e3d729d — now collecting outputs from all four test files (run_subtool, further_examples, yamler dry_run_false, CLI). 14 total CMOR outputs gathered (12 CMIP6, 2 CMIP7), 2 non-compliant files skipped.

Copilot AI requested a review from ilaflott April 14, 2026 16:48
@ilaflott
Copy link
Copy Markdown
Member

awyiss

@ilaflott ilaflott merged commit ed293a7 into main Apr 14, 2026
12 checks passed
@ilaflott ilaflott deleted the copilot/check-outputs-against-cc-plugin-wcrp branch April 16, 2026 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pipeline: check outputs against https://github.com/ESGF/cc-plugin-wcrp

3 participants