Skip to content

Add provenance cfg to output dir#713

Merged
forsyth2 merged 2 commits intomainfrom
issue-711
May 28, 2025
Merged

Add provenance cfg to output dir#713
forsyth2 merged 2 commits intomainfrom
issue-711

Conversation

@forsyth2
Copy link
Copy Markdown
Collaborator

@forsyth2 forsyth2 commented May 21, 2025

Summary

Objectives:

  • Add provenance cfg to output dir. This will mean the cfg used to run zppy will be immediately available to developers helping to debug users' output. Previously, developers had to ask users for a copy of their cfg.

Issue resolution:

Select one: This pull request is...

  • a bug fix: increment the patch version
  • a small improvement: increment the minor version
  • a new feature: increment the minor version
  • an incompatible (non-backwards compatible) API change: increment the major version

Small Change

  • To merge, I will use "Squash and merge". That is, this change should be a single commit.
  • Logic: I have visually inspected the entire pull request myself.
  • Pre-commit checks: All the pre-commits checks have passed.

@forsyth2 forsyth2 self-assigned this May 21, 2025
@forsyth2 forsyth2 added the semver: small improvement Small improvement (will increment patch version) label May 21, 2025
@forsyth2
Copy link
Copy Markdown
Collaborator Author

Tested with:

cd ~/ez/zppy
conda activate zppy_dev_20250407
pip install .
python tests/integration/utils.py 
zppy -c tests/integration/generated/test_min_case_nco_chrysalis.cfg
scancel -u ac.forsyth2 # cancelled, since we don't actually need the output; could have also turned on dry_run
cd /lcrc/group/e3sm/ac.forsyth2/zppy_min_case_nco_output/test-711-try2/v3.LR.historical_0051 # cd into output dir
cd post/scripts/
diff provenance.20250521_003137_133895.cfg ~/ez/zppy/tests/integration/generated/test_min_case_nco_chrysalis.cfg 
# No errors; the diffs match

@forsyth2
Copy link
Copy Markdown
Collaborator Author

@chengzhuzhang It looks there's an issue in the CI/CD workflow I need to check, but the functionality of this PR seems to work. It copies the provenance cfg to a timestamped file in the output directory, under post/scripts, e.g.: provenance.20250521_003137_133895.cfg. Let me know what you think.

@forsyth2
Copy link
Copy Markdown
Collaborator Author

Would you prefer it to be in the www rather than output? The latter already has all the sort of other provenance pieces -- e.g., .settings which specifies exactly how each parameter was set.

@forsyth2
Copy link
Copy Markdown
Collaborator Author

Testing GitHub actions with cherry-picked commit from #715.

@forsyth2 forsyth2 mentioned this pull request May 21, 2025
1 task
@forsyth2
Copy link
Copy Markdown
Collaborator Author

(Rebased off main after merging #715, which removes the cherry-picked commit since it's now in main)

@chengzhuzhang
Copy link
Copy Markdown
Collaborator

Would you prefer it to be in the www rather than output? The latter already has all the sort of other provenance pieces -- e.g., .settings which specifies exactly how each parameter was set.

thanks for working on this @forsyth2. I would prefer to have it under www, or both. We can discuss at EZ meeting about this design.

@forsyth2
Copy link
Copy Markdown
Collaborator Author

Now:

cd ~/ez/zppy
conda activate zppy_dev_20250407
pip install .
python tests/integration/utils.py 
zppy -c tests/integration/generated/test_min_case_nco_chrysalis.cfg
scancel -u ac.forsyth2 # cancelled, since we don't actually need the output; could have also turned on dry_run
cd /lcrc/group/e3sm/ac.forsyth2/zppy_min_case_nco_output/test-711-2025028/v3.LR.historical_0051 # cd into output dir
cd post/scripts/
diff provenance.20250528_190216_675289.cfg ~/ez/zppy/tests/integration/generated/test_min_case_nco_chrysalis.cfg 
# No errors; the diffs match
cd /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_min_case_nco_www/test-711-20250528 # cd into www dir
cd v3.LR.historical_0051/
diff provenance.20250528_190216_675289.cfg ~/ez/zppy/tests/integration/generated/test_min_case_nco_chrysalis.cfg

@forsyth2
Copy link
Copy Markdown
Collaborator Author

pytest tests/test_*.py passes; integration tests can be run on main as part of the weekly run.

@forsyth2
Copy link
Copy Markdown
Collaborator Author

@chengzhuzhang I decided to just add provenance to both output and www. It's not a big file, so why not? It could be useful in both places.

If you approve of the latest code, I'll merge. Thanks!

@chengzhuzhang chengzhuzhang self-requested a review May 28, 2025 21:26
Copy link
Copy Markdown
Collaborator

@chengzhuzhang chengzhuzhang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes look good to me. Thanks for the PR.

@forsyth2
Copy link
Copy Markdown
Collaborator Author

Changes approved, merging

@forsyth2 forsyth2 merged commit d04353d into main May 28, 2025
5 checks passed
@forsyth2 forsyth2 deleted the issue-711 branch May 28, 2025 22:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

semver: small improvement Small improvement (will increment patch version)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Save .cfg in output as provenance of a run

2 participants