Write output useful for Omega PRs

[xylar]

This merge adds the capability to write out a file like:

$ cat omega_pr_output_for_pr.log 
Polaris omega_pr suite
- PR build: /gpfs/fs1/home/ac.xylar/e3sm_work/polaris/fix-omega-manufatured-solution/build_omega/build_chrysalis_intel
- PR workdir: /lcrc/group/e3sm/ac.xylar/polaris_0.9/chrysalis/test_20251028/omega_pr_better_provenance
- Machine: chrysalis
- Partition: compute
- Compiler: intel
- Log: /lcrc/group/e3sm/ac.xylar/polaris_0.9/chrysalis/test_20251028/omega_pr_better_provenance/polaris_omega_pr.o995049
- Result:
  - Failures (1 of 4):
    - ocean/spherical/icos/cosine_bell/restart
- Build type: <Debug|Release>

that is useful for copying and pasting into an Omega PR to provide information on omega_pr testing. It works for any Polaris suite so could be used for MPAS-Ocean testing as well or other supported components in the future.

To support the capability, this merge also adds more robust output to the provenance file during setup. It overwrites the this file rather than appending to it because the redundancy is more confusing than helpful with appending.

Checklist

• Testing comment in the PR documents testing used to verify the changes

[xylar]

@sbrus89, this has been rebased and is now ready for you to have a look.

[sbrus89]

@xylar, I tested this on Perlmutter and it worked great. Thanks for adding this!

Polaris omega_pr suite

• PR build: /global/cfs/cdirs/e3sm/sbrus/polaris_worktrees/omega_pr_output/e3sm_submodules/Omega/components/omega/build
• PR workdir: /pscratch/sd/s/sbrus/polaris_omega_pr_output
• Machine: pm-cpu
• Compiler: gnu
• Build type: <Debug|Release>
• Log: /pscratch/sd/s/sbrus/polaris_omega_pr_output/polaris_omega_pr.o45055520
• Result: All tests passed

Just two comments:

• I ran this in an interactive session, so the log file listed in the output doesn't actually exist. I'm not sure how easy it is to query if a job is interactive or not?
• Is it possible to add the baseline directory for runs that do baseline comparison?

[xylar]

I ran this in an interactive session, so the log file listed in the output doesn't actually exist. I'm not sure how easy it is to query if a job is interactive or not?

I can add that but I think it isn't a good idea to run the suite interactively for purposes of posting on an Omega PR. We do want that log output available if needed.

But I do also understand that Omega is super flaky on Perlmutter right now so an interactive job is the only reasonable option.

[xylar]

Is it possible to add the baseline directory for runs that do baseline comparison?

That should already be present. That was a lot of the extra work I had to add to the provenance. Did you try that and did it not show up? I'm pretty sure I tested and it worked for me but I'll try again to be sure.

Update clearly I haven't done any recent baseline comparisons because I'm seeing #395

[xylar]

I tested this along with #398 and when I run omega_pr once to create a baseline and then again with a comparison branch, I see the following as expected.

Polaris omega_pr suite

• Baseline workdir: /lcrc/group/e3sm/ac.xylar/polaris_0.9/chrysalis/test_20251112/write_output_for_omega_prs_baseline
• Baseline build: /gpfs/fs1/home/ac.xylar/e3sm_work/polaris/write-ouput-for-omega-prs/build_omega/build_chrysalis_intel
• PR build: /gpfs/fs1/home/ac.xylar/e3sm_work/polaris/write-ouput-for-omega-prs/build_omega/build_chrysalis_intel
• PR workdir: /lcrc/group/e3sm/ac.xylar/polaris_0.9/chrysalis/test_20251112/write_output_for_omega_prs_check
• Machine: chrysalis
• Partition: compute
• Compiler: intel
• Build type: <Debug|Release>
• Log: /lcrc/group/e3sm/ac.xylar/polaris_0.9/chrysalis/test_20251112/write_output_for_omega_prs_check/polaris_omega_pr.o1003391
• Result: All tests passed

[xylar]

@sbrus89, I believe I have addressed both of your requests. I confirmed that the baseline information is, indeed, included when I run a comparison against a baseline. (Note that you need the baseline to also be created with this Polaris branch or it won't have the necessary provenance.) I also do not include the path to the log file if it doesn't exist.