Better document testing requirements for Omega PRs #304

xylar · 2025-10-27T12:52:59Z

This merge adds a "Code Testing" page to the documentation that describes running the omega_pr test suite for
each Omega PR.

It also updates the PR checklist to better describe testing. The updated version provides better links to the documentation
and a more thorough list of testing requirements.

It includes many fixes to typos in the Quick Start.

Checklist

Documentation:
- Developer's Guide has been updated
- Documentation has been built locally and changes look as expected

This page describes running the `omega_pr` test suite for each Omega PR.

The updated version provides better links to the documentation and a more thorough list of testing requirements.

xylar · 2025-10-27T12:53:33Z

A test build of the documentation can be found here:
https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/xasaydavis/test_docs/html/

xylar · 2025-10-27T12:55:00Z

@cbegeman, would you be willing to take a look at this and make sure it addresses the discussion we had 2 or 3 weeks ago about better documenting the requirement to run the omega_pr suite for each Omega PR?

xylar · 2025-10-27T12:55:19Z

@sbrus89, would you be willing to give this a look or recommend a different reviewer?

xylar · 2025-10-27T12:56:12Z

Note: I ran into problems with the manufactured_solution test documented in #303. We'll want to figure this out soon so developers have a valid baseline to work with.

cbegeman · 2025-10-27T15:34:20Z

@xylar This looks really great! One situation that could be helpful to address is how developers should proceed if they are making changes that affect the namelist/streams (i.e., make changes to mpaso_to_omega.yaml and test-specific yaml files if relevant).

sbrus89

@xylar, this is a really helpful guide. Thanks for putting it together. I just have a few minor comments.

components/omega/doc/devGuide/Testing.md

sbrus89 · 2025-10-27T16:11:41Z

components/omega/doc/devGuide/Testing.md

+CTest unit tests
+- Machine(s): <machine>[, <partition>]
+- Compiler(s): <compiler>[, ...]
+- Build type(s): <Debug|Release>[, ...]
+- Result: All tests passed | Failures: <name> (<one-line note>), ...
+
+Polaris omega_pr regression suite
+- Baseline build (-p): <path/to/develop/build>
+- Baseline workdir (-w): <path/to/baseline_omega_pr>
+- PR build (-p): <path/to/pr/build>
+- PR workdir (-w): <path/to/my_branch_omega_pr>
+- Machine/partition: <machine>[, <partition>]
+- Compiler/build type: <compiler>, <Debug|Release>
+- Result: All tests passed | Diffs/Failures: <brief summary>
+- Logs (if applicable): <path/to/polaris_omega_pr.o$JOBID>


Can we configure CTest/Polaris to produce this output in a log file, so the developer doesn't have to recreate it by hand?

Not really. It's the output from a bunch of different things.

Sorry for dismissing this yesterday. I've thought about it more and I think we can make each tool produce the output we want here, especially if we slightly alter the formatting.

I looked into this for CTest and I don't see an easy way to get it to produce this information. We could try to get the utility from Polaris to produce it, but that would require capturing the output from CTest itself and parsing it. Not at all impossible but maybe not trivial.

As for the Polaris tests, I don't think Polaris has a way to know whether Omega was build in debug or release mode but we could write out the rest of this info at the end of the omega_pr suite. I can work on that as I have time.

I implemented the remainder of this capability in E3SM-Project/polaris#387. We will need to add documentation here about where to find the file to copy/paste from once that PR has gone into Polaris.

xylar · 2025-10-27T16:17:37Z

One situation that could be helpful to address is how developers should proceed if they are making changes that affect the namelist/streams (i.e., make changes to mpaso_to_omega.yaml and test-specific yaml files if relevant).

Yes, very good point, @cbegeman.

rljacob · 2025-10-27T16:55:01Z

Do you want to try and add PR testing with github actions?

xylar · 2025-10-27T17:59:43Z

Do you want to try and add PR testing with github actions?

@rljacob, that would be a good idea but it's a pretty big project for us to take on right at the moment. I think we're going to have to deal with manual testing for a bit here.

xylar · 2025-10-28T10:22:57Z

One situation that could be helpful to address is how developers should proceed if they are making changes that affect the namelist/streams (i.e., make changes to mpaso_to_omega.yaml and test-specific yaml files if relevant).

@cbegeman, I attempted to address this in 11a2513

xylar · 2025-10-28T12:36:08Z

components/omega/doc/devGuide/Testing.md

+- Machine(s): <machine>[, <partition>]
+- Compiler(s): <compiler>[, ...]
+- Build type(s): <Debug|Release>[, ...]
+- Result: All tests passed | Failures: <name> (<one-line note>), ...


I updated the CTest utility to produce output like this in E3SM-Project/polaris#385.

grnydawn · 2025-11-04T19:51:50Z

components/omega/doc/devGuide/Testing.md

+For instructions on how to build Omega and run CTest unit tests, see
+{ref}`omega-dev-quick-start-build-test`.
+
+


This section introduces a link to build Omega, but it does not explicitly state that building the develop branch and the test branch of Omega are required steps for the subsequent instructions even though those information is provided in following sections. Adding one or two statements explaining the Omega build requirements and how they relate to the DEVELOP_BUILD and TEST_BRANCH_BUILD paths would be helpful for users trying this for the first time.

@grnydawn, thank you. I have attempted to clarify this in c0fa7a5, and also corrected some incorrect statements about what should be used for the baseline in aa1d5e3 and 3409373. Let me know if this now looks good to you. I rebuilt at:
https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/xasaydavis/test_docs/html/
but it might not stay there for long since I use that URL for test-building documentation for various purposes.

Thanks for the update. It looks good to me. I was able to successfully run the test by following the directions on the page.

xylar · 2025-11-05T08:20:03Z

@sbrus89, could you let me know if you have further comments here?

grnydawn

I successfully ran the test by following the directions on the page using the Intel compiler on Chrysalis.

There were a few failed tests, which I will summarize in a separate comment, but I don't think they block approval of this PR.

grnydawn · 2025-11-05T15:52:15Z

During testing for this PR review, I encountered the following failed test result.

To verify the b4b check, I used the same Omega commit for both the baseline and the test branch, so I expected the b4b tests to pass.

Below is the last part of the log from the test branch:

0:00:10 FAIL ocean/planar/manufactured_solution/convergence_both/default
0:03:48 FAIL ocean/spherical/icos/rotation_2d
0:00:10 FAIL ocean/spherical/icos/cosine_bell/decomp
0:00:06 FAIL ocean/spherical/icos/cosine_bell/restart
Total runtime: 0:04:15
FAIL: 4 tasks failed, see above.

and from the baseline branch:

0:00:02 FAIL ocean/planar/manufactured_solution/convergence_both/default
0:00:00 PASS ocean/spherical/icos/rotation_2d
0:00:00 PASS ocean/spherical/icos/cosine_bell/decomp
0:00:00 PASS ocean/spherical/icos/cosine_bell/restart
Total runtime: 0:00:04
FAIL: 1 task failed, see above.

I attached the full log file of the test too.

test_logs.txt

xylar · 2025-11-05T16:04:49Z

@grnydawn, it looks from the list of tests like you're using the Polaris main branch. That's totally understandable but we know that omega_pr is broken there. I have a branch to fix it here:
E3SM-Project/polaris#384
It might get merged very soon ~~but can you test with that branch in the meantime~~?

xylar · 2025-11-05T16:06:10Z

@grnydawn, also what machine and which compiler was this on?

xylar · 2025-11-05T16:07:03Z

And thank you so much for testing things out and reviewing!

grnydawn · 2025-11-05T16:10:25Z

@grnydawn, also what machine and which compiler was this on?

I ran the test using the Intel compiler on Chrysalis.

xylar · 2025-11-05T16:12:34Z

I ran the test using the Intel compiler on Chrysalis.

Okay, great! We've already tested the new branch with that configuration and it's fixed so I don't think you need to redo your testing. Thanks again for running through the process!

grnydawn · 2025-11-05T16:14:47Z

@grnydawn, it looks from the list of tests like you're using the Polaris main branch. That's totally understandable but we know that omega_pr is broken there. I have a branch to fix it here: E3SM-Project/polaris#384 It might get merged very soon but can you test with that branch in the meantime?

Sure, I will do that. By the way, what do you mean by the main branch? Was my test configuration not correct?

xylar · 2025-11-05T16:26:30Z

By the way, what do you mean by the main branch?

The main branch of the Polaris repo. omega_pr is broken there.

Was my test configuration not correct?

No, it's our fault on the Polaris side. We didn't notices several breaking changes that happened in Omega over the past couple of months so we're getting omega_pr back in working order. Hopefully, if folks start to use the workflow outlined here, we would notice this much sooner in the future.

xylar added 2 commits October 27, 2025 07:48

Add a "Code Testing" page to the documentation

1ea4802

This page describes running the `omega_pr` test suite for each Omega PR.

Update the PR checklist to better describe testing

c596e74

The updated version provides better links to the documentation and a more thorough list of testing requirements.

xylar requested review from cbegeman and sbrus89 October 27, 2025 12:53

sbrus89 reviewed Oct 27, 2025

View reviewed changes

Updates to Testing docs from review

11a2513

Fix Omega URLs in PR checklist

025d475

xylar commented Oct 28, 2025

View reviewed changes

Document new output capability of CTest util

1f5101e

xylar requested a review from grnydawn November 3, 2025 16:24

grnydawn reviewed Nov 4, 2025

View reviewed changes

xylar added 3 commits November 5, 2025 02:09

Clarify building Omega before running omega_pr suite

c0fa7a5

Correct documentation on Omega baseline

aa1d5e3

Update PR template with correct baseline

3409373

xylar removed the request for review from cbegeman November 5, 2025 08:19

grnydawn approved these changes Nov 5, 2025

View reviewed changes

		For instructions on how to build Omega and run CTest unit tests, see
		{ref}`omega-dev-quick-start-build-test`.

Better document testing requirements for Omega PRs #304

Are you sure you want to change the base?

Better document testing requirements for Omega PRs #304

Uh oh!

Conversation

xylar commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xylar commented Oct 27, 2025

Uh oh!

xylar commented Oct 27, 2025

Uh oh!

xylar commented Oct 27, 2025

Uh oh!

xylar commented Oct 27, 2025

Uh oh!

cbegeman commented Oct 27, 2025

Uh oh!

sbrus89 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xylar commented Oct 27, 2025

Uh oh!

rljacob commented Oct 27, 2025

Uh oh!

xylar commented Oct 27, 2025

Uh oh!

xylar commented Oct 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grnydawn Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xylar commented Nov 5, 2025

Uh oh!

grnydawn left a comment

Choose a reason for hiding this comment

Uh oh!

grnydawn commented Nov 5, 2025

Uh oh!

xylar commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xylar commented Nov 5, 2025

Uh oh!

xylar commented Nov 5, 2025

Uh oh!

grnydawn commented Nov 5, 2025

Uh oh!

xylar commented Nov 5, 2025

Uh oh!

grnydawn commented Nov 5, 2025

Uh oh!

xylar commented Nov 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

xylar commented Oct 27, 2025 •

edited

Loading

grnydawn Nov 4, 2025 •

edited

Loading

xylar commented Nov 5, 2025 •

edited

Loading