Skip to content

Port subset of em_real tests from regression tests #2215

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: develop
Choose a base branch
from

Conversation

islas
Copy link
Collaborator

@islas islas commented Apr 29, 2025

TYPE: enhancement

KEYWORDS: testing, em_real

SOURCE: internal

DESCRIPTION OF CHANGES:
Problem:
In the continued effort to port the regression tests to a usable format outside a single container environment, a generalized testing framework is now in place. However, this currently lacks the full set of tests.

Solution:
This implements a subset of the em_real tests, notably the tests that are stable, bitwise reproducible, and do not rely on in situ modification of configuration.

TESTS CONDUCTED:

  1. Testing was done on Derecho and local machine to recreate the running of the test namelists, gather the testing data, and coordinate the tests. Through process of elimination, tests were removed that did not provide consistent pass results. This is not ideal for robust testing, but demonstrates the fragility of the tests outside a pristine containerized environment.

@islas islas requested review from a team as code owners April 29, 2025 23:12
@islas islas added the em_real-tests Run em_real tests label Apr 29, 2025
@github-actions github-actions bot removed the em_real-tests Run em_real tests label Apr 29, 2025
@islas islas added the em_real-tests Run em_real tests label Apr 29, 2025
@github-actions github-actions bot removed the em_real-tests Run em_real tests label Apr 30, 2025
@weiwangncar
Copy link
Collaborator

The regression test results:

Test Type              | Expected  | Received |  Failed
= = = = = = = = = = = = = = = = = = = = = = = =  = = = =
Number of Tests        : 23           24
Number of Builds       : 60           57
Number of Simulations  : 158           150        0
Number of Comparisons  : 95           86        0

Failed Simulations are: 
None
Which comparisons are not bit-for-bit: 
None




################################################################################
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kkeene44 or @weiwangncar (who ever is more familiar with the Scala tests) I think it would be good for someone to spot-check this to make sure I have the equivalent logic being used in Scala tests to run the namelists.

The analogous info from large test matrix found in the Scala tests can be found in the .ci/wrf_em_real_tests.jsonc file.

@weiwangncar weiwangncar changed the base branch from develop to release-v4.7.1 May 30, 2025 18:26
@weiwangncar weiwangncar requested a review from a team as a code owner May 30, 2025 18:26
@weiwangncar weiwangncar changed the base branch from release-v4.7.1 to develop May 30, 2025 18:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants