Replies: 13 comments 42 replies
-
|
@forsyth2, I think the image diffs in the 3 pdfs you pointed to in the 3 mpas-analysis links above all look as expected (or are small differences that could be accounted for by a change in matplotlib or similar). Notably, there is a colorbar glitch in several old sea-ice images that has been fixed in the new version of mpas-analysis as expected. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Ok, that's good news, thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Chrysaliszppy-interfacesTesting stepscd ~/ez/zppy-interfaces
git status
# branch: main
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zppy-interfaces/commits/main
# Bump to 0.2.0rc2 => 5a88374fa18782afe8de4f015d0b4504c75e5d06
# e3sm_unified_1_12_0rc2
git checkout -b test_unified_1.12.0rc2_chrysalis 5a88374fa18782afe8de4f015d0b4504c75e5d06
git log
# Good, last commit: Bump to 0.2.0rc2 (#43)
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc2_chrysalis.sh
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 33.69s
pytest tests/unit/pcmdi_diags/test_*.py
# 7 passed, 1 warning in 22.28sThe warning has been noted at PCMDI/pcmdi_metrics#1330 (comment). Success zppyFor reference notes: Testing steps# We're specifically testing E3SM Unified,
# so we don't have to worry about making dev environments for e3sm_diags or zppy-interfaces.
# Step 2 (of https://docs.e3sm.org/zppy/_build/html/main/dev_guide/release_testing.html)
cd ~/ez/zppy
# Step 3
git status
# branch: main
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zppy/commits/main
# Bump to 3.1.0rc2 => 0f24ccd7e4e78330691b905481b791e2b3018a60
# e3sm_unified_1_12_0rc2
git checkout -b test_unified_1.12.0rc2_chrysalis 0f24ccd7e4e78330691b905481b791e2b3018a60
git log
# Good, last commit: Bump to 3.1.0rc2 (#750)
# Step 4
# Previously, we had been testing with one e3sm_diags subtask using a dev environment,
# to test environment_commands overriding, but we have discontinued that.
# Step 5
# Edit tests/integration/utils.py:
#
# TEST_SPECIFICS: Dict[str, Any] = {
# "diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc2_chrysalis.sh",
# "global_time_series_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc2_chrysalis.sh",
# "pcmdi_diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc2_chrysalis.sh",
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": [
# "e3sm_diags",
# "mpas_analysis",
# "global_time_series",
# "ilamb",
# "pcmdi_diags",
# ],
# "unique_id": "test_e3sm_unified_1.12.0rc2_20251020",
# }
#
# "environment_commands_test": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc2_chrysalis.sh",
#
# generate_cfgs(unified_testing=True, dry_run=False)
# Step 6
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc2_chrysalis.sh
# Step 7
pytest tests/test_*.py
# 35 passed in 1.68s
# Step 8
python tests/integration/utils.py
# Specifically don't want to do these steps, because we're not in a dev environment or making code changes
# pre-commit run --all-files
# python -m pip install .
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
# RUNNING 16:14 10/20
# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc2_20251020/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc2_20251020/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Now, run bundles part 2
cd ~/ez/zppy
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
# Review finished runs
### v2 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_e3sm_unified_1.12.0rc2_20251020/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/test_e3sm_unified_1.12.0rc2_20251020/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc2_20251020/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/test_e3sm_unified_1.12.0rc2_20251020/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
### bundles ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc2_20251020/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc2_20251020/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd ~/ez/zppy
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 1.49s, diffs appear to be expected based on recent changes on `main`
pytest tests/integration/test_campaign.py
# 6 failed in 2.29s, same as above
pytest tests/integration/test_defaults.py
# 1 failed in 0.45s, same as above
pytest tests/integration/test_last_year.py
# 1 passed in 0.32s
pytest tests/integration/test_bundles.py
# 2 passed in 0.19s
emacs tests/integration/utils.py
# Comment out pcmdi_diags since we don't have expected results yet
# #"pcmdi_diags",
pytest tests/integration/test_images.py
# FAILED tests/integration/test_images.py::test_images - assert 854 == 759
# 1 failed in 3840.61s (1:04:00)
cat test_images_summary.mdComplete resultsDiff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
Concise results
Notes:
|
Beta Was this translation helpful? Give feedback.
1 reply
-
Updating Expected results# First, let's copy over the old expected results
cd /lcrc/group/e3sm/public_html/
cp -r zppy_test_resources/ zppy_test_resources_previous/expected_results_until_20251020 # < 15 minutes
# Check old expected results were copied over correctly, so we have a record:
ls zppy_test_resources_previous/expected_results_until_20251020
# Now, make new expected results for all the tests
cd ~/ez/zppy
git status
# Reminder: only one of the update scripts has a diff:
# tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# That file uses multiple `#expand` parameters.
# The others only use `#expand` for the expected dir, which is consistent.
# template_update_{bash_generation, campaign, defaults}_expected_files.sh
# Let's update the simpler tests' results first:
ls tests/integration/test_*.py
# tests/integration/test_bash_generation.py tests/integration/test_defaults.py
# tests/integration/test_bundles.py tests/integration/test_images.py
# tests/integration/test_campaign.py tests/integration/test_last_year.py
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# 1 passed in 1.73s
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# 6 passed in 1.81s
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# 1 passed in 0.36s
# test_last_year doesn't need an update script because its cfg is stand-alone:
pytest -vv tests/integration/test_last_year.py
# 1 passed in 0.32s
# Now update the expected files for the major tests:
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh # < 20 minutes
# For these, we rerun the tests in distinct commands:
pytest tests/integration/test_bundles.py
# 2 passed in 0.94s
# Make some updates to test_images.py
# Test the intersection of asked-for tasks and available tasks for each test.
pytest tests/integration/test_images.py
# 1 passed in 3445.43s (0:57:25)
cat test_images_summary.mdDiff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
No errors to report, but interestingly Setting up PR for test updatescd ~/ez/zppy
git status
rm -rf www/
# Reset tests/integration/utils.py
python tests/integration/utils.py
git status
# Only remaining changes are in tests/integration/test_images.py
git branch -m update_test_images
git add -A
pre-commit run --all-files # Doesn't work in Unified environment; must be in dev environment
git commit -m "Update test_images" --no-verify
git push upstream update_test_images#751 adds some improvements to |
Beta Was this translation helpful? Give feedback.
-
|
Compyzppy-interfaces (rc2)Testing stepscd ~/ez/zppy-interfaces
git status
# branch: main
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zppy-interfaces/commits/main
# Bump to 0.2.0rc2 => 5a88374fa18782afe8de4f015d0b4504c75e5d06
# e3sm_unified_1_12_0rc2
git checkout -b test_unified_1.12.0rc2_compy 5a88374fa18782afe8de4f015d0b4504c75e5d06
git log
# Good, last commit: Bump to 0.2.0rc2 (#43)
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc2_compy.sh
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 17.07s
pytest tests/unit/pcmdi_diags/test_*.py
# 7 passed, 1 warning in 11.05sThe warning has been noted at PCMDI/pcmdi_metrics#1330 (comment). Success zppyFor reference notes:
Testing steps# We're specifically testing E3SM Unified,
# so we don't have to worry about making dev environments for e3sm_diags or zppy-interfaces.
# Step 2 (of https://docs.e3sm.org/zppy/_build/html/main/dev_guide/release_testing.html)
cd ~/ez/zppy
# Step 3
git status
# branch: test_unified_1.11.1_try2
# Handle uncommitted changes
git add -A
git commit -m "Testing" --no-verify
git status
# branch: test_unified_1.11.1_try2
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zppy/commits/main
# Bump to 3.1.0rc2 => 0f24ccd7e4e78330691b905481b791e2b3018a60
# e3sm_unified_1_12_0rc2
git checkout -b test_unified_1.12.0rc2_compy 0f24ccd7e4e78330691b905481b791e2b3018a60
git log
# Good, last commit: Bump to 3.1.0rc2 (#750)
# Step 4
# Previously, we had been testing with one e3sm_diags subtask using a dev environment,
# to test environment_commands overriding, but we have discontinued that.
# Step 5
# Let's add the testing improvements from https://github.com/E3SM-Project/zppy/pull/751
git cherry-pick 93cdf325f03d908e66866ca523951dfa032c77d2
git log
# Good, the commit was added
# Edit tests/integration/utils.py:
#
# TEST_SPECIFICS: Dict[str, Any] = {
# "diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc2_compy.sh",
# "global_time_series_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc2_compy.sh",
# "pcmdi_diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc2_compy.sh",
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": [
# "e3sm_diags",
# "mpas_analysis",
# "global_time_series",
# "ilamb",
# "pcmdi_diags",
# ],
# "unique_id": "test_e3sm_unified_1.12.0rc2_compy_20251023",
# }
#
# "environment_commands_test": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc2_compy.sh",
#
# generate_cfgs(unified_testing=True, dry_run=False)
# Step 6
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc2_compy.sh
# Step 7
pytest tests/test_*.py
# 35 passed in 0.99s
# Step 8
python tests/integration/utils.py
# Specifically don't want to do these steps, because we're not in a dev environment or making code changes
# pre-commit run --all-files
# python -m pip install .
zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_compy.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
# b'sbatch: error: invalid partition specified: compute\nsbatch: error: Batch job submission failed: Invalid partition name specified\n'
# RuntimeError: Problem submitting script /compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc2_compy_20251023/v3.LR.historical_0051/post/scripts/global_time_series_all_lnd_var_viewer_1985-1995.bash
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_compy.cfg
# Started 10/23 11:44
git grep -n compute tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
# tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg:291: partition = "compute"
emacs tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
emacs tests/integration/template_weekly_comprehensive_v3.cfg
# [[ all_lnd_var_viewer ]]
# # 1. plot ALL land variables
# make_viewer = True
# partition = "#expand partition_long#"
python tests/integration/utils.py
git diff
# - partition = "compute"
# + partition = "slurm"
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
sq | wc -l
# 11:50 => 108
# Picking back up 10/24
sq # No remaining jobs
cd ~/ez/zppy
# Figure out what the output paths are
git grep "output =" tests/integration/generated/test_weekly_*_compy.cfg | cat
# tests/integration/generated/test_weekly_bundles_compy.cfg:output = "/compyfs/fors729/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc2_compy_20251023/v3.LR.historical_0051"
# tests/integration/generated/test_weekly_comprehensive_v2_compy.cfg:output = "/compyfs/fors729/zppy_weekly_comprehensive_v2_output/test_e3sm_unified_1.12.0rc2_compy_20251023/v2.LR.historical_0201"
# tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg:output = "/compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc2_compy_20251023/v3.LR.historical_0051"
# tests/integration/generated/test_weekly_legacy_3.0.0_bundles_compy.cfg:output = "/compyfs/fors729/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc2_compy_20251023/v3.LR.historical_0051"
# tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_compy.cfg:output = "/compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/test_e3sm_unified_1.12.0rc2_compy_20251023/v2.LR.historical_0201"
# tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_compy.cfg:output = "/compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/test_e3sm_unified_1.12.0rc2_compy_20251023/v3.LR.historical_0051"
# Check on bundles status
cd /compyfs/fors729/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc2_compy_20251023/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# bundle1.status:ERROR
# bundle2.status:ERROR
# climo_atm_monthly_180x360_aave_1985-1986.status:ERROR (3)
# climo_atm_monthly_180x360_aave_1987-1988.status:ERROR (3)
# climo_atm_monthly_diurnal_8xdaily_180x360_aave_1985-1986.status:ERROR (1)
# climo_atm_monthly_diurnal_8xdaily_180x360_aave_1987-1988.status:ERROR (1)
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1985-1986.status:ERROR (1)
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.status:ERROR (1)
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
# e3sm_to_cmip_land_monthly_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_land_monthly_1987-1988-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (1)
# ts_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (1)
# ts_atm_monthly_glb_1985-1989-0005.status:ERROR (1)
# ts_atm_monthly_glb_1990-1994-0005.status:ERROR (1)
# ts_land_monthly_1985-1986-0002.status:ERROR (1)
# ts_land_monthly_1987-1988-0002.status:ERROR (1)
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc2_compy_20251023/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# bundle1.status:ERROR
# bundle2.status:ERROR
# climo_atm_monthly_180x360_aave_1985-1986.status:ERROR (3)
# climo_atm_monthly_180x360_aave_1987-1988.status:ERROR (3)
# climo_atm_monthly_diurnal_8xdaily_180x360_aave_1985-1986.status:ERROR (1)
# climo_atm_monthly_diurnal_8xdaily_180x360_aave_1987-1988.status:ERROR (1)
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1985-1986.status:ERROR (1)
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.status:ERROR (1)
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
# e3sm_to_cmip_land_monthly_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_land_monthly_1987-1988-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (1)
# ts_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (1)
# ts_atm_monthly_glb_1985-1989-0005.status:ERROR (1)
# ts_atm_monthly_glb_1990-1994-0005.status:ERROR (1)
# ts_land_monthly_1985-1986-0002.status:ERROR (1)
# ts_land_monthly_1987-1988-0002.status:ERROR (1)
# Now, run bundles part 2
# # Lots of errors, so we can't proceed to this block:
# cd ~/ez/zppy
# zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
# zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_compy.cfg
# Review finished runs
### v2 ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/test_e3sm_unified_1.12.0rc2_compy_20251023/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:WAITING 704012
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.status:WAITING 704013
# e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.status:WAITING 704006
# e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.status:WAITING 704007
# e3sm_to_cmip_land_monthly_1980-1981-0002.status:WAITING 704008
# e3sm_to_cmip_land_monthly_1982-1983-0002.status:WAITING 704009
# global_time_series_1980-1990.status:WAITING 704017
# ilamb_1980-1981.status:WAITING 704018
# ilamb_1982-1983.status:WAITING 704019
# ts_atm_daily_180x360_aave_1980-1981-0002.status:ERROR (2)
# ts_atm_daily_180x360_aave_1982-1983-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1980-1981-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1982-1983-0002.status:ERROR (2)
# ts_atm_monthly_glb_1980-1984-0005.status:ERROR (2)
# ts_atm_monthly_glb_1985-1989-0005.status:ERROR (2)
# ts_land_monthly_1980-1981-0002.status:ERROR (2)
# ts_land_monthly_1982-1983-0002.status:ERROR (2)
# ts_lnd_monthly_glb_1980-1984-0005.status:ERROR (2)
# ts_lnd_monthly_glb_1985-1989-0005.status:ERROR (2)
# ts_rof_monthly_1980-1981-0002.status:ERROR (2)
# ts_rof_monthly_1982-1983-0002.status:ERROR (2)
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/test_e3sm_unified_1.12.0rc2_compy_20251023/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:WAITING 704085
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.status:WAITING 704086
# e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.status:WAITING 704079
# e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.status:WAITING 704080
# e3sm_to_cmip_land_monthly_1980-1981-0002.status:WAITING 704081
# e3sm_to_cmip_land_monthly_1982-1983-0002.status:WAITING 704082
# global_time_series_1980-1990.status:WAITING 704090
# ilamb_1980-1981.status:WAITING 704091
# ilamb_1982-1983.status:WAITING 704092
# ts_atm_daily_180x360_aave_1980-1981-0002.status:ERROR (2)
# ts_atm_daily_180x360_aave_1982-1983-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1980-1981-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1982-1983-0002.status:ERROR (2)
# ts_atm_monthly_glb_1980-1984-0005.status:ERROR (2)
# ts_atm_monthly_glb_1985-1989-0005.status:ERROR (2)
# ts_land_monthly_1980-1981-0002.status:ERROR (2)
# ts_land_monthly_1982-1983-0002.status:ERROR (2)
# ts_lnd_monthly_glb_1980-1984-0005.status:ERROR (2)
# ts_lnd_monthly_glb_1985-1989-0005.status:ERROR (2)
# ts_rof_monthly_1980-1981-0002.status:ERROR (2)
# ts_rof_monthly_1982-1983-0002.status:ERROR (2)
### v3 ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc2_compy_20251023/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# climo_atm_monthly_180x360_aave_1985-1986.status:ERROR (3)
# climo_atm_monthly_180x360_aave_1987-1988.status:ERROR (3)
# climo_atm_monthly_diurnal_8xdaily_180x360_aave_1985-1986.status:ERROR (1)
# climo_atm_monthly_diurnal_8xdaily_180x360_aave_1987-1988.status:ERROR (1)
# climo_land_monthly_climo_1985-1986.status:ERROR (3)
# climo_land_monthly_climo_1987-1988.status:ERROR (3)
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.status:WAITING 704048
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.status:WAITING 704049
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.status:WAITING 704050
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:WAITING 704041
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:WAITING 704042
# e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.status:WAITING 704043
# e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.status:WAITING 704044
# e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.status:WAITING 704045
# e3sm_to_cmip_land_monthly_1985-1986-0002.status:WAITING 704046
# e3sm_to_cmip_land_monthly_1987-1988-0002.status:WAITING 704047
# global_time_series_all_lnd_var_viewer_1985-1995.status:WAITING 704137
# global_time_series_classic_original_8_no_ocn_1985-1995.status:WAITING 704138
# global_time_series_classic_pdf_both_1985-1995.status:WAITING 704056
# global_time_series_classic_pdf_component_1985-1995.status:WAITING 704058
# global_time_series_classic_pdf_original_1985-1995.status:WAITING 704057
# global_time_series_viewer_both_1985-1995.status:WAITING 704053
# global_time_series_viewer_component_1985-1995.status:WAITING 704055
# global_time_series_viewer_original_1985-1995.status:WAITING 704054
# ilamb_1985-1988.status:WAITING 704139
# mpas_analysis_ts_1985-1989_climo_1985-1989.status:ERROR (2)
# mpas_analysis_ts_1985-1995_climo_1990-1995.status:WAITING 704052
# pcmdi_diags_mean_climate_model_vs_obs_1985-1994.status:WAITING 704140
# pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 704143
# pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.status:WAITING 704142
# pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.status:WAITING 704141
# ts_atm_daily_180x360_aave_1985-1986-0002.status:ERROR (1)
# ts_atm_daily_180x360_aave_1987-1988-0002.status:ERROR (1)
# ts_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (1)
# ts_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (1)
# ts_atm_monthly_180x360_aave_1989-1990-0002.status:ERROR (1)
# ts_atm_monthly_180x360_aave_1991-1992-0002.status:ERROR (1)
# ts_atm_monthly_180x360_aave_1993-1994-0002.status:ERROR (1)
# ts_atm_monthly_glb_1985-1989-0005.status:ERROR (1)
# ts_atm_monthly_glb_1990-1994-0005.status:ERROR (1)
# ts_land_monthly_1985-1986-0002.status:ERROR (1)
# ts_land_monthly_1987-1988-0002.status:ERROR (1)
# ts_lnd_monthly_glb_1985-1989-0005.status:ERROR (1)
# ts_lnd_monthly_glb_1990-1994-0005.status:ERROR (1)
# ts_rof_monthly_1985-1986-0002.status:ERROR (1)
# ts_rof_monthly_1987-1988-0002.status:ERROR (1)
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/test_e3sm_unified_1.12.0rc2_compy_20251023/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# climo_atm_monthly_180x360_aave_1985-1986.status:ERROR (3)
# climo_atm_monthly_180x360_aave_1987-1988.status:ERROR (3)
# climo_atm_monthly_diurnal_8xdaily_180x360_aave_1985-1986.status:ERROR (1)
# climo_atm_monthly_diurnal_8xdaily_180x360_aave_1987-1988.status:ERROR (1)
# climo_land_monthly_climo_1985-1986.status:ERROR (3)
# climo_land_monthly_climo_1987-1988.status:ERROR (3)
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.status:WAITING 704115
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.status:WAITING 704116
# e3sm_diags_lnd_monthly_mvm_lnd_model_vs_model_1987-1988_vs_1985-1986.status:WAITING 704117
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:WAITING 704111
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:WAITING 704112
# e3sm_to_cmip_land_monthly_1985-1986-0002.status:WAITING 704113
# e3sm_to_cmip_land_monthly_1987-1988-0002.status:WAITING 704114
# global_time_series_1985-1995.status:WAITING 704120
# ilamb_1985-1988.status:WAITING 704121
# mpas_analysis_ts_1985-1989_climo_1985-1989.status:ERROR (2)
# mpas_analysis_ts_1985-1995_climo_1990-1995.status:WAITING 704119
# ts_atm_daily_180x360_aave_1985-1986-0002.status:ERROR (1)
# ts_atm_daily_180x360_aave_1987-1988-0002.status:ERROR (1)
# ts_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (1)
# ts_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (1)
# ts_atm_monthly_glb_1985-1989-0005.status:ERROR (1)
# ts_atm_monthly_glb_1990-1994-0005.status:ERROR (1)
# ts_land_monthly_1985-1986-0002.status:ERROR (1)
# ts_land_monthly_1987-1988-0002.status:ERROR (1)
# ts_lnd_monthly_glb_1985-1989-0005.status:ERROR (1)
# ts_lnd_monthly_glb_1990-1994-0005.status:ERROR (1)
# ts_rof_monthly_1985-1986-0002.status:ERROR (1)
# ts_rof_monthly_1987-1988-0002.status:ERROR (1)
# We definitely can't proceed with the test suite itself.
grep -in error *.o*
# Lots of files not being found
ls /compyfs/fors729//E3SMv3/v3.LR.historical_0051/
# No such file or directory
ls /compyfs/fors729/E3SMv2/v2.LR.historical_0201/
#archive build case_scripts init mpas-ocean mpas-seaice post run
# So... the v2 data exists
cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/test_e3sm_unified_1.12.0rc2_compy_20251023/v2.LR.historical_0201/post/scripts
grep -in error *.o*
# This has a variety of errors
# A lot of variables not being found
# Let's look at https://github.com/E3SM-Project/zppy/issues/671
ls /compyfs/fors729/zppy_test_data/E3SMv3/v3.LR.historical_0051
# archive run
ls /compyfs/fors729/zppy_test_data/E3SMv2
# ls: cannot access /compyfs/fors729/zppy_test_data/E3SMv2: No such file or directory
cd ~/ez/zppy
emacs tests/integration/utils.py
# "user_input_v2": "/compyfs/fors729/",
# "user_input_v3": "/compyfs/fors729/zppy_test_data",
# But that wouldn't fix the v2 errors.
# Let's check if we have any fixes to merge
git status
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc2_compy.sh
python tests/integration/utils.py
git add -A
git diff --staged tests/integration/template_weekly_comprehensive_v3.cfg
# - partition = "compute"
# + partition = "#expand partition_long#"
git diff --staged tests/integration/utils.py
# - "user_input_v3": "/compyfs/fors729/",
# + "user_input_v3": "/compyfs/fors729/zppy_test_data",There are substantial errors in the jobs. I wasn't able to run the test suite itself. As far as I can tell, the errors seem related to data availability, so given that the Chrysalis tests passed for |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Perlmutter (rc2)zppy-interfacesTesting stepscd ~/ez/zppy-interfaces
git status
# branch: main
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zppy-interfaces/commits/main
# Bump to 0.2.0rc2 => 5a88374fa18782afe8de4f015d0b4504c75e5d06
# e3sm_unified_1_12_0rc2
git checkout -b test_unified_1.12.0rc2_perlmutter 5a88374fa18782afe8de4f015d0b4504c75e5d06
git log
# Good, last commit: Bump to 0.2.0rc2 (#43)
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.12.0rc2_pm-cpu.sh
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 10.98s
pytest tests/unit/pcmdi_diags/test_*.py
# 7 passed, 1 warning in 8.41sThe warning has been noted at PCMDI/pcmdi_metrics#1330 (comment). Success zppyFor reference notes:
Testing steps# We're specifically testing E3SM Unified,
# so we don't have to worry about making dev environments for e3sm_diags or zppy-interfaces.
# Step 2 (of https://docs.e3sm.org/zppy/_build/html/main/dev_guide/release_testing.html)
cd ~/ez/zppy
# Step 3
git status
# branch: test-unified-20250825
# Handle uncommitted changes
git add -A
git commit -m "Testing" --no-verify
# branch: test-unified-20250825
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zppy/commits/main
# Bump to 3.1.0rc2 => 0f24ccd7e4e78330691b905481b791e2b3018a60
# e3sm_unified_1_12_0rc2
git checkout -b test_unified_1.12.0rc2_perlmutter 0f24ccd7e4e78330691b905481b791e2b3018a60
git log
# Good, last commit: Bump to 3.1.0rc2 (#750)
# Step 4
# Previously, we had been testing with one e3sm_diags subtask using a dev environment,
# to test environment_commands overriding, but we have discontinued that.
# Step 5
# Let's add the testing improvements from https://github.com/E3SM-Project/zppy/pull/751
git cherry-pick 93cdf325f03d908e66866ca523951dfa032c77d2
git log
# Good, the commit was added
# Edit tests/integration/utils.py:
#
# TEST_SPECIFICS: Dict[str, Any] = {
# "diags_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.12.0rc2_pm-cpu.sh",
# "global_time_series_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.12.0rc2_pm-cpu.sh",
# "pcmdi_diags_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.12.0rc2_pm-cpu.sh",
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": [
# "e3sm_diags",
# "mpas_analysis",
# "global_time_series",
# "ilamb",
# "pcmdi_diags",
# ],
# "unique_id": "test_e3sm_unified_1.12.0rc2_pm_20251022",
# }
#
# "environment_commands_test": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.12.0rc2_pm-cpu.sh",
#
# generate_cfgs(unified_testing=True, dry_run=False)
# Step 6
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.12.0rc2_pm-cpu.sh
# Step 7
pytest tests/test_*.py
# 35 passed in 0.37s
# Step 8
python tests/integration/utils.py
# Specifically don't want to do these steps, because we're not in a dev environment or making code changes
# pre-commit run --all-files
# python -m pip install .
zppy -c tests/integration/generated/test_weekly_bundles_pm-cpu.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_pm-cpu.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_pm-cpu.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_pm-cpu.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_pm-cpu.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_pm-cpu.cfg
# Started 10/22 13:00
sq | wc -l # Count tasks left
# 15:22 => 142
# 17:21 => 139
# 10/23 10:00 => 9
# 11:07 => 8
# 10/24 15:12 (~2 days later) => 1
sq
# Just the header, so no remaining tasks left
# Picking back up
cd ~/ez/zppy
# Figure out what the output paths are
git grep "output =" tests/integration/generated/test_weekly_*_pm-cpu.cfg | cat
# tests/integration/generated/test_weekly_bundles_pm-cpu.cfg:output = "/global/cfs/cdirs/e3sm/forsyth/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc2_pm_20251022/v3.LR.historical_0051"
# tests/integration/generated/test_weekly_comprehensive_v2_pm-cpu.cfg:output = "/global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v2_output/test_e3sm_unified_1.12.0rc2_pm_20251022/v2.LR.historical_0201"
# tests/integration/generated/test_weekly_comprehensive_v3_pm-cpu.cfg:output = "/global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc2_pm_20251022/v3.LR.historical_0051"
# tests/integration/generated/test_weekly_legacy_3.0.0_bundles_pm-cpu.cfg:output = "/global/cfs/cdirs/e3sm/forsyth/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc2_pm_20251022/v3.LR.historical_0051"
# tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_pm-cpu.cfg:output = "/global/cfs/cdirs/e3sm/forsyth/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/test_e3sm_unified_1.12.0rc2_pm_20251022/v2.LR.historical_0201"
# tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_pm-cpu.cfg:output = "/global/cfs/cdirs/e3sm/forsyth/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/test_e3sm_unified_1.12.0rc2_pm_20251022/v3.LR.historical_0051"
# Check on bundles status
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc2_pm_20251022/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# bundle1.status:ERROR
# bundle2.status:ERROR
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
# e3sm_to_cmip_land_monthly_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_land_monthly_1987-1988-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
# ts_atm_monthly_glb_1985-1989-0005.status:ERROR (2)
# ts_atm_monthly_glb_1990-1994-0005.status:ERROR (2)
# ts_land_monthly_1985-1986-0002.status:ERROR (2)
# ts_land_monthly_1987-1988-0002.status:ERROR (2)
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc2_pm_20251022/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# bundle1.status:ERROR
# bundle2.status:ERROR
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
# e3sm_to_cmip_land_monthly_1985-1986-0002.status:ERROR (2)
# e3sm_to_cmip_land_monthly_1987-1988-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
# ts_atm_monthly_glb_1985-1989-0005.status:ERROR (2)
# ts_atm_monthly_glb_1990-1994-0005.status:ERROR (2)
# ts_land_monthly_1985-1986-0002.status:ERROR (2)
# ts_land_monthly_1987-1988-0002.status:ERROR (2)
# Lots of errors, so we can't proceed to this block:
# cd ~/ez/zppy
# zppy -c tests/integration/generated/test_weekly_bundles_pm-cpu.cfg
# zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_pm-cpu.cfg
# Review finished runs
### v2 ###
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v2_output/test_e3sm_unified_1.12.0rc2_pm_20251022/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:WAITING 44305111
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.status:WAITING 44305112
# e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.status:WAITING 44305104
# e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.status:WAITING 44305105
# e3sm_to_cmip_land_monthly_1980-1981-0002.status:WAITING 44305106
# e3sm_to_cmip_land_monthly_1982-1983-0002.status:WAITING 44305107
# global_time_series_1980-1990.status:WAITING 44305116
# ilamb_1980-1981.status:WAITING 44305118
# ilamb_1982-1983.status:WAITING 44305120
# ts_atm_daily_180x360_aave_1980-1981-0002.status:ERROR (2)
# ts_atm_daily_180x360_aave_1982-1983-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1980-1981-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1982-1983-0002.status:ERROR (2)
# ts_atm_monthly_glb_1980-1984-0005.status:ERROR (2)
# ts_atm_monthly_glb_1985-1989-0005.status:ERROR (2)
# ts_land_monthly_1980-1981-0002.status:ERROR (2)
# ts_land_monthly_1982-1983-0002.status:ERROR (2)
# ts_lnd_monthly_glb_1980-1984-0005.status:ERROR (2)
# ts_lnd_monthly_glb_1985-1989-0005.status:ERROR (2)
# ts_rof_monthly_1980-1981-0002.status:ERROR (2)
# ts_rof_monthly_1982-1983-0002.status:ERROR (2)
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/test_e3sm_unified_1.12.0rc2_pm_20251022/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1982-1983.status:WAITING 44305232
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1980-1981_vs_1980-1981.status:WAITING 44305233
# e3sm_to_cmip_atm_monthly_180x360_aave_1980-1981-0002.status:WAITING 44305225
# e3sm_to_cmip_atm_monthly_180x360_aave_1982-1983-0002.status:WAITING 44305226
# e3sm_to_cmip_land_monthly_1980-1981-0002.status:WAITING 44305227
# e3sm_to_cmip_land_monthly_1982-1983-0002.status:WAITING 44305229
# global_time_series_1980-1990.status:WAITING 44305238
# ilamb_1980-1981.status:WAITING 44305239
# ilamb_1982-1983.status:WAITING 44305240
# ts_atm_daily_180x360_aave_1980-1981-0002.status:ERROR (2)
# ts_atm_daily_180x360_aave_1982-1983-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1980-1981-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1982-1983-0002.status:ERROR (2)
# ts_atm_monthly_glb_1980-1984-0005.status:ERROR (2)
# ts_atm_monthly_glb_1985-1989-0005.status:ERROR (2)
# ts_land_monthly_1980-1981-0002.status:ERROR (2)
# ts_land_monthly_1982-1983-0002.status:ERROR (2)
# ts_lnd_monthly_glb_1980-1984-0005.status:ERROR (2)
# ts_lnd_monthly_glb_1985-1989-0005.status:ERROR (2)
# ts_rof_monthly_1980-1981-0002.status:ERROR (2)
# ts_rof_monthly_1982-1983-0002.status:ERROR (2)
### v3 ###
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc2_pm_20251022/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.status:WAITING 44305159
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.status:WAITING 44305160
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:WAITING 44305151
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:WAITING 44305152
# e3sm_to_cmip_atm_monthly_180x360_aave_1989-1990-0002.status:WAITING 44305154
# e3sm_to_cmip_atm_monthly_180x360_aave_1991-1992-0002.status:WAITING 44305155
# e3sm_to_cmip_atm_monthly_180x360_aave_1993-1994-0002.status:WAITING 44305156
# e3sm_to_cmip_land_monthly_1985-1986-0002.status:WAITING 44305157
# e3sm_to_cmip_land_monthly_1987-1988-0002.status:WAITING 44305158
# global_time_series_all_lnd_var_viewer_1985-1995.status:WAITING 44305183
# global_time_series_classic_original_8_no_ocn_1985-1995.status:WAITING 44305186
# global_time_series_classic_pdf_both_1985-1995.status:WAITING 44305171
# global_time_series_classic_pdf_component_1985-1995.status:WAITING 44305178
# global_time_series_classic_pdf_original_1985-1995.status:WAITING 44305174
# global_time_series_viewer_both_1985-1995.status:WAITING 44305166
# global_time_series_viewer_component_1985-1995.status:WAITING 44305170
# global_time_series_viewer_original_1985-1995.status:WAITING 44305168
# ilamb_1985-1988.status:WAITING 44305187
# pcmdi_diags_mean_climate_model_vs_obs_1985-1994.status:WAITING 44305188
# pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 44305191
# pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.status:WAITING 44305190
# pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.status:WAITING 44305189
# ts_atm_daily_180x360_aave_1985-1986-0002.status:ERROR (2)
# ts_atm_daily_180x360_aave_1987-1988-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1989-1990-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1991-1992-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1993-1994-0002.status:ERROR (2)
# ts_atm_monthly_glb_1985-1989-0005.status:ERROR (2)
# ts_atm_monthly_glb_1990-1994-0005.status:ERROR (2)
# ts_land_monthly_1985-1986-0002.status:ERROR (2)
# ts_land_monthly_1987-1988-0002.status:ERROR (2)
# ts_lnd_monthly_glb_1985-1989-0005.status:ERROR (2)
# ts_lnd_monthly_glb_1990-1994-0005.status:ERROR (2)
# ts_rof_monthly_1985-1986-0002.status:ERROR (2)
# ts_rof_monthly_1987-1988-0002.status:ERROR (2)
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/test_e3sm_unified_1.12.0rc2_pm_20251022/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1987-1988.status:WAITING 44305279
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.status:WAITING 44305280
# e3sm_to_cmip_atm_monthly_180x360_aave_1985-1986-0002.status:WAITING 44305272
# e3sm_to_cmip_atm_monthly_180x360_aave_1987-1988-0002.status:WAITING 44305273
# e3sm_to_cmip_land_monthly_1985-1986-0002.status:WAITING 44305275
# e3sm_to_cmip_land_monthly_1987-1988-0002.status:WAITING 44305277
# global_time_series_1985-1995.status:WAITING 44305291
# ilamb_1985-1988.status:WAITING 44305293
# ts_atm_daily_180x360_aave_1985-1986-0002.status:ERROR (2)
# ts_atm_daily_180x360_aave_1987-1988-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1985-1986-0002.status:ERROR (2)
# ts_atm_monthly_180x360_aave_1987-1988-0002.status:ERROR (2)
# ts_atm_monthly_glb_1985-1989-0005.status:ERROR (2)
# ts_atm_monthly_glb_1990-1994-0005.status:ERROR (2)
# ts_land_monthly_1985-1986-0002.status:ERROR (2)
# ts_land_monthly_1987-1988-0002.status:ERROR (2)
# ts_lnd_monthly_glb_1985-1989-0005.status:ERROR (2)
# ts_lnd_monthly_glb_1990-1994-0005.status:ERROR (2)
# ts_rof_monthly_1985-1986-0002.status:ERROR (2)
# ts_rof_monthly_1987-1988-0002.status:ERROR (2)
# We definitely can't proceed with the test suite itself.
grep -in error *.o*
# This has a variety of errors
# We have the following in tests/integration/utils.py:
# "user_input_v2": "/global/cfs/cdirs/e3sm/forsyth/",
# "user_input_v3": "/global/cfs/cdirs/e3sm/forsyth/",
ls /global/cfs/cdirs/e3sm/forsyth/E3SMv2/v2.LR.historical_0201
# archive build case_scripts init mpas-ocean mpas-seaice post run v2.LR.historical_0201 zstash
ls /global/cfs/cdirs/e3sm/forsyth/E3SMv3/v3.LR.historical_0051
# add.mosart-h1 archive build case_scripts code disk.usage init mpas-ocean mpas-seaice post run tests
# So... unlike Compy, the data paths are correct for both v2 and v3
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v2_output/test_e3sm_unified_1.12.0rc2_pm_20251022/v2.LR.historical_0201/post/scripts
grep -in error *.o*
# This has a variety of errors
# A lot of variables not being found
cd ~/ez/zppy
# Did we hit a quota maybe?
showquota --hpss
# +---------------------------------------+------------+-------------+----------------+------------+-------------+----------------+
# | File system | Space used | Space quota | Space used (%) | Inode used | Inode quota | Inode used (%) |
# +---------------------------------------+------------+-------------+----------------+------------+-------------+----------------+
# | home | 12.97GiB | 40.00GiB | 32.4% | 307.48K | 1.00M | 30.7% |
# | pscratch | 125.92GiB | 20.00TiB | 0.6% | 121.00 | 10.00M | 0.0% |
# | forsyth usage on HPSS charged to e3sm | 1019.11TiB | 2.00PiB | 49.7% | - | - | - |
# +---------------------------------------+------------+-------------+----------------+------------+-------------+----------------+
# So... no not a disk space issue.
# Let's check if we have any fixes to merge
git status
git diff --staged tests/integration/utils.py
# No, these are just testing changesThere are substantial errors in the jobs. I wasn't able to run the test suite itself. As far as I can tell, the errors seem related to data availability, so given that the Chrysalis tests passed for |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
@forsyth2 I don't have permission to look into /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v2_output/test_e3sm_unified_1.12.0rc2_pm_20251022/v2.LR.historical_0201/post/scripts Based on the failed tasks, I think it is related to ncclimo, that it currently uses the ncclimo image path for chrysalis, to be able to test on Compy and Perlmutter, this needs to be fixed first. Line 67 in 7c975a1 |
Beta Was this translation helpful? Give feedback.
1 reply
-
Oh yes, that crossed my mind briefly as well. Thanks for the reminder! |
Beta Was this translation helpful? Give feedback.
-
|
Chrysaliszppy-interfacesTesting stepscd ~/ez/zppy-interfaces
git status
# branch: test_unified_1.12.0rc3_chrysalis
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zppy-interfaces/commits/main
# Bump to 0.2.0rc2 => 5a88374fa18782afe8de4f015d0b4504c75e5d06
# Remember, zppy-interfaces didn't get a new RC
git checkout -b test_unified_1.12.0rc3_chrysalis 5a88374fa18782afe8de4f015d0b4504c75e5d06
git log
# Good, last commit: Bump to 0.2.0rc2 (#43)
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc3_chrysalis.sh
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 26.05s
pytest tests/unit/pcmdi_diags/test_*.py
# 7 passed, 1 warning in 15.06sThe warning has been noted at PCMDI/pcmdi_metrics#1330 (comment). Success zppyTesting steps# We're specifically testing E3SM Unified,
# so we don't have to worry about making dev environments for e3sm_diags or zppy-interfaces.
# Step 2 (of https://docs.e3sm.org/zppy/_build/html/main/dev_guide/release_testing.html)
cd ~/ez/zppy
# Step 3
git status
# branch: main
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zppy/commits/main
# Bump to 3.1.0rc3 => 05acfd8cec4c14c75631068d1333e14311af9722
git checkout -b test_unified_1.12.0rc3_chrysalis 05acfd8cec4c14c75631068d1333e14311af9722
git log
# Good, last commit: Bump to 3.1.0rc3 (#754)
# Step 4
# Previously, we had been testing with one e3sm_diags subtask using a dev environment,
# to test environment_commands overriding, but we have discontinued that.
# Step 5
# Edit tests/integration/utils.py:
#
# TEST_SPECIFICS: Dict[str, Any] = {
# "diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc3_chrysalis.sh",
# "global_time_series_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc3_chrysalis.sh",
# "pcmdi_diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc3_chrysalis.sh",
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": [
# "e3sm_diags",
# "mpas_analysis",
# "global_time_series",
# "ilamb",
# "pcmdi_diags",
# ],
# "unique_id": "test_e3sm_unified_1.12.0rc3_20251030",
# }
#
# "environment_commands_test": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc3_chrysalis.sh",
#
# generate_cfgs(unified_testing=True, dry_run=False)
# Step 6
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc3_chrysalis.sh
# Step 7
pytest tests/test_*.py
# 35 passed in 1.26s
# Step 8
python tests/integration/utils.py
# Specifically don't want to do these steps, because we're not in a dev environment or making code changes
# pre-commit run --all-files
# python -m pip install .
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
# Count the number of jobs
alias sq
# alias sq='sqa -u ac.forsyth2'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
sq | wc -l # Thu 10/30 ~09:40 => 144 - header row = 143 jobs
sq | wc -l # Thu 10/30 10:00 => 32 - header row = 31 jobs
sq | wc -l # Thu 10/30 10:42 => 4 - header row = 3 jobs
sq | wc -l # Thu 10/30 10:48 => 1 - header row = 0 jobs (~1h15m of runtime)
# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc3_20251030/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc3_20251030/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Now, run bundles part 2
cd ~/ez/zppy
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
sq | wc -l # Thu 10/30 10:50 => 5 - header row = 4 jobs
# Review finished runs
### v2 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_e3sm_unified_1.12.0rc3_20251030/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/test_e3sm_unified_1.12.0rc3_20251030/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_20251030/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_20251030/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
### bundles ###
sq | wc -l # Thu 10/30 10:59 => 1 - header row = 0 jobs
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc3_20251030/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc3_20251030/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd ~/ez/zppy
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 2.16s, diffs appear to be expected based on recent changes on `main` (change NCO call)
pytest tests/integration/test_campaign.py
# 6 passed in 1.88s
pytest tests/integration/test_defaults.py
# 1 passed in 0.35s
pytest tests/integration/test_last_year.py
# 1 passed in 0.32s
pytest tests/integration/test_bundles.py
# 2 passed in 0.15s
pytest tests/integration/test_images.py
# Started 10/30 11:01am
# FAILED tests/integration/test_images.py::test_images - assert 3691 == 3659
# 1 failed in 3617.06s (1:00:17)
cat test_images_summary.mdComplete resultsDiff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
Steps to construct concise results:
Concise results
Initial notesReminders on the changes we're testing (E3SM Unified 1.12.0rc2 -> rc3)
Reminders regarding
Notes:
Re-test using recent testing improvementsLet's use the commits from #755 & #752. Notice in both of those pull requests, only files in cd ~/ez/zppy
git status
# branch fixes-for-rc4
# nothing to commit
git checkout test_unified_1.12.0rc3_chrysalis
git status
# branch test_unified_1.12.0rc3_chrysalis
# nothing to commit
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc3_chrysalis.sh
git fetch upstream
# Get commits from https://github.com/E3SM-Project/zppy/pull/755/commits
# We're not rerunning the jobs, so the first two commits aren't vitally important.
# git cherry-pick c30622e38882de1bfa46073f5974049e49e82b43
# git cherry-pick 14af8277481c972fdd2a67245d683a7462c6d6ae
git cherry-pick eb74f837bd35a86e6890d22158fa6b0afdc0de91
# Get commits from https://github.com/E3SM-Project/zppy/pull/752/commits
# Avoid conflicts by first squashing the 3 commits
git fetch upstream multithread-image-tests
git checkout -b multithread-image-tests-single-commit upstream/multithread-image-tests
git log
git rebase -i 93cdf325f03d908e66866ca523951dfa032c77d2
# pick 0e9b692a Attempt to multithread the image tests
# f a81a7c77 Working multiprocessing
# f b8881c76 Reset test utils
git log
# Last commit hash: 551ae585456219f9795a032a043999ef85257dae
git checkout test_unified_1.12.0rc3_chrysalis
git cherry-pick 551ae585456219f9795a032a043999ef85257dae
git log
# Attempt to multithread the image tests (squashed version)
# Fix image_diff_grid pagination
# Testing
# Ok, all set to re-test
pytest tests/integration/test_images.py
# Started: Thu 10/30 15:42
# 1 failed in 1063.28s (0:17:43)Log includes:
Diff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
Final results:
Testing improvements:
Final notes: For the most part, zppy using E3SM Unified @chengzhuzhang Can you please review the diffs noted above? Please note that diffs may be hard to see in the image diff grid; that's ok though -- the point is to quickly get a high-level overview to determine which files to investigate further. For example:
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Compyzppy-interfacesTesting stepscd ~/ez/zppy-interfaces
git status
# branch: test_unified_1.12.0rc2_compy
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zppy-interfaces/commits/main
# Bump to 0.2.0rc2 => 5a88374fa18782afe8de4f015d0b4504c75e5d06
# Remember, zppy-interfaces didn't get a new RC
git checkout -b test_unified_1.12.0rc3_compy 5a88374fa18782afe8de4f015d0b4504c75e5d06
git log
# Good, last commit: Bump to 0.2.0rc2 (#43)
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc3_compy.sh
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 19.24s
pytest tests/unit/pcmdi_diags/test_*.py
# 7 passed, 1 warning in 12.10sThe warning has been noted at PCMDI/pcmdi_metrics#1330 (comment). Success zppyTesting steps# We're specifically testing E3SM Unified,
# so we don't have to worry about making dev environments for e3sm_diags or zppy-interfaces.
# Step 2 (of https://docs.e3sm.org/zppy/_build/html/main/dev_guide/release_testing.html)
cd ~/ez/zppy
# Step 3
git status
# branch test_unified_1.12.0rc2_compy
# Handle uncommitted changes
git add -A
git commit -m "Testing" --no-verify
git status
# branch test_unified_1.12.0rc2_compy
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zppy/commits/main
# Bump to 3.1.0rc3 => 05acfd8cec4c14c75631068d1333e14311af9722
git checkout -b test_unified_1.12.0rc3_compy 05acfd8cec4c14c75631068d1333e14311af9722
git log
# Good, last commit: Bump to 3.1.0rc3 (#754)
# Step 4
# Previously, we had been testing with one e3sm_diags subtask using a dev environment,
# to test environment_commands overriding, but we have discontinued that.
# Step 5
# Edit tests/integration/utils.py:
#
# TEST_SPECIFICS: Dict[str, Any] = {
# "diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc3_compy.sh",
# "global_time_series_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc3_compy.sh",
# "pcmdi_diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc3_compy.sh",
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": [
# "e3sm_diags",
# "mpas_analysis",
# "global_time_series",
# "ilamb",
# "pcmdi_diags",
# ],
# "unique_id": "test_e3sm_unified_1.12.0rc3_compy_20251030",
# }
#
# "environment_commands_test": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc3_compy.sh",
#
# generate_cfgs(unified_testing=True, dry_run=False)
# Step 6
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc3_compy.sh
# Step 7
pytest tests/test_*.py
# 35 passed in 1.10s
# Step 8
python tests/integration/utils.py
# Specifically don't want to do these steps, because we're not in a dev environment or making code changes
# pre-commit run --all-files
# python -m pip install .
zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_compy.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_compy.cfg
# Count the number of jobs
alias sq
# alias sq='sqa -u fors729'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
sq | wc -l # Thu 10/30 10:00 => 114 - header row = 113 jobs
sq | wc -l # Thu 10/30 10:43 => 19 - header row = 18 jobs
sq | wc -l # Thu 10/30 10:58 => 13 - header row = 12 jobs
sq | wc -l # Thu 10/30 10:58 => 2 - header row = 1 jobs
sq # ~11:12 (~1h15m of runtime)
# USER ACCOUNT NODE PARTITION JOBID ST REASON TIME TIME_LIMIT PRIORITY NAME
# fors729 e3sm 1 short 705355 PD Dependency 0:00 2:00:00 425 pcmdi_diags_synthetic_plots_model_vs_obs
# That's not good. That means a dependency failed.
# First, let's check on the bundles status
cd /compyfs/fors729/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc3_compy_20251030/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc3_compy_20251030/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Now, run bundles part 2
cd ~/ez/zppy
zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_compy.cfg
sq | wc -l # Thu 10/30 11:16 => 6 - header row = 5 jobs (including the one waiting for a dependency)
# Review finished runs
### v2 ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/test_e3sm_unified_1.12.0rc3_compy_20251030/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# ilamb_1980-1981.status:ERROR (3)
# ilamb_1982-1983.status:ERROR (3)
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/test_e3sm_unified_1.12.0rc3_compy_20251030/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# ilamb_1980-1981.status:ERROR (3)
# ilamb_1982-1983.status:ERROR (3)
### v3 ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_compy_20251030/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# ilamb_1985-1988.status:ERROR (3)
# pcmdi_diags_mean_climate_model_vs_obs_1985-1994.status:ERROR (11)
# pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 705355
# pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.status:ERROR (8)
# pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.status:ERROR (8)
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_compy_20251030/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# ilamb_1985-1988.status:ERROR (3)
### bundles ###
sq | wc -l # Thu 10/30 11:19 => 4 - header row = 3 jobs (including the one waiting for a dependency)
sq | wc -l # Thu 10/30 11:21 => 2 - header row = 1 jobs (including the one waiting for a dependency)
# Let's cancel that 1 job
scancel -u fors729
cd /compyfs/fors729/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc3_compy_20251030/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# ilamb_1985-1986.status:ERROR (3)
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc3_compy_20251030/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# ilamb_1985-1986.status:ERROR (3)
# We definitely can't proceed with the test suite itself.
# Let's debug...
cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/test_e3sm_unified_1.12.0rc3_compy_20251030/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# ilamb_1980-1981.status:ERROR (3)
# ilamb_1982-1983.status:ERROR (3)
grep -in error *.o* | wc -l
# 7566 matching lines!
grep -in error ilamb*.o* | wc -l
# 6
grep -in error ilamb*.o*
# ilamb_1980-1981.o705308:11:srun: error: n0033: task 0: Exited with exit code 1
# ilamb_1980-1981.o705308:28: raise RuntimeError("\n".join(errors))
# ilamb_1980-1981.o705308:29:RuntimeError: cannot load MPI library
# ilamb_1982-1983.o705309:11:srun: error: n0011: task 0: Exited with exit code 1
# ilamb_1982-1983.o705309:28: raise RuntimeError("\n".join(errors))
# ilamb_1982-1983.o705309:29:RuntimeError: cannot load MPI library
emacs ilamb_1980-1981.o705308In there, we see the problem we keep have from time-to-time re: "cannot open shared object file" # Let's make sure there's nothing chrysalis-specific in this file #############
grep -in lcrc ilamb*
# Good, no matches
grep -in "ac\." ilamb*
# Good, no matches
# Let's check the other ilamb errors ##########################################
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/test_e3sm_unified_1.12.0rc3_compy_20251030/v2.LR.historical_0201/post/scripts
grep -in error ilamb*.o* | wc -l
# 6
grep -in error ilamb*.o*
# ilamb_1980-1981.o705388:11:srun: error: n0029: task 0: Exited with exit code 1
# ilamb_1980-1981.o705388:28: raise RuntimeError("\n".join(errors))
# ilamb_1980-1981.o705388:29:RuntimeError: cannot load MPI library
# ilamb_1982-1983.o705389:11:srun: error: n0037: task 0: Exited with exit code 1
# ilamb_1982-1983.o705389:28: raise RuntimeError("\n".join(errors))
# ilamb_1982-1983.o705389:29:RuntimeError: cannot load MPI library
grep -inl "cannot open shared object" *.o*
# ilamb_1980-1981.o705388
# ilamb_1982-1983.o705389
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_compy_20251030/v3.LR.historical_0051/post/scripts
grep -inl "cannot open shared object" *.o*
# ilamb_1985-1988.o705351
# NOTE: so, this isn't the cause of the pcmdi_diags errors as well
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_compy_20251030/v3.LR.historical_0051/post/scripts
grep -nl "cannot open shared object" *.o*
# ilamb_1985-1988.o705418
cd /compyfs/fors729/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc3_compy_20251030/v3.LR.historical_0051/post/scripts
grep -l "cannot open shared object" *.o*
# ilamb_1985-1986.o705743
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc3_compy_20251030/v3.LR.historical_0051/post/scripts
grep -l "cannot open shared object" *.o*
# ilamb_1985-1986.o705745
# Let's check the pcmdi_diags errors ##########################################
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_compy_20251030/v3.LR.historical_0051/post/scripts
grep -in error pcmdi_diags*.o* | wc -l
# 178
grep -in error pcmdi_diags*.o*
grep "ncra: ERROR received 1 positional filename(s); need at least two" pcmdi_diags*.o* | wc -l
# 132
# => 46 other errors. Let's filter the ncra errors out:
grep -in error pcmdi_diags*.o* | grep -v "ncra: ERROR received 1 positional filename(s); need at least two"
grep -in "not found. It does not exist on the local filesystem, nor does it match remote filename patterns" pcmdi_diags*.o* | wc -l
# 22
# 46-22 => Still need to account for 24 errors
grep -in error pcmdi_diags*.o* | grep -v "ncra: ERROR received 1 positional filename(s); need at least two" | grep -v "not found. It does not exist on the local filesystem, nor does it match remote filename patterns"
grep -in error pcmdi_diags*.o* | grep -v "ncra: ERROR received 1 positional filename(s); need at least two" | grep -v "not found. It does not exist on the local filesystem, nor does it match remote filename patterns" | grep -v "HINT file-not-found errors usually arise from filename typos"
# pcmdi_diags_mean_climate_model_vs_obs_1985-1994.o705352:10920: raise ValueError("obs_dic has no keys!")
# pcmdi_diags_mean_climate_model_vs_obs_1985-1994.o705352:10921:ValueError: obs_dic has no keys!
# Let's make sure there's nothing chrysalis-specific in these files ###########
grep -in lcrc pcmdi_diags*
# pcmdi_diags_mean_climate_model_vs_obs_1985-1994.bash:1048:command="zi-pcmdi-link-observation --model_name_ref ${model_name_ref} --tableID_ref ${tableID_ref} --vars=${source_vars} --obs_sets default --obs_ts /lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series --obstmp_dir ${obstmp_dir} --debug ${debug,,}"
# pcmdi_diags_mean_climate_model_vs_obs_1985-1994.o705352:10716:Running a zi-pcmdi command: zi-pcmdi-link-observation --model_name_ref obs.historical.put_model_here.00 --tableID_ref Amon --vars=pr,psl,rsus,rsuscs,rlus,rlut,rsds,rsdscs,rsdt,rsut,sfcWind,tas,tauu,tauv,ts,ta-200,ta-850,ua-200,ua-850,va-200,va-850 --obs_sets default --obs_ts /lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series --obstmp_dir obs_link --debug false
# pcmdi_diags_mean_climate_model_vs_obs_1985-1994.settings:81: 'obs_ts': '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series',
# pcmdi_diags_synthetic_plots_model_vs_obs.settings:74: 'obs_ts': '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series',
# pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.bash:1055:command="zi-pcmdi-link-observation --model_name_ref ${model_name_ref} --tableID_ref ${tableID_ref} --vars=${source_vars} --obs_sets alternate2 --obs_ts /lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series --obstmp_dir ${obstmp_dir} --debug ${debug,,}"
# pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.o705354:5:Running a zi-pcmdi command: zi-pcmdi-link-observation --model_name_ref obs.historical.put_model_here.00 --tableID_ref Amon --vars=psl --obs_sets alternate2 --obs_ts /lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series --obstmp_dir obs_link --debug false
# pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.settings:81: 'obs_ts': '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series',
# pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.bash:1055:command="zi-pcmdi-link-observation --model_name_ref ${model_name_ref} --tableID_ref ${tableID_ref} --vars=${source_vars} --obs_sets alternate1 --obs_ts /lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series --obstmp_dir ${obstmp_dir} --debug ${debug,,}"
# pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.o705353:5:Running a zi-pcmdi command: zi-pcmdi-link-observation --model_name_ref obs.historical.put_model_here.00 --tableID_ref Amon --vars=ts --obs_sets alternate1 --obs_ts /lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series --obstmp_dir obs_link --debug false
# pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.settings:81: 'obs_ts': '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series',
grep -in "ac\." pcmdi_diags*
# This is counting AC
grep -n "ac\." pcmdi_diags*
# Good, no matches
cd ~/ez/zppy
git add -A # Do this so `git grep` will pick up new files
grep -in lcrc tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
# 159: dc_obs_climo = '/lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/obs/climatology'
# 324:obs_ts = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series'
# git blame: https://github.com/E3SM-Project/zppy/blame/main/tests/integration/template_weekly_comprehensive_v3.cfg
# dc_obs_climo => This was added hard-coded in https://github.com/E3SM-Project/zppy/pull/604/files (2024-07-31!)
# obs_ts => This was added hard-coded in https://github.com/E3SM-Project/zppy/pull/732/files (3 weeks ago)Notes:
Re-test using recent testing improvementsLet's use the commits from #755 & #752. Notice in both of those pull requests, only files in cd ~/ez/zppy
git status
# branch test_unified_1.12.0rc3_compy
# Handle uncommitted changes
git add -A
git commit -m "Testing" --no-verify
git status
# branch test_unified_1.12.0rc3_compy
# nothing to commit
git fetch upstream
git checkout -b test_unified_1.12.0rc3_compy_try2 upstream/fixes-for-rc4
git log
# Good, has the 3 commits from https://github.com/E3SM-Project/zppy/pull/755/commits
# Now, get the 3 commits from https://github.com/E3SM-Project/zppy/pull/752/commits
# Avoid conflicts by first squashing the 3 commits
git checkout -b multithread-image-tests-single-commit upstream/multithread-image-tests
git log
git rebase -i 93cdf325f03d908e66866ca523951dfa032c77d2
# pick 0e9b692a Attempt to multithread the image tests
# f a81a7c77 Working multiprocessing
# f b8881c76 Reset test utils
git log
# Last commit hash: 015aa301ab84b1885e99a0cb7f436638b8558b23
git checkout test_unified_1.12.0rc3_compy_try2
git cherry-pick 015aa301ab84b1885e99a0cb7f436638b8558b23
git log
# Attempt to multithread the image tests (squashed version)
# Fix image_diff_grid pagination
# Testing
# Ok, all set to re-run jobs + run test suite in the first place
# NOTE: we're not expecting the shared library errors to be gone.
# But the path errors should be resolved now.
# Edit tests/integration/utils.py. Notice TEST_SPECIFICS is the *only* part we need to change now!
#
# TEST_SPECIFICS: Dict[str, Any] = {
# # These are custom environment_commands for specific tasks.
# # Never set these to "", because they will print the line
# # `environment_commands = ""` for the corresponding task,
# # thus overriding the value set higher up in the cfg.
# # That is, there will be no environment set.
# # (`environment_commands = ""` only redirects to Unified
# # if specified under the [default] task)
# "diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc3_compy.sh",
# "global_time_series_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc3_compy.sh",
# "pcmdi_diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc3_compy.sh",
# # This is the environment setup for other tasks.
# # Leave as "" to use the latest Unified environment.
# "environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc3_compy.sh",
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": [
# "e3sm_diags",
# "mpas_analysis",
# "global_time_series",
# "ilamb",
# "pcmdi_diags",
# ],
# "unique_id": "test_e3sm_unified_1.12.0rc3_compy_20251030_try2",
# }
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc3_compy.sh
pytest tests/test_*.py
# 35 passed in 0.66s
python tests/integration/utils.py
# Specifically don't want to do these steps, because we're not in a dev environment or making code changes
# pre-commit run --all-files
# python -m pip install .
zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_compy.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_compy.cfg
# Count the number of jobs
alias sq
# alias sq='sqa -u fors729'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
sq | wc -l # Thu 10/30 17:09 => 132 - header row = 131 jobs
sq | wc -l # Thu 10/30 17:13 => 85 - header row = 84 jobs
sq | wc -l # Thu 10/30 18:08 => 9 - header row = 8 jobs
sq | wc -l # Thu 10/30 18:21 => 2 - header row = 1 job
sq
# USER ACCOUNT NODE PARTITION JOBID ST REASON TIME TIME_LIMIT PRIORITY NAME
# fors729 e3sm 1 short 707475 PD Dependency 0:00 2:00:00 420 pcmdi_diags_synthetic_plots_model_vs_obs
# Once again, a dependency failed.
# Let's cancel that.
scancel -u fors729
# First, let's check on the bundles status
cd /compyfs/fors729/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc3_compy_20251030_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc3_compy_20251030_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Now, run bundles part 2
cd ~/ez/zppy
zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_compy.cfg
sq | wc -l # Thu 10/30 18:25 => 5 - header row = 4 jobs
# Review finished runs
### v2 ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/test_e3sm_unified_1.12.0rc3_compy_20251030_try2/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# ilamb_1980-1981.status:ERROR (3)
# ilamb_1982-1983.status:ERROR (3)
# Those are expected
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/test_e3sm_unified_1.12.0rc3_compy_20251030_try2/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# ilamb_1980-1981.status:ERROR (3)
# ilamb_1982-1983.status:ERROR (3)
# Those are expected
### v3 ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_compy_20251030_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# ilamb_1985-1988.status:ERROR (3)
# pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 707475
# pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.status:ERROR (8)
# pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.status:ERROR (8)
# The ilamb error is expected
# The pcmdi_diags errors are NOT expected.
# It does look like we fixed pcmdi_diags mean_climate though.
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_compy_20251030_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# ilamb_1985-1988.status:ERROR (3)
# That is expected
### bundles ###
sq | wc -l # Thu 10/30 18:26 => 3 - header row = 2 jobs
sq | wc -l # Thu 10/30 18:37 => 1 - header row = 0 jobs
cd /compyfs/fors729/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc3_compy_20251030_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# ilamb_1985-1986.status:ERROR (3)
# That is expected
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc3_compy_20251030_try2/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# ilamb_1985-1986.status:ERROR (3)
# That is expected
# Let's debug the remaining pcmdi_diags errors
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_compy_20251030_try2/v3.LR.historical_0051/post/scripts
grep -in error pcmdi_diags*.o* | wc -l
# 432
grep -in error pcmdi_diags*.o* | grep -v ": HINT" | wc -l
# 290
grep -in error pcmdi_diags*.o* | grep -v ": HINT" | grep -v "not found. It does not exist on the local filesystem, nor does it match remote filename patterns" | wc -l
# 148
grep -in error pcmdi_diags*.o* | grep -v ": HINT" | grep -v "not found. It does not exist on the local filesystem, nor does it match remote filename patterns"
# Alot of these are from mean_climate actually...
grep -in error pcmdi_diags_variability_modes*.o* | grep -v ": HINT" | grep -v "not found. It does not exist on the local filesystem, nor does it match remote filename patterns" | wc -l
# 0
grep -in error pcmdi_diags_variability_modes*.o* | wc -l
# 0
emacs pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.o707474
# create_links_ts: linking from /compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_compy_20251030_try2/v3.LR.historical_0051/post/atm/180x360_aave/cmip_ts/monthly to t\
# s
# ncap2: WARNING ncap_def_dim(): dim "bnds" - already exists in input/output.
# Linking observational data into obs_link...
# Linking observational data using SLURM...
# Running a zi-pcmdi command: zi-pcmdi-link-observation --model_name_ref obs.historical.put_model_here.00 --tableID_ref Amon --vars=psl --obs_sets alternate2 --obs_ts /compyfs/diagnostics/observati\
# ons/Atm/time-series/ --obstmp_dir obs_link --debug false
# [Warning] Observation file not found for psl (alternate2)
# real 0m16.104s
# user 0m11.067s
# sys 0m1.140s
# create_links_ts_obs: linking from /compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_compy_20251030_try2/v3.LR.historical_0051/post/scripts/tmp.pcmdi_diags_variabili\
# ty_modes_atm_model_vs_obs_1985-1994.707474.ZVFw/obs_link to ts_ref
# create_links_ts_obs: checking if nc file: *
# create_links_ts_obs: ts_ref was not updated!
emacs pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.o707473
# create_links_ts: linking from /compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_compy_20251030_try2/v3.LR.historical_0051/post/atm/180x360_aave/cmip_ts/monthly to t\
# s
# ncap2: WARNING ncap_def_dim(): dim "bnds" - already exists in input/output.
# Linking observational data into obs_link...
# Linking observational data using SLURM...
# Running a zi-pcmdi command: zi-pcmdi-link-observation --model_name_ref obs.historical.put_model_here.00 --tableID_ref Amon --vars=ts --obs_sets alternate1 --obs_ts /compyfs/diagnostics/observatio\
# ns/Atm/time-series/ --obstmp_dir obs_link --debug false
# [Warning] Observation file not found for ts (alternate1)
# real 0m16.168s
# user 0m11.098s
# sys 0m1.119s
# create_links_ts_obs: linking from /compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_compy_20251030_try2/v3.LR.historical_0051/post/scripts/tmp.pcmdi_diags_variabili\
# ty_modes_cpl_model_vs_obs_1985-1994.707473.eXNH/obs_link to ts_ref
# create_links_ts_obs: checking if nc file: *
# create_links_ts_obs: ts_ref was not updated!
# It looks like we don't have sufficient data available on Compy
# Or in any case, I haven't find the correct path on Compy.
# Let's trying running the test suite to check the jobs that did manage to complete successfully
cd ~/ez/zppy
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 7.30s, diffs appear to be expected, based on changes made during this development period
pytest tests/integration/test_campaign.py
# 6 failed in 10.23s, same note as above
pytest tests/integration/test_defaults.py
# 1 failed in 1.12s, same note as above
pytest tests/integration/test_last_year.py
# 1 passed in 1.13s
pytest tests/integration/test_bundles.py
# 2 passed in 0.60s
pytest tests/integration/test_images.py
# Started Thu 10/30 18:42
# 1 failed, 3 warnings in 1172.74s (0:19:32)That gives: cat early_test_images_summary.md
# # Summary of test results
#
# Diff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
# Note image diff grids can not yet be constructed automatically.
# | Test name | Total images | Correct images | Missing images | Mismatched images | Diff subdir |
# | --- | --- | --- | --- | --- | --- |
# Unfortunately, nothing prints.
# Edit tests/integration/utils.py to remove the legacy tests, since we don't have those set up on Compy
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# ],
# Let's re-run
pytest tests/integration/test_images.py
# Started Thu 10/30 19:19That gives: # Edit tests/integration/utils.py to remove the pcmdi_diags task, since we don't have that set up on Compy
# "tasks_to_run": [
# "e3sm_diags",
# "mpas_analysis",
# "global_time_series",
# "ilamb",
# ],
# Let's re-run
pytest tests/integration/test_images.py
# Started Thu 10/30 19:45That gives:
Diff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
Concise results:
Final notes:
|
Beta Was this translation helpful? Give feedback.
3 replies
-
|
@forsyth2 thanks for testing and the instruction on reviewing. |
Beta Was this translation helpful? Give feedback.
-
|
Are you asking about Chrysalis? Those are results generated using Directions for how to find version of expected resultsIn general, there are 3 ways to see what version/date the expected results were run off: # 1.
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# Notice everything was updated on 10/20
# 2.
ls /lcrc/group/e3sm/public_html/zppy_test_resources_previous
# We can see the last previous set of expected results was:
# expected_results_until_20251020
# 3. Find the latest comment on updating the expected results
# This option is a little more difficult because the comments can be scattered around.
# But you can find the actual version used.E.g., on #746 (reply in thread) we can see that we updated expected results based off the If you did mean to put this on the Compy thread, remember the expected results are quite out of date. Still, we can find when the expected results were updated. ls -lt /compyfs/www/zppy_test_resources/
# We can see most directories were updated on 4/29/2025Based on the dates of this thread, I would say these are the "official" results for E3SM Unified 1.11.1 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! I only reviewed the results on Chrysalis. For the land model vs. model plots, RC3 definitely has the correct masks compared to RC2 (which shows zero values for masked points). I’m not sure, but it might be due to using a newer version of NCO... |
Beta Was this translation helpful? Give feedback.
-
|
Perlmutterzppy-interfacesTesting stepscd ~/ez/zppy-interfaces
git status
# branch: test_unified_1.12.0rc3_perlmutter
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zppy-interfaces/commits/main
# Bump to 0.2.0rc2 => 5a88374fa18782afe8de4f015d0b4504c75e5d06
# Remember, zppy-interfaces didn't get a new RC
git checkout -b test_unified_1.12.0rc3_perlmutter_try2 5a88374fa18782afe8de4f015d0b4504c75e5d06
git log
# Good, last commit: Bump to 0.2.0rc3 (#43)
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.12.0rc3_pm-cpu.sh
# Successful this time
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 13.50s
pytest tests/unit/pcmdi_diags/test_*.py
# 7 passed, 1 warning in 9.74sThe warning has been noted at PCMDI/pcmdi_metrics#1330 (comment). Success zppyTesting steps# We're specifically testing E3SM Unified,
# so we don't have to worry about making dev environments for e3sm_diags or zppy-interfaces.
cd ~/ez/zppy
git status
# branch test_unified_1.12.0rc2_perlmutter
# Handle uncommitted changes
git add -A
git commit -m "Testing" --no-verify
git status
# branch test_unified_1.12.0rc3_perlmutter
# nothing to commit
# Let's get the latest testing updates
git fetch upstream
git checkout -b test_unified_1.12.0rc3_perlmutter_try2 upstream/fixes-for-rc4
git log
# Good:
# The 3 commits from https://github.com/E3SM-Project/zppy/pull/755
# Bump to 3.1.0rc3 (#754)
#
# Now, let's get the 3 commits from https://github.com/E3SM-Project/zppy/pull/752
# Avoid conflicts by first squashing the 3 commits
git checkout -b multithread-image-tests-single-commit upstream/multithread-image-tests
git log
git rebase -i 93cdf325f03d908e66866ca523951dfa032c77d2
# pick 0e9b692 Attempt to multithread the image tests
# f a81a7c7 Working multiprocessing
# f b8881c7 Reset test utils
git log
# Last commit hash: 38cf1481537929a855dd5de156feacb8f257b388
git checkout test_unified_1.12.0rc3_perlmutter_try2
git cherry-pick 38cf1481537929a855dd5de156feacb8f257b388
git log
# Attempt to multithread the image tests (squashed version)
# The 3 commits from https://github.com/E3SM-Project/zppy/pull/755
# Bump to 3.1.0rc3 (#754)
# Edit tests/integration/utils.py:
# Now, we have to update *only* TEST_SPECIFICS
#
# TEST_SPECIFICS: Dict[str, Any] = {
# # These are custom environment_commands for specific tasks.
# # Never set these to "", because they will print the line
# # `environment_commands = ""` for the corresponding task,
# # thus overriding the value set higher up in the cfg.
# # That is, there will be no environment set.
# # (`environment_commands = ""` only redirects to Unified
# # if specified under the [default] task)
# "diags_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.12.0rc3_pm-cpu.sh",
# "global_time_series_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.12.0rc3_pm-cpu.sh",
# "pcmdi_diags_environment_commands": "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.12.0rc3_pm-cpu.sh",
# # This is the environment setup for other tasks.
# # Leave as "" to use the latest Unified environment.
# "environment_commands": "",
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": [
# "e3sm_diags",
# "mpas_analysis",
# "global_time_series",
# "ilamb",
# "pcmdi_diags",
# ],
# "unique_id": "test_e3sm_unified_1.12.0rc3_pm_20251031",
# }
git diff
# Diff looks right
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.12.0rc3_pm-cpu.sh
pytest tests/test_*.py
# 35 passed in 0.17s
python tests/integration/utils.py
# Specifically don't want to do these steps, because we're not in a dev environment or making code changes
# pre-commit run --all-files
# python -m pip install .
git add -A # Just to make it obvious if we add any new diffs
alias sq
# alias sq='sqa -u forsyth'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %8i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
sq
# No running jobs
# Good, that will make it easy to see when the jobs we're about to launch are done.
zppy -c tests/integration/generated/test_weekly_bundles_pm-cpu.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_pm-cpu.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_pm-cpu.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_pm-cpu.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_pm-cpu.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_pm-cpu.cfg
sq | wc -l
# Fri 10/31 ~12:00 => 144 - header row = 143 jobs
# Fri 10/31 13:30 => 144 - header row = 143 jobs (no change)
# Fri 10/31 16:50 => 144 - header row = 142 jobs (1 job ran)
# Mon 11/03 09:30 => 1 - header row = 0 jobs (all jobs finished)
# Picking back up, 11/03
cd ~/ez/zppy
git status
# branch test_unified_1.12.0rc3_perlmutter_try2
#
# Figure out what the output paths are
git grep "output =" tests/integration/generated/test_weekly_*_pm-cpu.cfg | cat
# tests/integration/generated/test_weekly_bundles_pm-cpu.cfg:output = "/global/cfs/cdirs/e3sm/forsyth/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v3.LR.historical_0051"
# tests/integration/generated/test_weekly_comprehensive_v2_pm-cpu.cfg:output = "/global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v2_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v2.LR.historical_0201"
# tests/integration/generated/test_weekly_comprehensive_v3_pm-cpu.cfg:output = "/global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v3.LR.historical_0051"
# tests/integration/generated/test_weekly_legacy_3.0.0_bundles_pm-cpu.cfg:output = "/global/cfs/cdirs/e3sm/forsyth/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v3.LR.historical_0051"
# tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_pm-cpu.cfg:output = "/global/cfs/cdirs/e3sm/forsyth/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v2.LR.historical_0201"
# tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_pm-cpu.cfg:output = "/global/cfs/cdirs/e3sm/forsyth/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v3.LR.historical_0051"
# Check on bundles status
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd ~/ez/zppy
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.12.0rc3_pm-cpu.sh # Re-activate
zppy -c tests/integration/generated/test_weekly_bundles_pm-cpu.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_pm-cpu.cfg
# Mon 11/03 9:33 => 5 - header row = 4 jobs
# Review finished runs
### v2 ###
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v2_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# ilamb_1980-1981.status:ERROR (3)
# ilamb_1982-1983.status:ERROR (3)
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# ilamb_1980-1981.status:ERROR (3)
# ilamb_1982-1983.status:ERROR (3)
### v3 ###
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# ilamb_1985-1988.status:ERROR (3)
# pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 44662856
# pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.status:ERROR (8)
# pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.status:ERROR (8)
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# ilamb_1985-1988.status:ERROR (3)
sq | wc -l
# Mon 11/03 9:38 => 5 - header row = 4 jobs (no change)
# Let's debug the errors while we wait for the bundles jobs to run.
# Notice that the erroring files match those of the Compy run.
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v2_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v2.LR.historical_0201/post/scripts
emacs ilamb_1980-1981.o44662745
# ImportError: libmpi.so.12: cannot open shared object file: No such file or directory
#
# Yep, it's the same error.
grep -l "cannot open shared object file" *.o*
# ilamb_1980-1981.o44662745
# ilamb_1982-1983.o44662746
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v2.LR.historical_0201/post/scripts
grep -l "cannot open shared object file" *.o*
# ilamb_1980-1981.o44662933
# ilamb_1982-1983.o44662937
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v3.LR.historical_0051/post/scripts
grep -l "cannot open shared object file" *.o*
# ilamb_1985-1988.o44662851
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v3.LR.historical_0051/post/scripts
grep -l "cannot open shared object file" *.o*
# ilamb_1985-1988.o44662980
# grep: tmp.ilamb_1985-1988.44662980.ojDZ: Is a directory
#
# So, Perlmutter needs the same update as Compy -- https://github.com/E3SM-Project/mache/pull/321
# Commented: https://github.com/E3SM-Project/mache/pull/321#issuecomment-3481760255
#
# Now, let's make the sure the pcmdi_diags error is the same
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v3.LR.historical_0051/post/scripts
grep -n "Observation file not found for" pcmdi_diags_*.o*
# pcmdi_diags_mean_climate_model_vs_obs_1985-1994.o44662852:10727:[Warning] Observation file not found for sfcWind (default)
# pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.o44662855:6:[Warning] Observation file not found for psl (alternate2)
# pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.o44662853:6:[Warning] Observation file not found for ts (alternate1)
#
# Again, same errors
sq | wc -l
# Mon 11/03 9:49 => 4 - header row = 3 jobs (1 job ran)
# Mon 11/03 10:02 => 2 - header row = 1 jobs (2 more jobs ran)
# Mon 11/03 10:06 => 1 - header row = 0 jobs (All jobs finished)
### bundles ###
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# ilamb_1985-1986.status:ERROR (3)
#
# We were expecting that, based on the above.
cd /global/cfs/cdirs/e3sm/forsyth/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc3_pm_20251031/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# ilamb_1985-1986.status:ERROR (3)
#
# We were expecting that, based on the above.
# Run tests:
cd ~/ez/zppy
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 0.90s, diffs appear to be expected, based on changes made during this development period
pytest tests/integration/test_campaign.py
# 6 failed in 2.88s, same note as above
pytest tests/integration/test_defaults.py
# 1 failed in 0.48s, same note as above
pytest tests/integration/test_last_year.py
# 1 passed in 0.37s
pytest tests/integration/test_bundles.py
# 2 passed in 0.42s
pytest tests/integration/test_images.py
# 1 failed in 705.99s (0:11:45)
# We needed to remove the tests that don't have expected results...
# Edit TEST_SPECIFICS in tests/integration/utils.py
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# #"weekly_legacy_3.0.0_bundles",
# #"weekly_legacy_3.0.0_comprehensive_v2",
# #"weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": [
# "e3sm_diags",
# "mpas_analysis",
# "global_time_series",
# "ilamb",
# #"pcmdi_diags",
# ],
# }
rm early_test_images_summary.md test_*.log
pytest tests/integration/test_images.py
# Cancel because the changes didn't save in VS code...
rm early_test_images_summary.md test_*.log
git status
git diff tests/integration/utils.py
# Good, the changes show up
pytest tests/integration/test_images.py
# 1 failed in 660.36s (0:11:00)
Test output: Complete resultsDiff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
Concise results:
Final notes (nearly idential to the final notes for Compy)
|
Beta Was this translation helpful? Give feedback.
11 replies
-
I'm not 100% certain, but I'm fairly confident. Claude appears to confirm: Looking at the error, the issue is clear: What went wrong:
Root cause: The observation dataset specified ( Looking at the
So Solutions:
The script is correctly detecting the problem and exiting with error code 6/8, but the underlying issue is missing or inaccessible observation data files. |
Beta Was this translation helpful? Give feedback.
-
|
I'm fairly sure we'd just need to get the correct observation paths for Perlmutter and Compy. |
Beta Was this translation helpful? Give feedback.
-
|
the obersvation paths for Perlmutter and Compy seems correct, except they have extra trailing |
Beta Was this translation helpful? Give feedback.
-
|
The paths as specified here: |
Beta Was this translation helpful? Give feedback.
-
|
In general, not having a trailing |
Beta Was this translation helpful? Give feedback.
-
|
@forsyth2 @chengzhuzhang : Apologies for the delayed response — I just saw your message in my email. The issue is indeed related to the observations. The NOAA20C dataset was processed and added by me to the e3sm_diags on Chrysalis, and it’s not part of the default dataset collection. I believe you can simply transfer the data from Chrysalis to the Perlmutter data sharing directory, which should resolve the problem. |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
thanks @zhangshixuan1987 ! yes, we should add that dataset into the input data directory /lcrc/group/e3sm/public_html/diagnostics/observations/Atm/time-series/. And plan to use these path for testing. Is time-series/NOAA-20C the only dataset being added? Could you also provide the processing script of this dataset, so that I can include in e3sm_diags for potential modification? |
Beta Was this translation helpful? Give feedback.
-
|
@zhangshixuan1987 Thanks! We'll need the data on both Perlmutter and Compy. I think if the data is in |
Beta Was this translation helpful? Give feedback.
-
|
yes, if the data is in /lcrc/group/e3sm/public_html/diagnostics/observations/Atm/time-series/, by running |
Beta Was this translation helpful? Give feedback.
-
|
@chengzhuzhang: Hi Jill, Yes, the NOAA20C dataset is one of the standard datasets used by PCMDI for extratropical modes variability analysis, primarily because it provides a long time series of sea-level pressure (PSL) data. I should also point out that the PCMDI mean climate diagnostics are derived from time series data as well, so the only dataset required for all PCMDI diagnostics is the corresponding time series data. I’ve placed the full directory containing my data processing setup for NOAA2OC at the following shared location on Chrysalis:
By the way, some of the observation datasets specifically prepared for PCMDI diagnostics (and potentially useful in the near future) are available here:
|
Beta Was this translation helpful? Give feedback.
7 replies
-
|
@forsyth2 I'm syncing files on Perlmutter. Could you help to update for Compy, the command should be: |
Beta Was this translation helpful? Give feedback.
-
|
Running now. |
Beta Was this translation helpful? Give feedback.
-
@chengzhuzhang Do I need to rerun or are those errors fine? |
Beta Was this translation helpful? Give feedback.
-
|
there is an error with rsync. I think you need to update .ssh/config for a proxy change logining into chrysalis, add something like following: Host chrysalis.lcrc.anl.gov chrysalis |
Beta Was this translation helpful? Give feedback.
-
|
Xylar completed the sync for Compy. For future reference, he added syncing instructions at https://docs.e3sm.org/mache/main/users_guide/sync/diags.html. |
Beta Was this translation helpful? Give feedback.
-
|
@forsyth2 @chengzhuzhang: E3SM typically reports monthly means with the timestamp corresponding to the end of the averaging period — e.g., a file labeled 2012-02-01 actually represents the January 2012 mean. This difference in time convention can lead to apparent mismatches when comparing model output with observational datasets that use the beginning or midpoint of the averaging period instead. Additionally, recent updates to the library handling date and time functions in matplotlib (or related dependencies) might also influence how timestamps are interpreted or displayed, potentially amplifying these differences. I’m not sure whether safeguards are already implemented in e3sm_diags or mpas_analysis to prevent such mismatches, but I wanted to share these observations from my experience with the PCMDI analysis for your reference. |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
@zhangshixuan1987 could you elaborate on: the differences between mpas_analysis and e3sm_diags? Did you notice inconsistency between both tools for the same field? |
Beta Was this translation helpful? Give feedback.
-
|
@chengzhuzhang: I am talking about the following differences noticed by @forsyth2 :
I personally did not perform a comparison between E3SM-Diags and MPAS-Analysis. My comments were based on some observations I made during the development of PCMDI-Diags. |
Beta Was this translation helpful? Give feedback.
-
|
@zhangshixuan1987 Sorry, the context of that statement was that the |
Beta Was this translation helpful? Give feedback.
-
|
Apologies for the confusion — please ignore my previous comment, as those issues turned out to be unrelated. |
Beta Was this translation helpful? Give feedback.
-
|
thanks for clarifying, Ryan and Shixuan! |
Beta Was this translation helpful? Give feedback.
-
|
Chrysaliszppy-interfacesTesting stepscd ~/ez/zppy-interfaces
git status
# branch test_unified_1.12.0rc3_chrysalis
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zppy-interfaces/commits/main
# Bump to 0.2.0rc2 => 5a88374fa18782afe8de4f015d0b4504c75e5d06
# Remember, zppy-interfaces didn't get a new RC
git checkout -b test_unified_1.12.0rc4_chrysalis 5a88374fa18782afe8de4f015d0b4504c75e5d06
git log
# Good, last commit: Bump to 0.2.0rc2 (#43)
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc4_chrysalis.sh
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 27.65s
pytest tests/unit/pcmdi_diags/test_*.py
# 7 passed, 1 warning in 12.60sThe warning has been noted at PCMDI/pcmdi_metrics#1330 (comment). Success zppyTesting steps# We're specifically testing E3SM Unified,
# so we don't have to worry about making dev environments for e3sm_diags or zppy-interfaces.
cd ~/ez/zppy
git status
# branch main
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zppy/commits/main
# Multithread image tests (#752)
# Fixes for rc4 (#755)
# Bump to 3.1.0rc3 (#754)
# The two most recent commits only touched `tests/`, so we can keep them.
git checkout -b test_unified_1.12.0rc4_chrysalis upstream/main
git log
# Good, matches https://github.com/E3SM-Project/zppy/commits/main
# Determine pcmdi_diags data pathFrom #746 (reply in thread): In #746 (comment), the NOAA20C dataset was specifically listed as required. Data was transferred after these comments were made. Let's confirm the most recent locations. ls /lcrc/group/e3sm/diagnostics/observations/Atm/time-series
# 20250310_HadISST ceres_ebaf_toa_v2.8 ERA5_Daily GPCP_v2.2 HadISST2 NOAA-20C
# 20250527_IMERG_Daily ceres_ebaf_toa_v4.0 ERA5_ext GPCP_v2.3 IMERG_Daily NOAA-20C_v2
# ceres_ebaf_surface_v2.8 ceres_ebaf_toa_v4.1 ERA-Interim GPCP_v3.2 ISLSCPII_GRDC NOAA-OLR_Daily
# ceres_ebaf_surface_v4.0 COREv2_Flux GPCP_1DD GSIM MERRA2 OAFlux
# ceres_ebaf_surface_v4.1 ERA5 GPCP_OAFLux HadISST MERRA2_Aerosols OMI-MLS
ls /lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series
# 20250310_HadISST ceres_ebaf_toa_v2.8 ERA5_Daily GPCP_v2.2 HadISST2 NOAA-20C
# 20250527_IMERG_Daily ceres_ebaf_toa_v4.0 ERA5_ext GPCP_v2.3 IMERG_Daily NOAA-20C_v2
# ceres_ebaf_surface_v2.8 ceres_ebaf_toa_v4.1 ERA-Interim GPCP_v3.2 ISLSCPII_GRDC NOAA-OLR_Daily
# ceres_ebaf_surface_v4.0 COREv2_Flux GPCP_1DD GSIM MERRA2 OAFlux
# ceres_ebaf_surface_v4.1 ERA5 GPCP_OAFLux HadISST MERRA2_Aerosols OMI-MLS
# These match now, good!
# Edit tests/integration/utils.py
# get_chyrsalis_expansions:
# "path_pcmdi_diags_obs_ts": "/lcrc/group/e3sm/diagnostics/observations/Atm/time-series",
# TEST_SPECIFICS:
# TEST_SPECIFICS: Dict[str, Any] = {
# # These are custom environment_commands for specific tasks.
# # Never set these to "", because they will print the line
# # `environment_commands = ""` for the corresponding task,
# # thus overriding the value set higher up in the cfg.
# # That is, there will be no environment set.
# # (`environment_commands = ""` only redirects to Unified
# # if specified under the [default] task)
# "diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc4_chrysalis.sh",
# "global_time_series_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc4_chrysalis.sh",
# "pcmdi_diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc4_chrysalis.sh",
# # This is the environment setup for other tasks.
# # Leave as "" to use the latest Unified environment.
# "environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc4_chrysalis.sh",
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": [
# "e3sm_diags",
# "mpas_analysis",
# "global_time_series",
# "ilamb",
# "pcmdi_diags",
# ],
# "unique_id": "test_e3sm_unified_1.12.0rc4_20251106",
# }
git diff
# Diff looks good
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc4_chrysalis.sh
pytest tests/test_*.py
# 35 passed in 1.27s
python tests/integration/utils.py
git status
# Good, only generated files changed
# Specifically don't want to do these steps, because we're not in a dev environment or making code changes:
# pre-commit run --all-files
# python -m pip install .
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
# Count the number of jobs
alias sq
# alias sq='sqa -u ac.forsyth2'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
sq | wc -l # Thu 11/06 11:00 => 144 - header row = 143 jobs
sq | wc -l # Thu 11/06 13:00 => 1 - header row = 0 jobs
# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc4_20251106/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc4_20251106/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Now, run bundles part 2
cd ~/ez/zppy
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
sq | wc -l # Thu 11/06 13:01 => 5 - header row = 4 jobs
sq | wc -l # Thu 11/06 13:38 => 1 - header row = 0 jobs
# Review finished runs
### v2 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_e3sm_unified_1.12.0rc4_20251106/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/test_e3sm_unified_1.12.0rc4_20251106/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc4_20251106/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/test_e3sm_unified_1.12.0rc4_20251106/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
### bundles ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc4_20251106/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc4_20251106/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd ~/ez/zppy
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 2.10s, diffs appear to be expected based on recent changes on `main` (change NCO call)
pytest tests/integration/test_campaign.py
# 6 passed in 2.41s
pytest tests/integration/test_defaults.py
# 1 passed in 0.36s
pytest tests/integration/test_last_year.py
# 1 passed in 0.31s
pytest tests/integration/test_bundles.py
# 2 passed in 0.24s
salloc --nodes=1 --partition=debug --time=00:30:00 --account=e3sm
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.12.0rc4_chrysalis.sh
pytest tests/integration/test_images.py
# Started Thu 11/06 13:43
# 1 failed in 1010.71s (0:16:50)
cat test_images_summary.mdResults table:
Final notes: zppy using E3SM Unified Once I have successful testing on Perlmutter and Compy, I will update expected results on all 3 machines. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Compyzppy-interfacesTesting stepscd ~/ez/zppy-interfaces
git status
# branch test_unified_1.12.0rc3_compy
# nothing to commit
git fetch upstream main
# https://github.com/E3SM-Project/zppy-interfaces/commits/main
# Bump to 0.2.0rc2 => 5a88374fa18782afe8de4f015d0b4504c75e5d06
# Remember, zppy-interfaces didn't get a new RC
git checkout -b test_unified_1.12.0rc4_compy 5a88374fa18782afe8de4f015d0b4504c75e5d06
git log
# Good, last commit: Bump to 0.2.0rc2 (#43)
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc4_compy.sh
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 16.33s
pytest tests/unit/pcmdi_diags/test_*.py
# 7 passed, 1 warning in 11.52sThe warning has been noted at PCMDI/pcmdi_metrics#1330 (comment). Success zppyTesting steps# We're specifically testing E3SM Unified,
# so we don't have to worry about making dev environments for e3sm_diags or zppy-interfaces.
cd ~/ez/zppy
git status
# branch test-compy-multiprocessing
# Handle uncommitted changes:
git add -A
git commit -m "Testing" --no-verify
# REMINDER: `git fetch upstream main` is NOT guaranteed to work on Compy!
# Checking out a branch from upstream/main might not actually
# base it on ALL the most recent commits!
# Not specifying a branch at all seems to be the best method:
git fetch upstream
# https://github.com/E3SM-Project/zppy/commits/main
# Multithread image tests (#752)
# Fixes for rc4 (#755)
# Bump to 3.1.0rc3 (#754)
# The two most recent commits only touched `tests/`, so we can keep them.
git checkout -b test_unified_1.12.0rc4_compy upstream/main
git log
# Good, matches https://github.com/E3SM-Project/zppy/commits/main
# Determine pcmdi_diags data path
ls /compyfs/diagnostics/observations/Atm/time-series/
# Good, has NOAA-20C
# Edit tests/integration/utils.py
# get_compy_expansions (no change needed):
# "path_pcmdi_diags_obs_ts": "/compyfs/diagnostics/observations/Atm/time-series/",
# TEST_SPECIFICS:
# TEST_SPECIFICS: Dict[str, Any] = {
# # These are custom environment_commands for specific tasks.
# # Never set these to "", because they will print the line
# # `environment_commands = ""` for the corresponding task,
# # thus overriding the value set higher up in the cfg.
# # That is, there will be no environment set.
# # (`environment_commands = ""` only redirects to Unified
# # if specified under the [default] task)
# "diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc4_compy.sh",
# "global_time_series_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc4_compy.sh",
# "pcmdi_diags_environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc4_compy.sh",
# # This is the environment setup for other tasks.
# # Leave as "" to use the latest Unified environment.
# "environment_commands": "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc4_compy.sh",
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": [
# "e3sm_diags",
# "mpas_analysis",
# "global_time_series",
# "ilamb",
# "pcmdi_diags",
# ],
# "unique_id": "test_e3sm_unified_1.12.0rc4_compy_20251106",
# }
git diff
# Diff looks good
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc4_compy.sh
pytest tests/test_*.py
# 35 passed in 0.42s
git add -A # So future changes stand out on `git status`
python tests/integration/utils.py
git status
# Good, only generated files changed
# Specifically don't want to do these steps, because we're not in a dev environment or making code changes:
# pre-commit run --all-files
# python -m pip install .
alias sq
# alias sq='sqa -u fors729'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p'
sq
# No running jobs
# Good, that will make it easy to see when the jobs we're about to launch are done.
zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_compy.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_compy.cfg
sq | wc -l
# Thu 11/06 11:32 => 133 - header row = 132 jobs
# Thu 11/06 13:02 => 2 - header row = 1 job
sq
# USER ACCOUNT NODE PARTITION JOBID ST REASON TIME TIME_LIMIT PRIORITY NAME
# fors729 e3sm 1 short 710371 PD Dependency 0:00 2:00:00 417 pcmdi_diags_synthetic_plots_model_vs_obs
# That's not good. We have a dependency with no remaining running jobs.
# First, let's check on the bundles status
cd /compyfs/fors729/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc4_compy_20251106/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc4_compy_20251106/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Now, run bundles part 2
cd ~/ez/zppy
zppy -c tests/integration/generated/test_weekly_bundles_compy.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_compy.cfg
sq | wc -l # Thu 11/06 13:04 => 6 - header row = 5 jobs (including the one waiting for a dependency)
sq | wc -l # Thu 11/06 13:44 => 2 - header row = 1 jobs (including the one waiting for a dependency)
sq # Get Job ID
scancel 710371 # Cancel taht job since it's not going to run anyway.
sq # No remaining jobs
# Review finished runs
### v2 ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v2_output/test_e3sm_unified_1.12.0rc4_compy_20251106/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/test_e3sm_unified_1.12.0rc4_compy_20251106/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
### v3 ###
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc4_compy_20251106/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 710371
# pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.status:ERROR (8)
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/test_e3sm_unified_1.12.0rc4_compy_20251106/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
### bundles ###
cd /compyfs/fors729/zppy_weekly_bundles_output/test_e3sm_unified_1.12.0rc4_compy_20251106/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /compyfs/fors729/zppy_weekly_legacy_3.0.0_bundles_output/test_e3sm_unified_1.12.0rc4_compy_20251106/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Let's take a look at the pcmdi_diags error
cd /compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc4_compy_20251106/v3.LR.historical_0051/post/scripts
cat pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.o710369That gives: ls /compyfs/fors729/zppy_weekly_comprehensive_v3_output/test_e3sm_unified_1.12.0rc4_compy_20251106/v3.LR.historical_0051/post/scripts/tmp.pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.710369.Xhfd/obs_link
# Empty
# It looks to me like we still don't have all the data transferred.
# atm was fixed, but not cpl.
cd ~/ez/zppy
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 5.60s, diffs appear to be expected based on changes made during this development period
pytest tests/integration/test_campaign.py
# 6 failed in 9.12s, same note as above
pytest tests/integration/test_defaults.py
# 1 failed in 1.04s, same note as above
pytest tests/integration/test_last_year.py
# 1 passed in 1.08s
pytest tests/integration/test_bundles.py
# 2 passed in 0.66s
# Edit tests/integration/utils.py
# TEST_SPECIFICS:
# Comment out the following since we don't have expected results for them yet:
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
#
# "pcmdi_diags",
git diff
# Diff looks good
salloc --nodes=1 --partition=short --time=00:30:00 --account=e3sm
source ~/.bash_profile
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc4_compy.sh
pytest tests/integration/test_images.py
# Started Thu 11/06 13:57
# tests/integration/test_images.py salloc: Job 710465 has exceeded its time limit and its allocation has been revoked.
# https://e3sm.org/model/running-e3sm/supported-machines/compy-pnnl/
# Max time for Short is 2 hours
# So, we can increase it.
salloc --nodes=1 --partition=short --time=01:30:00 --account=e3sm
source ~/.bash_profile # To set up my aliases
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.12.0rc4_compy.sh
pytest tests/integration/test_images.py
# Started Thu 11/06 14:33
# 1 failed in 939.93s (0:15:39)
# Not really sure why it hit the time limit before then...
cat test_images_summary.mdResults table:
Final notes:
|
Beta Was this translation helpful? Give feedback.
10 replies
-
|
@forsyth2 I have updated perlmutter with HadISST2 included, could you re-run test there, it should fix the failed test for pcmdi-diags. If so, we can do a Compy sync after. |
Beta Was this translation helpful? Give feedback.
-
|
Re-running. |
Beta Was this translation helpful? Give feedback.
-
|
@zhangshixuan1987 Thanks, that made it further. We completed the variability metrics, but synthetic plots still don't have sufficient data. We use these paths on Chrysalis: But the equivalent paths don't exist on Perlmutter: We're going to need to sync that too. |
Beta Was this translation helpful? Give feedback.
-
|
thanks a lot @forsyth2 . |
Beta Was this translation helpful? Give feedback.
-
|
@forsyth2 @chengzhuzhang:
@chengzhuzhang — could you please sync this data to Perlmutter and Compy? Please let me know if the issue persists. Thank you! |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Chrysalis
zppy-interfaces
Unit testing steps
Success
zppy
For reference notes, see docs or the last weekly testing log. Note that the former is somewhat outdated.
Testing steps
Complete results
Diff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
Note image diff grids can not yet be constructed automatically.
Analysis: overall,
rc1looks mostly ok to me. Just a few notes below:mpas_analysis, but they are mostly minor, perhaps caused by something as simple asmatplotlibversion changes. An alternative explanation is the differences are to be expected, since this is the first time testing with a latermpas_analysisthan the one from Spring of this year. @xylar @andrewdnolan Let me know what you think here.Note: to review diffs, go to the "Diff link" in the table above.
mismatched_images.txthas a list of mismatched images.image_diff_grid.pdfshows some of the diffs in an overview PDF, but that unfortunately has a bug where it won't display all the diffs. To individually inspect diffs, drill down into the subdirectories.I didn't run the
legacy_3.0.0_comprehensive_v3tests because of the known issue observed on the last "weekly" test regarding backwards compatibility withinfer_path_parameters = False. @chengzhuzhang Let me know how we should proceed with this (see link for tradeoff discussion).I will update expected results after we test
rc2, becausepcmdi_diagsisn't working properly in thisrc. Note this was fully expected; fixes were merged for inclusion inrc2.Beta Was this translation helpful? Give feedback.
All reactions