Add debugging lines for rgn #35
Conversation
|
I added these debugging lines and then ran I then ran |
|
@czender Is there anything in the latest NCO (i.e., the one used in E3SM-Project/zppy#717) that would cause @chengzhuzhang FYI ^^^ |
|
@forsyth2 I'm looking at the global mean time-series file: /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/weekly_test_20250926/v2.LR.historical_0201/post/lnd/glb/ts/monthly/5yr/LAISHA_198001_198412.nc. I don't think global time series operation was applied successfully: I'm also attaching the log file for ncclimo for troubleshooting: |
|
I'm looking into this now... |
|
@forsyth2 and @chengzhuzhang Thanks for reporting this. I can reproduce the error on v2 ELM data. There's apparently a bug in the new automated 3D variable rejector that only manifests on bigrid simulations, i.e., |
|
Great, thanks @czender! |
|
@forsyth2 Here's a demonstration that the 3D checker works for EAMv2,v3 and ELMv3 but fails for ELMv2: It also works on MPAS... Code is now rebuilt on Chrysalis, Perlmutter, and Andes. |
|
@chengzhuzhang It sounds like @czender already has a fix, but if you still want to take a look at the logs: Here's the relevant directories:
|
Thanks @czender! It seems promising; I was just testing a PR (E3SM-Project/zppy#722) rebased off |
Never mind on that; that turns out to be because of different error handling in #26. (Although, that does mean this error-handling PR won't have to merge). |
Testing processcd ~/ez/zppy-interfaces
git status
# Check for uncommitted changes
git checkout fix-gts-rgn
# We won't actually merge this branch.
# But let's keep the debugging lines since
# we don't have the error handling from the global time series split PR yet.
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zi-rgn-error-20250929-try2
conda activate zi-rgn-error-20250929-try2
pre-commit run --all-files
python -m pip install .
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 25.80s
cd ~/ez/zppy
git status
# Check for uncommitted changes
git fetch upstream main
git checkout -b fix-gts-rgn-try2 upstream/main
rm -rf build
conda clean --all --y # 16:10
conda env create -f conda/dev.yml -n zppy-rgn-error-20250929-try2
conda activate zppy-rgn-error-20250929-try2
pytest tests/test_*.py
# 25 passed in 0.53s
# Edit tests/integration/utils.py:
# TEST_SPECIFICS: Dict[str, Any] = {
# "diags_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate e3sm-diags-main-20250926",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-rgn-error-20250929-try2",
# "cfgs_to_run": [
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# ],
# "tasks_to_run": ["global_time_series"],
# "unique_id": "rgn_error_test_20250929_try2",
# }
python tests/integration/utils.py
pre-commit run --all-files
python -m pip install .
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
# ~40 min to run
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/rgn_error_test_20250929_try2/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# No errors
cd ~/ez/zppy
pytest tests/integration/test_images.py
# pytest tests/integration/test_images.py
cat test_images_summary.mdDiff subdir is where to find the lists of missing/mismatched images, the image diff grid, and the individual diffs.
v3 just failed because I only ran v2, so it works fine! @czender Thanks, this works in my short v2-only test! |
|
@czender The tests now pass on I'm going to close this PR since #26 will add something similar. |
Summary
Objectives:
KeyErrorencountered on the 2025-09-26 weekly testing (zppy weekly testing log zppy#634 (comment)).Select one: This pull request is...
Small Change