Add main branch testing automation by forsyth2 · Pull Request #774 · E3SM-Project/zppy

forsyth2 · 2026-01-23T20:16:45Z

Summary

Objectives:

Automate the testing workflow. Specifically, using the latest workflow, as described in the 2025-12-30 test.

Issue resolution:

Closes Further automate tests #516

Select one: This pull request is...

a bug fix: increment the patch version
a small improvement: increment the minor version
a new feature: increment the minor version
an incompatible (non-backwards compatible) API change: increment the major version

Small Change

To merge, I will use "Squash and merge". That is, this change should be a single commit.
Logic: I have visually inspected the entire pull request myself.
Pre-commit checks: All the pre-commits checks have passed.

forsyth2 · 2026-01-24T01:31:20Z

Current status: made it to the end of phase 1 (launched jobs), but the MPAS-Analysis jobs didn't have sufficient wall times allocated. (Note: because of #764, it is usually better to rerun from scratch in this case).

Action items:

Rerun automated workflow, iterating until it reaches completion without failing.

forsyth2 · 2026-01-27T02:28:44Z

Using the latest commit (88cdcb2), I've made it to phase 2. It turns out I needed to update LEGACY_BUNDLES_OUTPUT to be "${OUTPUT_WORKSPACE}/zppy_weekly_legacy_3.0.0_bundles_output/${UNIQUE_ID}/v3.LR.historical_0051/post/scripts" (it was missing a / before), as that caused phase 2 to crash. After making that fix, I restarted from the phase 2 start point, but ran into the following error on job launch:

Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/test-zppy-test-fixes-20260126_run4/bin/zppy", line 7, in <module>
    sys.exit(main())
             ~~~~^^
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/test-zppy-test-fixes-20260126_run4/lib/python3.14/site-packages/zppy/__main__.py", line 43, in main
    user_config, plugins = _handle_plugins(user_config, default_config, args)
                           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/test-zppy-test-fixes-20260126_run4/lib/python3.14/site-packages/zppy/__main__.py", line 106, in _handle_plugins
    if "plugins" in user_config["default"].keys():
                    ~~~~~~~~~~~^^^^^^^^^^^
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/test-zppy-test-fixes-20260126_run4/lib/python3.14/site-packages/configobj/__init__.py", line 549, in __getitem__
    val = dict.__getitem__(self, key)
KeyError: 'default'

forsyth2 · 2026-01-30T22:43:13Z

I've run through phase 3 now. There are a number of issues to resolve before fully switching from the manual testing workflow to this one.

Immediate KeyError on later zppy runs

I cannot get Phase 2 to work in an automated manner. I can launch zppy -c manually but running from Phase 2, I get:

[2026-01-30 14:52:37] Submitting bundles part 2...
2026-01-30 14:52:38,000 [INFO]: __main__.py(main:33) >> For help, please see https://e3sm-project.github.io/zppy. Ask questions at https://github.com/E3SM-Project/zppy/discussions/categories/q-a.
2026-01-30 14:52:38,000 [INFO]: __main__.py(main:33) >> For help, please see https://e3sm-project.github.io/zppy. Ask questions at https://github.com/E3SM-Project/zppy/discussions/categories/q-a.
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/test-zppy-test-fixes-20260126_run4/bin/zppy", line 7, in <module>
    sys.exit(main())
             ~~~~^^
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/test-zppy-test-fixes-20260126_run4/lib/python3.14/site-packages/zppy/__main__.py", line 43, in main
    user_config, plugins = _handle_plugins(user_config, default_config, args)
                           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/test-zppy-test-fixes-20260126_run4/lib/python3.14/site-packages/zppy/__main__.py", line 106, in _handle_plugins
    if "plugins" in user_config["default"].keys():
                    ~~~~~~~~~~~^^^^^^^^^^^
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/test-zppy-test-fixes-20260126_run4/lib/python3.14/site-packages/configobj/__init__.py", line 549, in __getitem__
    val = dict.__getitem__(self, key)
KeyError: 'default'

Status file errors

I'm not sure how these could possibly be real failures (i.e., not caused by this branch), considering the last change to zppy was on 12/23 (according to the commit log), which was before the 12/30 test.

To be certain though, I will have to do the usual workflow and see if the same thing happens or not.

/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_20260126_run4/v2.LR.historical_0201/post/scripts
global_time_series_1980-1990.status:WAITING 1135590
mpas_analysis_ts_1980-1984_climo_1980-1984.status:RUNNING 1135588
mpas_analysis_ts_1980-1990_climo_1985-1990.status:WAITING 1135589

/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_test_20260126_run4/v2.LR.historical_0201/post/scripts
global_time_series_1980-1990.status:WAITING 1135670
mpas_analysis_ts_1980-1984_climo_1980-1984.status:RUNNING 1135668
mpas_analysis_ts_1980-1990_climo_1985-1990.status:WAITING 1135669

/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260126_run4/v3.LR.historical_0051/post/scripts
pcmdi_diags_synthetic_plots_model_vs_obs.status:WAITING 1135638
pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.status:ERROR (14)

MPAS Analysis errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_20260126_run4/v2.LR.historical_0201/post/scripts
cat mpas_analysis_ts_1980-1984_climo_1980-1984.o1135588

gives:

Output

===== SET UP MPAS-ANALYSIS DIRECTORY STRUCTURE =====

sending incremental file list

sent 62 bytes  received 12 bytes  148.00 bytes/sec
total size is 0  speedup is 0.00
sending incremental file list

sent 62 bytes  received 12 bytes  148.00 bytes/sec
total size is 0  speedup is 0.00
sending incremental file list

sent 62 bytes  received 12 bytes  148.00 bytes/sec
total size is 0  speedup is 0.00
cp: cannot stat 'cache/timeseries/mpasIndexOcean.nc': No such file or directory
cp: cannot stat 'cache/timeseries/mpasTimeSeriesOcean.nc': No such file or directory
cp: cannot stat 'cache/timeseries/seaIceAreaVolNH.nc': No such file or directory
cp: cannot stat 'cache/timeseries/seaIceAreaVolSH.nc': No such file or directory

===== RUN MPAS-ANALYSIS =====

Detected E3SM supported machine: chrysalis
Using the following config files:
   /lcrc/soft/climate/e3sm-unified/e3smu_1_12_0/chrysalis/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/mpas_analysis/default.cfg
   /lcrc/soft/climate/e3sm-unified/e3smu_1_12_0/chrysalis/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/mache/machines/chrysalis.cfg
   /lcrc/soft/climate/e3sm-unified/e3smu_1_12_0/chrysalis/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/mpas_analysis/configuration/chrysalis.cfg
   /lcrc/soft/climate/e3sm-unified/e3smu_1_12_0/chrysalis/conda/envs/e3sm_unified_1.12.0_compute/lib/python3.13/site-packages/mpas_analysis/__main__.py
   /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_20260126_run4/v2.LR.historical_0201/post/analysis/mpas_analysis/cfg/mpas_analysis_ts_1980-1984_climo_1980-1984.cfg
copying /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_20260126_run4/v2.LR.historical_0201/post/analysis/mpas_analysis/cfg/mpas_analysis_ts_1980-1984_climo_1980-1984.cfg to HTML dir.
Atlantic
 * merging features
 * combining features
Pacific
 * merging features
 * combining features
Indian
 * merging features
 * combining features
Arctic
 * merging features
 * combining features
Southern_Ocean
 * merging features
 * combining features
Mediterranean
 * merging features
 * combining features

[1769470584.390866] [chr-0159:3633137:0]     ucp_context.c:1951 UCX  WARN  UCP API version is incompatible: required >= 1.16, actual 1.14.0 (loaded from /usr/lib64/libucp.so.0)
[1769470584.472043] [chr-0159:3633137:0]     ucp_context.c:1951 UCX  WARN  UCP API version is incompatible: required >= 1.16, actual 1.14.0 (loaded from /usr/lib64/libucp.so.0)
 Starting weight generation with these inputs: 
   Source File: /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_20260126_run4/v2.LR.historical_0201/post/analysis/mpas_analysis/ts_1980-1984_climo_1980-1984/mapping/tmpilna9gk5/src_mesh.nc
   Destination File: /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_20260126_run4/v2.LR.historical_0201/post/analysis/mpas_analysis/ts_1980-1984_climo_1980-1984/mapping/tmpilna9gk5/dst_mesh.nc
   Weight File: /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_20260126_run4/v2.LR.historical_0201/post/scripts/../analysis/mpas_analysis/ts_1980-1984_climo_1980-1984/mapping/map_EC30to60E2r2_to_AntShip_transects_5km_bilinear.nc
   Source File is in SCRIP format
   Source Grid is a regional grid
   Source Grid is an unstructured grid
   Use the center coordinates of the source grid to do the regrid
   Destination File is in SCRIP format
   Destination Grid is a regional grid
   Destination Grid is an unstructured grid
   Use the center coordinates of the destination grid to do the regrid
   Regrid Method: bilinear
   Pole option: NONE
   Ignore unmapped destination points
   Output weight file in NetCDF4 file format
   Line Type: cartesian
   Norm Type: dstarea
   Extrap. Method: none

srun: error: chr-0159: task 0: Segmentation fault
srun: Job step aborted: Waiting up to 92 seconds for job step to finish.
slurmstepd: error: *** STEP 1135588.0 ON chr-0159 CANCELLED AT 2026-01-26T19:34:17 DUE TO TIME LIMIT ***
slurmstepd: error: *** JOB 1135588 ON chr-0159 CANCELLED AT 2026-01-26T19:34:17 DUE TO TIME LIMIT ***
srun: got SIGCONT
srun: forcing job termination

I had extended the time limit (here) because it was hitting the time limit, but it turns out the time limit isn't the problem at all. The job stays active even after the segmentation fault, but doesn't actually do anything to recover from it.

Notes:

global_time_series never runs because this job is a dependency.
legacy v2 has the equivalent error.

PCMDI Diags errors

cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260126_run4/v3.LR.historical_0051/post/scripts
cat pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.o1135637

gives:

Output

create_links_ts: linking from /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260126_run4/v3.LR.historical_0051/post/atm/180x360_aave/cmip_ts/monthly to ts
ncap2: WARNING ncap_def_dim(): dim "bnds" - already exists in input/output.
Linking observational data into obs_link...
Linking observational data using SLURM...
Running a zi-pcmdi command: zi-pcmdi-link-observation --model_name_ref obs.historical.put_model_here.00 --tableID_ref Amon --vars=psl --obs_sets alternate2 --obs_ts /lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series --obstmp_dir obs_link --debug false
[Info] Linked psl → obs_link/obs.historical.NOAA-20C.00.Amon.psl.183601-201512.nc

real	0m26.226s
user	0m15.557s
sys	0m0.558s
create_links_ts_obs: linking from /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260126_run4/v3.LR.historical_0051/post/scripts/tmp.pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.1135637.Jp6U/obs_link to ts_ref
create_links_ts_obs: checking if nc file: obs.historical.NOAA-20C.00.Amon.psl.183601-201512.nc
create_links_ts_obs: processing /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260126_run4/v3.LR.historical_0051/post/scripts/tmp.pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.1135637.Jp6U/obs_link/obs.historical.NOAA-20C.00.Amon.psl.183601-201512.nc
ncrcat successful
Adding missing calendar attribute to time...
ncatted successful
ncap2: WARNING ncap_def_dim(): dim "bnds" - already exists in input/output.
ncap2 successful
About to create parameterfile.py, which will be passed in with -p
The current directory is: /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260126_run4/v3.LR.historical_0051/post/scripts/tmp.pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.1135637.Jp6U

===== RUN PCMDI DIAGS =====

About to set up a zi-pcmdi command
The current directory is: /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260126_run4/v3.LR.historical_0051/post/scripts/tmp.pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.1135637.Jp6U
Running a zi-pcmdi command: zi-pcmdi-variability-modes --num_workers 24 --multiprocessing True --subsection variability_modes_atm --climo_ts_dir_primary ts --climo_ts_dir_ref ts_ref --model_name e3sm.historical.v3-LR.0051 --model_tableID Amon --figure_format png  --run_type model_vs_obs --obs_sets alternate2 --model_name_ref obs.historical.put_model_here.00 --vars psl --tableID_ref Amon --generate_sftlf True --case_id v20260126 --results_dir model_vs_obs --debug false --var_modes NAM,NAO,PNA,NPO,SAM,PSA1,PSA2
The current directory is: /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260126_run4/v3.LR.historical_0051/post/scripts/tmp.pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.1135637.Jp6U
2026-01-26 18:07:46,241 [INFO]: pcmdi_setup.py(build_catalogues:76) >> DataCatalogueBuilder's vars: ['psl']
2026-01-26 18:07:46,241 [INFO]: pcmdi_setup.py(build_catalogues:78) >> Building catalogue for psl
2026-01-26 18:07:46,241 [INFO]: pcmdi_setup.py(build_catalogues:80) >> Looking for psl, the base var name of psl
2026-01-26 18:07:46,241 [INFO]: pcmdi_setup.py(build_catalogues:81) >> Finding test files in ts
2026-01-26 18:07:46,241 [INFO]: pcmdi_setup.py(build_catalogues:85) >> Finding ref files in ts_ref
2026-01-26 18:07:46,244 [INFO]: pcmdi_setup.py(build_catalogues:96) >> Extracting & assigining metadata for psl, the base var name of psl
2026-01-26 18:07:46,244 [INFO]: pcmdi_setup.py(_extract_metadata:135) >> Extracting metadata from e3sm.historical.v3-LR.0051.Amon.psl.198501-199412.nc, dervied from ts/e3sm.historical.v3-LR.0051.Amon.psl.198501-199412.nc
2026-01-26 18:07:46,244 [INFO]: pcmdi_setup.py(_extract_metadata:148) >> Parsing e3sm.historical.v3-LR.0051.Amon.psl.198501-199412.nc, determined mip=e3sm, exp=historical, model=v3-LR, realization=0051, tableID=Amon, yymms=198501, yymme=199412
2026-01-26 18:07:46,246 [INFO]: pcmdi_setup.py(_extract_metadata:135) >> Extracting metadata from obs.historical.NOAA-20C.00.Amon.psl.198501-199412.nc, dervied from ts_ref/obs.historical.NOAA-20C.00.Amon.psl.198501-199412.nc
2026-01-26 18:07:46,246 [INFO]: pcmdi_setup.py(_extract_metadata:148) >> Parsing obs.historical.NOAA-20C.00.Amon.psl.198501-199412.nc, determined mip=obs, exp=historical, model=NOAA-20C, realization=00, tableID=Amon, yymms=198501, yymme=199412
2026-01-26 18:07:46,246 [INFO]: pcmdi_setup.py(build_catalogues:111) >> test_files=['ts/e3sm.historical.v3-LR.0051.Amon.psl.198501-199412.nc']
2026-01-26 18:07:46,246 [INFO]: pcmdi_setup.py(build_catalogues:112) >> ref_files=['ts_ref/obs.historical.NOAA-20C.00.Amon.psl.198501-199412.nc']
2026-01-26 18:07:46,246 [INFO]: pcmdi_setup.py(build_catalogues:114) >> test_files[0]=ts/e3sm.historical.v3-LR.0051.Amon.psl.198501-199412.nc
2026-01-26 18:07:46,246 [INFO]: pcmdi_setup.py(build_catalogues:116) >> ref_files[0]=ts_ref/obs.historical.NOAA-20C.00.Amon.psl.198501-199412.nc
2026-01-26 18:07:46,246 [INFO]: pcmdi_setup.py(_save_catalogue:184) >> Saving catalogue pcmdi_diags/ts_variability_modes_atm_catalogue.json, absolute path /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260126_run4/v3.LR.historical_0051/post/scripts/tmp.pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.1135637.Jp6U/pcmdi_diags/ts_variability_modes_atm_catalogue.json
2026-01-26 18:07:46,248 [INFO]: pcmdi_setup.py(_save_catalogue:184) >> Saving catalogue pcmdi_diags/ts_ref_variability_modes_atm_catalogue.json, absolute path /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260126_run4/v3.LR.historical_0051/post/scripts/tmp.pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.1135637.Jp6U/pcmdi_diags/ts_ref_variability_modes_atm_catalogue.json
2026-01-26 18:07:47,798 [WARNING]: bounds.py(create_bounds:983) >> The 'lat' coordinate variable is missing a 'units' attribute. Assuming 'units' is 'degrees_north'.
2026-01-26 18:07:47,998 [WARNING]: bounds.py(create_bounds:983) >> The 'lat' coordinate variable is missing a 'units' attribute. Assuming 'units' is 'degrees_north'.
2026-01-26 18:07:48,017 [INFO]: pcmdi_setup.py(set_up:354) >> out_path=model_vs_obs/%(group_type)
2026-01-26 18:07:48,042 [INFO]: utils.py(run_parallel_jobs:61) >> Running 7 subprocesses...
2026-01-26 18:07:59,224 [ERROR]: utils.py(run_parallel_jobs:67) >> ERROR: Process failed: 'variability_modes_driver.py -p parameterfile.py --variability_mode NAM --eofn_mod 1 --eofn_obs 1 --varOBS psl --osyear 1985 --oeyear 1994 --reference_data_name NOAA-20C --reference_data_path ts_ref/obs.historical.NOAA-20C.00.Amon.psl.198501-199412.nc --case_id v20260126'
Error: /gpfs/fs1/home/ac.forsyth2/miniforge3/envs/test-zi-main-20260126_run4/lib/python3.13/site-packages/xcdat/temporal.py:2223: UserWarning: The 'unit' keyword is only used when the Timedelta input is an integer or float, not timedelta64. To specify the storage unit of the output use `td.as_unit(unit)`
  median_delta = pd.to_timedelta(np.median(time_deltas), unit="ns")
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/test-zi-main-20260126_run4/bin/variability_modes_driver.py", line 338, in <module>
    obs_timeseries_season = adjust_timeseries(
        obs_timeseries, obs_var, mode, season, regions_specs, RmDomainMean
    )
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/test-zi-main-20260126_run4/lib/python3.13/site-packages/pcmdi_metrics/variability_mode/lib/adjust_timeseries.py", line 36, in adjust_timeseries
    ds_anomaly = get_anomaly_timeseries(ds, data_var, season)
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/test-zi-main-20260126_run4/lib/python3.13/site-packages/pcmdi_metrics/variability_mode/lib/adjust_timeseries.py", line 75, in get_anomaly_timeseries
    ds_anomaly_all_seasons = ds_anomaly.temporal.departures(
        data_var,
    ...<2 lines>...
        season_config={"dec_mode": "DJF", "drop_incomplete_djf": True},
    )
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/test-zi-main-20260126_run4/lib/python3.13/site-packages/xcdat/temporal.py", line 869, in departures
    ds = self._preprocess_dataset(ds)
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/test-zi-main-20260126_run4/lib/python3.13/site-packages/xcdat/temporal.py", line 1219, in _preprocess_dataset
    ds = self._shift_djf_decembers(ds)
  File "/gpfs/fs1/home/ac.forsyth2/miniforge3/envs/test-zi-main-20260126_run4/lib/python3.13/site-packages/xcdat/temporal.py", line 1364, in _shift_djf_decembers
    time_coords.values[dec_indexes] = [
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
ValueError: assignment destination is read-only
Land mask estimated using regionmask method.
Land mask estimated using regionmask method.
Execution failed: Subprocess failed: variability_modes_driver.py -p parameterfile.py --variability_mode NAM --eofn_mod 1 --eofn_obs 1 --varOBS psl --osyear 1985 --oeyear 1994 --reference_data_name NOAA-20C --reference_data_path ts_ref/obs.historical.NOAA-20C.00.Amon.psl.198501-199412.nc --case_id v20260126
successfully finish all jobs....

real	0m24.830s
user	0m32.299s
sys	0m2.342s

===== COPY FILES TO WEB SERVER =====

rsync: [sender] link_stat "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260126_run4/v3.LR.historical_0051/post/scripts/tmp.pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.1135637.Jp6U/model_vs_obs" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.4.1]

Relevant lines, though I'm not sure why they aren't working:

ValueError: assignment destination is read-only
rsync: [sender] link_stat "/lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_20260126_run4/v3.LR.historical_0051/post/scripts/tmp.pcmdi_diags_variability_modes_atm_model_vs_obs_1985-1994.1135637.Jp6U/model_vs_obs" failed: No such file or directory (2)

Notes:

pcmdi_diags_synthetic_plots_model_vs_obs never runs because this job is a dependency.

forsyth2 · 2026-01-31T03:01:30Z

After running the standard manual weekly test, I've confirmed the mpas_analysis error is an issue with this branch but that the pcmdi_diags error is a real issue.

That means this branch for automated testing has two major issues to resolve:

The mpas_analysis segmentation fault
The KeyError when running Bundles Part 2

forsyth2 · 2026-04-02T18:13:45Z

#803 has some helper functions that can at least partially automate the process until this more complete automation is merged.

Add main branch testing automation

48dd737

forsyth2 self-assigned this Jan 23, 2026

forsyth2 added Testing Files in `tests` modified semver: small improvement Small improvement (will increment patch version) labels Jan 23, 2026

forsyth2 added 2 commits January 23, 2026 19:22

Changes tested

c9eee0d

Further changes to test

6b42def

Changes to test 20260126

ed8e6fc

forsyth2 force-pushed the automate-main-branch-testing branch from f33b563 to ed8e6fc Compare January 26, 2026 20:44

forsyth2 mentioned this pull request Jan 26, 2026

Test fixes #769

Merged

7 tasks

forsyth2 added 2 commits January 26, 2026 16:29

Changes to test 20260126_v2

a36c71f

Changes to test 20260126_v3

88cdcb2

Updates as of 20260130

f8bb71a

Add activation command

b35347d

forsyth2 mentioned this pull request Jan 31, 2026

[Bug]: pcmdi_diags_variability_modes_atm fails with assignment destination is read-only #777

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add main branch testing automation#774

Add main branch testing automation#774
forsyth2 wants to merge 8 commits intomainfrom
automate-main-branch-testing

forsyth2 commented Jan 23, 2026

Uh oh!

forsyth2 commented Jan 24, 2026

Uh oh!

forsyth2 commented Jan 27, 2026

Uh oh!

forsyth2 commented Jan 30, 2026

Uh oh!

forsyth2 commented Jan 31, 2026

Uh oh!

forsyth2 commented Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

forsyth2 commented Jan 23, 2026

Summary

Small Change

Uh oh!

forsyth2 commented Jan 24, 2026

Uh oh!

forsyth2 commented Jan 27, 2026

Uh oh!

forsyth2 commented Jan 30, 2026

Immediate KeyError on later zppy runs

Status file errors

MPAS Analysis errors

PCMDI Diags errors

Uh oh!

forsyth2 commented Jan 31, 2026

Uh oh!

forsyth2 commented Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant