[Bug]: "SVD did not converge in Linear Least Squares" in `pcmdi_diags`

[forsyth2]

What happened?

Initially reported as part of the 2026-04-08 Compy test using a dev environment for zppy and E3SM Unified 1.13.0rc4 for zppy-interfaces.

Copying comment from that thread:

---

From /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts/pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.o750968:

Excerpt from output file

2026-04-08 17:39:51,216 [ERROR]: utils.py(run_parallel_jobs:67) >> ERROR: Process failed: 'variability_modes_driver.py -p parameterfile.py --variability_mode PDO --eofn_mod 1 --eofn_obs 1 --varOBS ts --osyear 1985 --oeyear 1994 --reference_data_name HadISST2 --reference_data_path ts_ref/obs.historical.HadISST2.00.Amon.ts.198501-199412.nc --case_id v20260408'
Error: /qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc4/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/pcmdi_metrics/utils/land_sea_mask.py:216: UserWarning: landfrac is not provided thus generated using the 'create_land_sea_mask' function
  warnings.warn(
Traceback (most recent call last):
  File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc4/compy/pixi/.pixi/envs/default/bin/variability_modes_driver.py", line 379, in <module>
    ) = linear_regression_on_globe_for_teleconnection(
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        pc_obs[season],
        ^^^^^^^^^^^^^^^
    ...<5 lines>...
        debug=debug,
        ^^^^^^^^^^^^
    )
    ^
  File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc4/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/pcmdi_metrics/variability_mode/lib/eof_analysis.py", line 270, in linear_regression_on_globe_for_teleconnection
    slope, intercept = linear_regression(pc, ds[data_var], debug=debug)
                       ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc4/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/pcmdi_metrics/variability_mode/lib/eof_analysis.py", line 327, in linear_regression
    slope_1d, intercept_1d = np.polyfit(np.array(x), np.array(y_2d), 1)
                             ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc4/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/numpy/lib/_polynomial_impl.py", line 675, in polyfit
    c, resids, rank, s = lstsq(lhs, rhs, rcond)
                         ~~~~~^^^^^^^^^^^^^^^^^
  File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc4/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/numpy/linalg/_linalg.py", line 2539, in lstsq
    x, resids, rank, s = _umath_linalg.lstsq(a, b, rcond,
                         ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
                                             signature=signature)
                                             ^^^^^^^^^^^^^^^^^^^^
  File "/qfs/projects/ops/rh7/apps/E3SM/conda_envs/e3smu_1_13_0rc4/compy/pixi/.pixi/envs/default/lib/python3.13/site-packages/numpy/linalg/_linalg.py", line 157, in _raise_linalgerror_lstsq
    raise LinAlgError("SVD did not converge in Linear Least Squares")
numpy.linalg.LinAlgError: SVD did not converge in Linear Least Squares
Land mask estimated using regionmask method.
Land mask estimated using regionmask method.
Execution failed: Subprocess failed: variability_modes_driver.py -p parameterfile.py --variability_mode PDO --eofn_mod 1 --eofn_obs 1 --varOBS ts --osyear 1985 --oeyear 1994 --reference_data_name HadISST2 --reference_data_path ts_ref/obs.historical.HadISST2.00.Amon.ts.198501-199412.nc --case_id v20260408
successfully finish all jobs....

real	0m28.414s
user	0m50.263s
sys	0m6.089s
verifying deployed e3sm-unified version...
  Verified version 1.13.0rc4.
loading compute pixi env...
   pixi env loaded.
activating spack env...
   spack env activated.
loading e3sm-unified environment variables...
   e3sm-unified environment variables loaded.

===== COPY FILES TO WEB SERVER =====

rsync: [sender] link_stat "/compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts/tmp.pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.750968.EGYm/model_vs_obs" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.4.1]

Notably numpy.linalg.LinAlgError: SVD did not converge in Linear Least Squares. (Also UserWarning: landfrac is not provided thus generated using the 'create_land_sea_mask' function, but I'm not sure how relevant that is).

@zhangshixuan1987 do you know why this error would occur in variability_modes_cpl?

• Is the underlying data bad or insufficient?
• Is it a bug?
  • We haven't really changed anything in the pcmdi_diags code since the release of E3SM Unified 1.12.0 though. I'm not seeing any pcmdi_diags-relevant commits in zppy or zppy-interfaces since that release.
  • Furthermore, pcmdi_diags didn't have this issue on the 2026-03-31 Chrysalis test, which implies the error has been introduced in the last ~10 days (or because we're now testing on Compy rather than Chrysalis).

You can see the full provenance zppy cfg here: /compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/scripts/provenance.20260409_002040_555380.cfg

I'll also reproduce the pcmdi_diags section here:

pcmdi_diags section

[pcmdi_diags]
active = True
environment_commands_secondary = "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh"
walltime = "2:00:00"
model_name = 'e3sm.historical.v3-LR.0051'
model_tableID = 'Amon'
e3sm_to_cmip_atm_subsection = "atm_monthly_180x360_aave" # Can be inferred if needed
multiprocessing = True
num_workers = 24
obs_ts = '/compyfs/diagnostics/observations/Atm/time-series/'
pcmdi_debug = False
ts_num_years = 2
ts_years = "1985-1994",

  [[ mean_climate ]]
  current_set = "mean_climate" # Can be inferred if needed
  obs_sets = "default"
  ref_final_yr = 1994
  ref_start_yr = 1985
  ref_years = "1985-1994",
  cmip_vars = "pr,psl,rlut,rsut,rsds,rsdscs,rsdt,rsus,rsuscs,rlus,rtmt,sfcWind,tas,tauu,tauv,ts,ta,ua,va,zg"
  target_grid = "2.5x2.5"
  target_grid_string = "2p5x2p5"
  clim_vars = "pr,psl,rsus,rsuscs,rlus,rlut,rsds,rsdscs,rsdt,rsut,sfcWind,tas,tauu,tauv,ts,ta-200,ta-850,ua-200,ua-850,va-200,va-850"
  clim_regions = "global,ocean,land,Tropics,NHEX,SHEX"

  [[ variability_modes_cpl ]]
  current_set = "variability_modes_cpl" # Can be inferred if needed
  obs_sets = "alternate1"
  ref_final_yr = 1994
  ref_start_yr = 1985
  ref_years = "1985-1994",
  frequency = 'mo'
  landmask = True
  ModUnitsAdjust = '(True,"subtract",273.15)'
  ObsUnitsAdjust = '(True,"subtract",273.15)'
  seasons = "yearly,monthly"
  movc_vars = "ts"
  movc_modes = "PDO,NPGO,AMO"

  [[ variability_modes_atm ]]
  current_set = "variability_modes_atm" # Can be inferred if needed
  obs_sets = "alternate2"
  ref_final_yr = 1994
  ref_start_yr = 1985
  ref_years = "1985-1994",
  frequency = 'mo'
  landmask = False
  ModUnitsAdjust = '(True,"divide",100.0)'
  ObsUnitsAdjust = '(True,"divide",100.0)'
  seasons = "DJF,MAM,JJA,SON,yearly,monthly"
  mova_vars = "psl"
  mova_modes = "NAM,NAO,PNA,NPO,SAM,PSA1,PSA2"

  [[ synthetic_plots ]]
  current_set = "synthetic_plots" # Can be inferred if needed
  clim_vars = "pr,psl,ua-200"
  clim_regions = "global,ocean,land"
  mova_modes = "NAM,NAO,PNA,NPO,SAM"
  movc_modes = "PDO,NPGO"
  clim_viewer = True
  clim_years = "1985-1994"
  mova_viewer = True
  mova_years = "1985-1994"
  movc_viewer = True
  movc_years = "1985-1994"
  enso_viewer = False
  enso_years = ""
  cmip_clim_dir = /compyfs/diagnostics/pcmdi_data/metrics_data/mean_climate
  cmip_movs_dir = /compyfs/diagnostics/pcmdi_data/metrics_data/variability_modes

What machine were you running on?

Compy

Environment

dev environment for zppy and E3SM Unified 1.13.0rc4 for zppy-interfaces.

What command did you run?

zppy -c tests/integration/generated/test_weekly_comprehensive_v3_compy.cfg

Copy your cfg file

Complete cfg

[default]
case = "v3.LR.historical_0051"
constraint = ""
dry_run = "False"
environment_commands = "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh"
fail_on_dependency_skip = True
infer_path_parameters = False
infer_section_parameters = False
input = /compyfs/fors729/zppy_test_data/E3SMv3/v3.LR.historical_0051
input_subdir = archive/atm/hist
mapping_file = "map_ne30pg2_to_cmip6_180x360_aave.20200201.nc"
output = "/compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051"
partition = "short"
qos = "regular"
www = "/compyfs/www/fors729/zppy_weekly_comprehensive_v3_www/zppy_main_branch_plus_unified_1_13rc4"
years = "1985:1989:2",

[climo]
active = True
frequency = "monthly"
walltime = "00:30:00"

  [[ atm_monthly_180x360_aave ]]
  input_files = "eam.h0"
  input_subdir = "archive/atm/hist"
  vars = ""

  [[ atm_monthly_diurnal_8xdaily_180x360_aave ]]
  active = True
  frequency = "diurnal_8xdaily"
  input_files = "eam.h3"
  input_subdir = "archive/atm/hist"
  vars = "PRECT"

  [[ land_monthly_climo ]]
  active = True
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  mapping_file = "map_r05_to_cmip6_180x360_aave.20231110.nc"
  vars = ""

  [[ land_monthly_180x360_traave ]]
   active = True
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  mapping_file = "/lcrc/group/e3sm/ac.forsyth2/maps/map_r05_to_cmip6_180x360_traave.20240901.nc"
  vars = ""
  years = "1985:1994:10",

  [[ land_monthly_climo_native ]]
  active = True
  climo_jobs = 12
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  mapping_file = ""
  vars = ""
  years = "1985:1994:10"

  [[ land_monthly_climo_racmo_gis ]]
  active = True
  climo_jobs = 12
  climo_subsection = "racmo_gis"
  grid = "racmo_gis"
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  mapping_file = /lcrc/group/e3sm/ac.forsyth2/maps/map_r05_to_racmo_gis_566x438_traave.20240801.nc
  vars = ""
  years = "1985:1994:10"

  [[ land_monthly_climo_racmo_ais ]]
  active = True
  climo_jobs = 12
  input_files = "elm.h0"
  climo_subsection = "racmo_ais"
  grid = "racmo_ais"
  input_subdir = "archive/lnd/hist"
  mapping_file = /lcrc/group/e3sm/public_html/diagnostics/maps/map_r05_to_racmo_ais_591x726.nc
  vars = ""
  years = "1985:1994:10"

  [[ land_monthly_climo_merra2 ]]
  active = True
  climo_jobs = 12
  input_files = "elm.h0"
  climo_subsection = "merra2"
  grid = "merra2"
  input_subdir = "archive/lnd/hist"
  mapping_file = /lcrc/group/e3sm/public_html/diagnostics/maps/map_r05_to_merra2_traave.20250124.nc
  vars = ""
  years = "1985:1994:10"

  [[ land_monthly_climo_era5 ]]
  active = True
  climo_jobs = 12
  input_files = "elm.h0"
  climo_subsection = "era5"
  grid = "era5"
  input_subdir = "archive/lnd/hist"
  mapping_file = /lcrc/group/e3sm/public_html/diagnostics/maps/map_r05_to_era5_s2n_721x1440.traave.20240124.nc
  vars = ""
  years = "1985:1994:10"

[ts]
active = True
walltime = "00:30:00"

  [[ atm_monthly_180x360_aave ]]
  frequency = "monthly"
  input_files = "eam.h0"
  input_subdir = "archive/atm/hist"
  years = "1985:1995:2", # Need 10 years for pcmdi_diags task
  # We need to add extra vars to the default `vars=""` value
  # in order to enable the e3sm_diags mp_partition set.
  vars = "FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,CLDTOT,CLDHGH,CLDMED,CLDLOW,U,PSL,LANDFRAC,CLD_CAL_TMPICE,CLD_CAL_TMPLIQ,CLDLIQ,T"

  [[ atm_daily_180x360_aave ]]
  active = True
  frequency = "daily"
  input_files = "eam.h1"
  input_subdir = "archive/atm/hist"
  vars = "PRECT"

  [[ rof_monthly ]]
  active = True
  extra_vars = 'areatotal2'
  frequency = "monthly"
  input_files = "mosart.h0"
  input_subdir = "archive/rof/hist"
  mapping_file = ""
  vars = "RIVER_DISCHARGE_OVER_LAND_LIQ"

  [[ atm_monthly_glb ]]
  # Note global average won't work for 3D variables.
  active = True
  frequency = "monthly"
  input_files = "eam.h0"
  input_subdir = "archive/atm/hist"
  mapping_file = "glb"
  years = "1985:1995:5",

  [[ lnd_monthly_glb ]]
  active = True
  frequency = "monthly"
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  mapping_file = "glb"
  vars = ""
  years = "1985:1995:5",

  [[ land_monthly ]]
  extra_vars = "landfrac"
  frequency = "monthly"
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  mapping_file = "map_r05_to_cmip6_180x360_aave.20231110.nc"
  vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR"

  [[ land_monthly_energy ]]
  active = True
  frequency = "monthly"
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  mapping_file = ""
  vars = "EFLX_LH_TOT,FIRA,FLDS,FSA,FSDS,FSRND,FSRVD,FSDSND,FSDSVD,FSH,TSA"
  years = "1985:1994:10"

  [[ land_monthly_smb ]]
  # The data necessary for this is not found in the data input path.
  active = False
  frequency = "monthly"
  input_files = "elm.h0"
  input_subdir = "archive/lnd/hist"
  mapping_file = ""
  vars = "QICE,QICE_MELT,QRUNOFF,QSNOFRZ,QSNOMELT,QSOIL,RAIN,SNOW"
  years = "1985:1994:10"

[e3sm_to_cmip]
active = True
frequency = "monthly"
ts_num_years=2
walltime = "00:30:00"

  [[ atm_monthly_180x360_aave ]]
  cmip_plevdata = "/compyfs/diagnostics/e3sm_to_cmip_data/maps/vrt_remap_plev19.nc"
  cmip_vars = "ua, va, ta, wa, zg, hur, tas, ts, psl, ps, sfcWind, huss, pr, prc, prsn, evspsbl, tauu, tauv, hfls, clt, rlus, rsds, rsus, hfss, clivi, clwvi, rlut, rsdt, rsuscs, rsut, rtmt, abs550aer, od550aer, rsdscs, tasmax, tasmin"
  input_files = "eam.h0"
  ts_subsection = "atm_monthly_180x360_aave"
  vars = "ICEFRAC,LANDFRAC,OCNFRAC,PSL,FSNTC,FSNTOAC,SWCF,LWCF,FLUT,FSNT,FSNTOA,FLNT,FLNTC,FSNS,FLNS,FSNS,SHFLX,QFLX,LHFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,U10,QREFHT,TMQ,CLDTOT,CLDHGH,CLDMED,CLDLOW,FLDS,FSDS,TGCLDIWP,TGCLDCWP,TGCLDLWP,FLNSC,FLUTC,FSDSC,SOLIN,FSNSC,AODABS,AODVIS,AODDUST,AREL,TREFMNAV,TREFMXAV,PS,PHIS,U,V,T,Z3"
  years = "1985:1995:2", # Need 10 years for pcmdi_diags task

  [[ land_monthly ]]
  input_files = "elm.h0"
  ts_subsection = "land_monthly"

# TODO: Add "tc_analysis" back in after empty dat is resolved.
# [tc_analysis]
# active = True
# walltime = "00:30:00"

[e3sm_diags]
active = True
climo_diurnal_frequency = "diurnal_8xdaily"
climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
climo_subsection = "atm_monthly_180x360_aave"
environment_commands = "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh"
grid = '180x360_aave'
multiprocessing = True
num_workers = 8
partition = "slurm"
qos = "regular"
ref_final_yr = 1986
ref_start_yr = 1985
ref_years = "1985-1986",
# Include all sets
# min_case_e3sm_diags_depend_on_climo: "lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","annual_cycle_zonal_mean","zonal_mean_2d_stratosphere","aerosol_aeronet","aerosol_budget",
# min_case_e3sm_diags_depend_on_ts: "enso_diags","qbo",
# min_case_e3sm_diags_diurnal_cycle: "diurnal_cycle",
# min_case_e3sm_diags_streamflow: "streamflow",
# min_case_e3sm_diags_tc_analysis: "tc_analysis",
# min_case_e3sm_diags_tropical_subseasonal: "tropical_subseasonal",
# TODO: Add "tc_analysis" back in after empty dat is resolved.
# TODO: Add "aerosol_budget" back in once that's working for v3.
sets = "lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","annual_cycle_zonal_mean","zonal_mean_2d_stratosphere","enso_diags","qbo","diurnal_cycle","streamflow","tropical_subseasonal","aerosol_aeronet",
short_name = "v3.LR.historical_0051"
ts_daily_subsection = "atm_daily_180x360_aave"
ts_num_years = 2
ts_subsection = "atm_monthly_180x360_aave"
walltime = "03:00:00"
years = "1987:1989:2"
# Reference paths
# Used for mvo and mvm, if ts_num_years is set
obs_ts = "/compyfs/diagnostics/observations/Atm/time-series/"
# mvo & mvm tc_analysis only
tc_obs = "/compyfs/diagnostics/observations/Atm/tc-analysis/"

  [[ atm_monthly_180x360_aave ]]
  # Reference paths
  reference_data_path = "/compyfs/diagnostics/observations/Atm/climatology/"
  # mvo diurnal_cycle only
  dc_obs_climo = '/compyfs/diagnostics/observations/Atm/climatology/'
  # mvo streamflow only
  streamflow_obs_ts = "/compyfs/diagnostics/observations/Atm/time-series/"
  sets = "lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","annual_cycle_zonal_mean","zonal_mean_2d_stratosphere","enso_diags","qbo","diurnal_cycle","streamflow","mp_partition","tropical_subseasonal","precip_pdf","aerosol_aeronet",

  [[ atm_monthly_180x360_aave_mvm ]]
  # Test model-vs-model using the same files as the reference
  diff_title = "Difference"
  partition = "slurm"
  qos = "regular"
  ref_name = "v3.LR.historical_0051"
  run_type = "model_vs_model"
  short_ref_name = "same simulation"
  swap_test_ref = False
  tag = "model_vs_model"
  ts_num_years_ref = 2
  # Reference paths
  reference_data_path = "/compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/atm/180x360_aave/clim"
  # mvm streamflow only
  gauges_path = "/compyfs/diagnostics/observations/Atm/time-series/GSIM/GSIM_catchment_characteristics_all_1km2.csv"
  reference_data_path_ts_rof = "/compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/rof/native/ts/monthly"
  # mvm diurnal_cycle only
  reference_data_path_climo_diurnal = "/compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/atm/180x360_aave/clim_diurnal_8xdaily"
  # mvm "enso_diags", "qbo", "area_mean_time_series"
  reference_data_path_ts = "/compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/atm/180x360_aave/ts/monthly"
  # mvm tropical_subseasonal only
  reference_data_path_ts_daily = "/compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/atm/180x360_aave/ts/daily"

  [[ lnd_monthly_mvm_lnd ]]
  # Test model-vs-model using the same files as the reference
  climo_subsection = "land_monthly_climo"
  diff_title = "Difference"
  partition = "slurm"
  qos = "regular"
  ref_name = "v3.LR.historical_0051"
  run_type = "model_vs_model"
  sets = "lat_lon_land",
  short_ref_name = "same simulation"
  swap_test_ref = False
  tag = "model_vs_model"
  ts_num_years_ref = 2
  # Reference paths
  reference_data_path = "/compyfs/fors729/zppy_weekly_comprehensive_v3_output/zppy_main_branch_plus_unified_1_13rc4/v3.LR.historical_0051/post/lnd/180x360_aave/clim"

[mpas_analysis]
active = True
anomalyRefYear = 1985
environment_commands = "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh"
mesh = "IcoswISC30E3r5"
parallelTaskCount = 6
partition = "slurm"
qos = "regular"
shortTermArchive = True
walltime = "02:00:00"

  [[ reference ]]
  ts_years = "1985-1989",
  climo_years = "1985-1989",
  enso_years = "1985-1989",

  [[ test ]]
  ts_years = "1985-1995",
  climo_years = "1990-1995",
  enso_years = "1990-1995",

  [[ mvm ]]
  reference_data_path = [[ reference ]]
  test_data_path = [[ test ]]

[global_time_series]
active = True
climo_years = "1985-1989", "1990-1995",
environment_commands = "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh"
experiment_name = "v3.LR.historical_0051"
figstr = "v3.LR.historical_0051"
#moc_file=mocTimeSeries_1985-1995.nc
mpas_analysis_subsections = "reference", "test",
# plots_lnd = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILWATER_10CM,TSA,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR"
ts_num_years = 5
ts_years = "1985-1989", "1985-1995",
walltime = "00:30:00"
years = "1985-1995",

  # Important parameter combinations ##########################################

  [[ viewer_both ]]
  # 1. make_viewer = True, plots_original set, >= 1 plots_<component> set
  # NOTE: This is the case displayed in examples/post.v3.LR.historical_zppy_v3.cfg
  make_viewer = True
  moc_file = "mocTimeSeries_1985-1995.nc"
  plots_atm = "TREFHT"
  plots_lnd = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILWATER_10CM,TSA,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR"
  # Default plots_original = "net_toa_flux_restom,global_surface_air_temperature,toa_radiation,net_atm_energy_imbalance,change_ohc,max_moc,change_sea_level,net_atm_water_imbalance",

  [[ viewer_original ]]
  # 2. make_viewer = True, plots_original set, 0 plots_<component> set
  make_viewer = True
  moc_file = "mocTimeSeries_1985-1995.nc"
  # Default plots_original = "net_toa_flux_restom,global_surface_air_temperature,toa_radiation,net_atm_energy_imbalance,change_ohc,max_moc,change_sea_level,net_atm_water_imbalance",

  [[ viewer_component ]]
  # 3. make_viewer = True, plots_original set, >= 1 plots_<component> set
  make_viewer = True
  plots_atm = "TREFHT"
  plots_lnd = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILWATER_10CM,TSA,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR"
  plots_original = ""

  # We can ignore this case for image checking.
  # [[ viewer_none ]]
  # # 4. make_viewer = True, plots_original not set, 0 plots_<component> set
  # make_viewer = True
  # plots_original = ""

  [[ classic_pdf_both ]]
  # 5. make_viewer = False, plots_original set, >= 1 plots_<component> set
  # Default make_viewer = False
  moc_file = "mocTimeSeries_1985-1995.nc"
  plots_atm = "TREFHT"
  plots_lnd = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILWATER_10CM,TSA,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR"
  # Default plots_original = "net_toa_flux_restom,global_surface_air_temperature,toa_radiation,net_atm_energy_imbalance,change_ohc,max_moc,change_sea_level,net_atm_water_imbalance",

  [[ classic_pdf_original ]]
  # 6. make_viewer = False, plots_original set, 0 plots_<component> set
  # NOTE: This is the default case -- where all of the above parameters are at their default values.
  # Default make_viewer = False
  moc_file = "mocTimeSeries_1985-1995.nc"
  # Default plots_original = "net_toa_flux_restom,global_surface_air_temperature,toa_radiation,net_atm_energy_imbalance,change_ohc,max_moc,change_sea_level,net_atm_water_imbalance",

  [[ classic_pdf_component ]]
  # 7. make_viewer = False, plots_original set, >= 1 plots_<component> set
  # Default make_viewer = False
  plots_atm = "TREFHT"
  plots_lnd = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILWATER_10CM,TSA,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR"
  plots_original = ""

  # We can ignore this case for image checking.
  # [[ classic_pdf_none ]]
  # # 8. make_viewer = False, plots_original not set, 0 plots_<component> set
  # # Default make_viewer = False
  # plots_original = ""

  # Special cases #############################################################

  [[ all_lnd_var_viewer ]]
  # 1. plot ALL land variables
  make_viewer = True
  partition = "slurm"
  plots_lnd = "all"
  plots_original = ""
  walltime = "03:00:00"

  [[ classic_original_8_no_ocn ]]
  # 2. exclude ocean from original plots
  plots_original = "net_toa_flux_restom,global_surface_air_temperature,toa_radiation,net_atm_energy_imbalance,net_atm_water_imbalance"


[ilamb]
active = True
e3sm_to_cmip_atm_subsection = "atm_monthly_180x360_aave"
e3sm_to_cmip_land_subsection = "land_monthly"
ilamb_obs = "/compyfs/diagnostics/ilamb_data"
nodes = 8
partition = "short"
short_name = "v3.LR.historical_0051"
ts_atm_subsection = "atm_monthly_180x360_aave"
ts_land_subsection = "land_monthly"
ts_num_years = 2
walltime = "2:00:00"
years = "1985:1989:4"

[livvkit]
active = True
climo_subsections = "land_monthly_180x360_traave","land_monthly_climo_native","land_monthly_climo_racmo_gis","land_monthly_climo_racmo_ais","land_monthly_climo_merra2","land_monthly_climo_era5"
cfg = "inclusions/livvkit/livvkit_r05.jinja"
debug = True
environment_commands = "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh"
icesheets = "gis,ais"
nodes = 1
sets = "energy_racmo,energy_era5,energy_merra2,energy_ceres"
ts_land_subsection = "land_monthly_energy"
walltime = "0:30:00"
years = "1985:1994:10"


[pcmdi_diags]
active = True
environment_commands_secondary = "source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh"
walltime = "2:00:00"
model_name = 'e3sm.historical.v3-LR.0051'
model_tableID = 'Amon'
e3sm_to_cmip_atm_subsection = "atm_monthly_180x360_aave" # Can be inferred if needed
multiprocessing = True
num_workers = 24
obs_ts = '/compyfs/diagnostics/observations/Atm/time-series/'
pcmdi_debug = False
ts_num_years = 2
ts_years = "1985-1994",

  [[ mean_climate ]]
  current_set = "mean_climate" # Can be inferred if needed
  obs_sets = "default"
  ref_final_yr = 1994
  ref_start_yr = 1985
  ref_years = "1985-1994",
  cmip_vars = "pr,psl,rlut,rsut,rsds,rsdscs,rsdt,rsus,rsuscs,rlus,rtmt,sfcWind,tas,tauu,tauv,ts,ta,ua,va,zg"
  target_grid = "2.5x2.5"
  target_grid_string = "2p5x2p5"
  clim_vars = "pr,psl,rsus,rsuscs,rlus,rlut,rsds,rsdscs,rsdt,rsut,sfcWind,tas,tauu,tauv,ts,ta-200,ta-850,ua-200,ua-850,va-200,va-850"
  clim_regions = "global,ocean,land,Tropics,NHEX,SHEX"

  [[ variability_modes_cpl ]]
  current_set = "variability_modes_cpl" # Can be inferred if needed
  obs_sets = "alternate1"
  ref_final_yr = 1994
  ref_start_yr = 1985
  ref_years = "1985-1994",
  frequency = 'mo'
  landmask = True
  ModUnitsAdjust = '(True,"subtract",273.15)'
  ObsUnitsAdjust = '(True,"subtract",273.15)'
  seasons = "yearly,monthly"
  movc_vars = "ts"
  movc_modes = "PDO,NPGO,AMO"

  [[ variability_modes_atm ]]
  current_set = "variability_modes_atm" # Can be inferred if needed
  obs_sets = "alternate2"
  ref_final_yr = 1994
  ref_start_yr = 1985
  ref_years = "1985-1994",
  frequency = 'mo'
  landmask = False
  ModUnitsAdjust = '(True,"divide",100.0)'
  ObsUnitsAdjust = '(True,"divide",100.0)'
  seasons = "DJF,MAM,JJA,SON,yearly,monthly"
  mova_vars = "psl"
  mova_modes = "NAM,NAO,PNA,NPO,SAM,PSA1,PSA2"

  [[ synthetic_plots ]]
  current_set = "synthetic_plots" # Can be inferred if needed
  clim_vars = "pr,psl,ua-200"
  clim_regions = "global,ocean,land"
  mova_modes = "NAM,NAO,PNA,NPO,SAM"
  movc_modes = "PDO,NPGO"
  clim_viewer = True
  clim_years = "1985-1994"
  mova_viewer = True
  mova_years = "1985-1994"
  movc_viewer = True
  movc_years = "1985-1994"
  enso_viewer = False
  enso_years = ""
  cmip_clim_dir = /compyfs/diagnostics/pcmdi_data/metrics_data/mean_climate
  cmip_movs_dir = /compyfs/diagnostics/pcmdi_data/metrics_data/variability_modes

What jobs are failing?

pcmdi_diags_variability_modes_cpl_model_vs_obs_1985-1994.o750968

What stack trace are you encountering?

[forsyth2]

So the two differences from the last successful tests are:

• zppy-interfaces is no longer using a dev environment, but rather E3SM Unified 1.13.0rc4
• We're testing on a Compy rather than Chrysalis/LCRC.

[forsyth2]

On Bebop (substituting for Chrysalis):

conda activate test-zi-main-20260406 # Activate last dev environment made for zppy-interfaces
conda list | grep python
python                    3.14.3

Now, on Compy:

source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.13.0rc4_compy.sh
conda list | grep python
# bash: conda: command not found...
python
# Python 3.13.13

Could it be because the dev environments used Python 3.14 rather than Python 3.13?

[forsyth2]

After running python -c "import numpy; numpy.show_config()" on both, AI suggests:

Both show BLAS/LAPACK 3.9.0 with OpenBLAS, so the linear algebra library version is identical — that rules out the LAPACK backend as the cause.

the PDO computation for this particular 10-year window (1985–1994) is numerically marginal — it works on Bebop's hardware but fails on Compy's. The root fix really should be in pcmdi_metrics itself, adding NaN/convergence guards around the np.polyfit call.

In the meantime, is there any flexibility on the test parameters — e.g. trying a longer time window than 1985–1994 to see if the problem is specific to that short 10-year range? That would help confirm whether it's a marginal numerics issue versus something more systematic on Compy.

[forsyth2]

Asking AI if this is perhaps an "intermittently recurring issue that's just happened to never to show up before?":

That's actually quite plausible. SVD convergence failures in LAPACK can be non-deterministic in the sense that they depend on:

• Thread scheduling — OpenBLAS uses multithreading, and the order in which threads execute floating-point operations can vary run to run
• Memory alignment — subtle differences in how data lands in memory between runs

So it's possible this has always been a latent fragility in the PDO computation for this particular short 10-year window, and you've just been lucky until now. The fact that it's a 10-year window for a low-frequency mode like PDO makes it a natural candidate for being right on the edge of numerical stability.

If that's the case, the most pragmatic path forward might be to just rerun the job and see if it passes. If it passes on a second attempt, that pretty strongly suggests intermittent numerical instability rather than a systematic environment issue — and the right response would be to file an upstream issue with pcmdi_metrics about adding convergence guards, without blocking the 1.13.0rc4 release over it.

[forsyth2]

I guess we could try to increase the years included, but zppy testing already has a long runtime and I'm wary of having it take even longer.