[Bug]: Intermittent viewer generation error occurring at the end of a run #1000
Description
What happened?
I'm running a few e3sm_diags runs for v3.LR and v3.HR on Perlmutter. In one of the run, I saw viewer generation error at the end, the main viewer was not created successfully.
Error message:
2025-08-08 13:55:27,666 [INFO]: main.py(create_viewer:145) >> [('Latitude-Longitude contour maps', 'lat_lon/index.html'), ('Table', 'table/index.html'), ('Taylor Diagram', 'taylor/index.html'), ('CMIP6 Comparison', 'cmip6/index.html')]
2025-08-08 13:55:27,695 [INFO]: main.py(create_viewer:142) >> zonal_mean_xy model_vs_obs_1995-2004/viewer
2025-08-08 13:57:12,454 [INFO]: main.py(create_viewer:145) >> ('Zonal mean line plots', 'zonal_mean_xy/index.html')
2025-08-08 13:57:12,474 [INFO]: main.py(create_viewer:142) >> zonal_mean_2d model_vs_obs_1995-2004/viewer
2025-08-08 13:58:11,437 [INFO]: main.py(create_viewer:145) >> ('Pressure-Latitude zonal mean contour plots', 'zonal_mean_2d/index.html')
2025-08-08 13:58:11,438 [INFO]: main.py(create_viewer:142) >> polar model_vs_obs_1995-2004/viewer
2025-08-08 14:00:22,438 [INFO]: main.py(create_viewer:145) >> ('Polar contour maps', 'polar/index.html')
2025-08-08 14:00:22,445 [INFO]: main.py(create_viewer:142) >> cosp_histogram model_vs_obs_1995-2004/viewer
2025-08-08 14:01:37,505 [INFO]: main.py(create_viewer:145) >> ('CloudTopHeight-Tau joint histograms', 'cosp_histogram/index.html')
2025-08-08 14:01:37,506 [INFO]: main.py(create_viewer:142) >> meridional_mean_2d model_vs_obs_1995-2004/viewer
2025-08-08 14:02:23,897 [INFO]: main.py(create_viewer:145) >> ('Pressure-Longitude meridional mean contour plots', 'meridional_mean_2d/index.html')
2025-08-08 14:02:23,898 [INFO]: main.py(create_viewer:142) >> annual_cycle_zonal_mean model_vs_obs_1995-2004/viewer
2025-08-08 14:05:54,522 [INFO]: main.py(create_viewer:145) >> ('Annual Cycle Zonal Mean Contour Plots', 'annual_cycle_zonal_mean/index.html')
2025-08-08 14:05:54,538 [INFO]: main.py(create_viewer:142) >> qbo model_vs_obs_1995-2004/viewer
2025-08-08 14:07:14,230 [INFO]: main.py(create_viewer:145) >> ('Quasi-biennial Oscillation', 'qbo/index.html')
2025-08-08 14:07:14,249 [INFO]: main.py(create_viewer:142) >> diurnal_cycle model_vs_obs_1995-2004/viewer
2025-08-08 14:08:34,869 [INFO]: main.py(create_viewer:145) >> ('Diurnal cycle phase maps', 'diurnal_cycle/index.html')
2025-08-08 14:08:34,870 [INFO]: main.py(create_viewer:142) >> zonal_mean_2d_stratosphere model_vs_obs_1995-2004/viewer
2025-08-08 14:09:43,437 [INFO]: main.py(create_viewer:145) >> ('Pressure-Latitude zonal mean contour plots (Stratosphere)', 'zonal_mean_2d_stratosphere/index.html')
2025-08-08 14:09:43,451 [INFO]: main.py(create_viewer:142) >> aerosol_aeronet model_vs_obs_1995-2004/viewer
2025-08-08 14:12:02,167 [ERROR]: run.py(run_diags:108) >> Error traceback:
Traceback (most recent call last):
File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.11.1_pm-cpu/lib/python3.10/site-packages/e3sm_diags/run.py", line 106, in run_diags
params_results = main(params)
File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.11.1_pm-cpu/lib/python3.10/site-packages/e3sm_diags/e3sm_diags_driver.py", line 445, in main
index_path = create_viewer(parameters_results)
File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.11.1_pm-cpu/lib/python3.10/site-packages/e3sm_diags/viewer/main.py", line 144, in create_viewer
result = viewer_function(viewer_dir, parameters)
File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.11.1_pm-cpu/lib/python3.10/site-packages/e3sm_diags/viewer/default_viewer.py", line 189, in create_viewer
name, url = SET_TO_NAME[set_name], viewer.generate_page()
File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.11.1_pm-cpu/lib/python3.10/site-packages/e3sm_diags/viewer/core_viewer.py", line 90, in generate_page
rechmod(self.path, default_mask)
File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.11.1_pm-cpu/lib/python3.10/site-packages/output_viewer/utils.py", line 30, in rechmod
os.chmod(d, perms)
OSError: [Errno 5] Input/output error: '/global/cfs/cdirs/e3sm/chengzhu/tests/zppy_example_v3/v3.LR.amip_0101/post/scripts/tmp.e3sm_diags_atm_monthly_180x360_aave_model_vs_obs_1995-2004.41500861.FfvU/model_vs_obs_1995-2004/viewer/lat_lon/aero-no-ref-data/_mass_bc_850-global'
What did you expect to happen? Are there are possible answers you came across?
No response
Minimal Complete Verifiable Example (MVCE)
Relevant log output
Anything else we need to know?
results directory: https://portal.nersc.gov/cfs/e3sm/chengzhu/tests/zppy_example_v3/v3.LR.amip_0101/v3.LR.amip_0101/e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_1995-2004/
Environment
e3sm_unified_1.11.1_pm-cpu (e3sm_diags3.0.0)