[Bug]: MPAS variable modules use incorrect logger and saves cmor_logs in PWD #298
Description
What happened?
MPAS variables defined as mpas_vars modules have their own cmor.write function, which saves the cmor_logs under the PWD. However after merging #289, variables defined via handlers.yaml (using the VarHandler class) now save cmor_logs output under the -o/--output directory. As a result, there are two separate cmor_logs directories.
Related lines of code:
- Line where arguments are passed to cmorize:
e3sm_to_cmip/e3sm_to_cmip/__main__.py
Lines 903 to 919 in 01890f8
- All of the
mpas_varsmodules here withcmorize()function: https://github.com/E3SM-Project/e3sm_to_cmip/tree/master/e3sm_to_cmip/cmor_handlers/mpas_vars
What did you expect to happen? Are there are possible answers you came across?
The mpas_vars modules should use the same specified cmor_logs directory as all other variables.
Minimal Complete Verifiable Example (MVCE)
"""
This script is a minimal working example (MWE) to reproduce the issue reported
in issue #297 of the `e3sm_to_cmip` GitHub repository.
Source: https://github.com/E3SM-Project/e3sm_to_cmip/issues/297#
Original Command: e3sm_to_cmip -v mlotst -u /lcrc/group/e3sm2/DSM/Ops/DSM_Manager/tmp/v3.LR.historical_0051/metadata/historical_r1i1p1f1.json -t /lcrc/group/e3sm2/DSM/Staging/Resource/cmor/cmip6-cmor-tables/Tables -o YOUR_OUTPUT_DIR -i /lcrc/group/e3sm2/DSM/Ops/test/e2c_input -s --realm mpaso --map /lcrc/group/e3sm2/DSM/Staging/Resource/maps/map_IcoswISC30E3r5_to_cmip6_180x360_traave.20240221.nc
"""
import os
from e3sm_to_cmip.__main__ import main
OUTPUT_PATH = "/lcrc/group/e3sm/public_html/e3sm_to_cmip/297-v3-fails"
args = [
"--var-list",
"mlotst",
"--user-metadata",
"/lcrc/group/e3sm2/DSM/Ops/DSM_Manager/tmp/v3.LR.historical_0051/metadata/historical_r1i1p1f1.json",
"--tables-path",
"/lcrc/group/e3sm2/DSM/Staging/Resource/cmor/cmip6-cmor-tables/Tables",
"--output-path",
f"{OUTPUT_PATH}",
"--input-path",
"/lcrc/group/e3sm2/DSM/Ops/test/e2c_input",
"--serial",
"--realm",
"mpaso",
"--map",
"/lcrc/group/e3sm2/DSM/Staging/Resource/maps/map_IcoswISC30E3r5_to_cmip6_180x360_traave.20240221.nc",
]
main(args)
# Ensure the path and its contents have the correct permissions recursively
for root, dirs, files in os.walk(OUTPUT_PATH):
os.chmod(root, 0o505) # o=rx (read and execute for others)
for d in dirs:
os.chmod(os.path.join(root, d), 0o505)
for f in files:
os.chmod(os.path.join(root, f), 0o404) # o=r (read for others)Relevant log output
2025-05-22 12:49:05.607233 [INFO]: __main__.py(__init__:163) >> --------------------------------------
2025-05-22 12:49:05.608262 [INFO]: __main__.py(__init__:164) >> | E3SM to CMIP Configuration
2025-05-22 12:49:05.608943 [INFO]: __main__.py(__init__:165) >> --------------------------------------
2025-05-22 12:49:05.624792 [INFO]: __main__.py(__init__:189) >> * Timestamp: 20250522_174905_575700
2025-05-22 12:49:05.625317 [INFO]: __main__.py(__init__:189) >> * Version Info: branch bugfix/297-v3-fails with commit 01890f837b353fdb47622785a32c6e221e4b00e5
2025-05-22 12:49:05.625760 [INFO]: __main__.py(__init__:189) >> * Mode: Serial
2025-05-22 12:49:05.626092 [INFO]: __main__.py(__init__:189) >> * Variable List: ['mlotst']
2025-05-22 12:49:05.626372 [INFO]: __main__.py(__init__:189) >> * Input Path: /lcrc/group/e3sm2/DSM/Ops/test/e2c_input
2025-05-22 12:49:05.626631 [INFO]: __main__.py(__init__:189) >> * Output Path: /lcrc/group/e3sm/public_html/e3sm_to_cmip/297-v3-fails
2025-05-22 12:49:05.626942 [INFO]: __main__.py(__init__:189) >> * Precheck Path: None
2025-05-22 12:49:05.627202 [INFO]: __main__.py(__init__:189) >> * Log Path: /lcrc/group/e3sm/public_html/e3sm_to_cmip/297-v3-fails/20250522_174905_575700.log
2025-05-22 12:49:05.627468 [INFO]: __main__.py(__init__:189) >> * CMOR Log Path: /lcrc/group/e3sm/public_html/e3sm_to_cmip/297-v3-fails/cmor_logs
2025-05-22 12:49:05.627743 [INFO]: __main__.py(__init__:189) >> * Temp Path for Processing MPAS Files: /lcrc/group/e3sm/public_html/e3sm_to_cmip/297-v3-fails/tmp
2025-05-22 12:49:05.628015 [INFO]: __main__.py(__init__:189) >> * Frequency: mon
2025-05-22 12:49:05.628279 [INFO]: __main__.py(__init__:189) >> * Realm: mpaso
2025-05-22 12:49:06.016511 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:06.193088 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:06.320772 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:06.520533 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:06.832609 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:06.969639 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:07.351278 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:07.485674 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:07.619889 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:07.786347 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:07.905681 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:08.101718 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:08.320813 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:08.509535 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:08.691679 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:08.889517 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:09.021515 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:09.192565 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:09.330324 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:09.538591 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:09.671763 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:09.936960 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:10.254115 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:10.406400 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:10.636718 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:10.788577 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
2025-05-22 12:49:10.920747 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
...
2025-05-22 12:49:30.827848 [WARNING]: warnings.py(_showwarnmsg:110) >> /gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedeltaCoder' instance.
ds = xr.open_dataset(path)
0%| | 0/1 [00:00<?, ?it/s]INFO:root:Starting mlotst
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...
Masking mlotst
INFO:root:mpas.py: remap: ncremap args = ['ncremap', '-P', 'mpasocean', '-7', '--dfl_lvl=1', '--no_stdin', '--no_cll_msr', '--no_frm_trm', '--no_permute', '--rnr_thr=0.05', '--map=/lcrc/group/e3sm2/DSM/Staging/Resource/maps/map_IcoswISC30E3r5_to_cmip6_180x360_traave.20240221.nc', '/lcrc/group/e3sm/public_html/e3sm_to_cmip/297-v3-fails/tmp/tmpyk0ia5z2', '/lcrc/group/e3sm/public_html/e3sm_to_cmip/297-v3-fails/tmp/tmp_lluawf5']
INFO:root:b'Input #00: /lcrc/group/e3sm/public_html/e3sm_to_cmip/297-v3-fails/tmp/tmpyk0ia5z2\natt(in) : /lcrc/group/e3sm/public_html/e3sm_to_cmip/297-v3-fails/tmp/tmpyk0ia5z2\natt(out) : /tmp/ncremap_tmp_att.nc.pid886652.flidx0\nMap/Wgt : /lcrc/group/e3sm2/DSM/Staging/Resource/maps/map_IcoswISC30E3r5_to_cmip6_180x360_traave.20240221.nc\n'
ERROR:root:Error in cmor.write for mlotst
Traceback (most recent call last):
File "/gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/__main__.py", line 907, in _run_serial
name = handler_method(
vars_to_filepaths,
...<3 lines>...
self.cmor_log_dir,
)
TypeError: handle() takes 3 positional arguments but 5 were given
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/gpfs/fs1/home/ac.tvo/E3SM-Project/e3sm_to_cmip/e3sm_to_cmip/mpas.py", line 427, in write_cmor
cmor.write(
~~~~~~~~~~^
varid,
^^^^^^
...<2 lines>...
time_bnds=ds.time_bnds.values,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/ac.tvo/miniforge3/envs/e3sm_diags_dev_297/lib/python3.13/site-packages/cmor/pywrapper.py", line 834, in write
return _cmor.write(var_id, data, data_type, file_suffix, ntimes_passed,
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
time_vals, time_bnds, store_with)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_cmor.CMORError: Problem with 'cmor.write'. Please check the logfile (if defined).
INFO:e3sm_to_cmip.__main__:Finished , 1/1 jobs complete
100%|██████████| 1/1 [00:59<00:00, 59.58s/it]
INFO:e3sm_to_cmip.__main__:1 of 1 handlers completeAnything else we need to know?
No response
Environment
Latest e3sm_to_cmip on main