v0.4.0 Testing Feedback

[lee1043]

Discussion related to v0.4.0

[lee1043]

Was decode_non_cf_time deprecated? I got AttributeError: module 'xcdat' has no attribute 'decode_non_cf_time' message while running my code after updating to v0.4.0.

[pochedls]

It was deprecated here because we handle decoding of CF and non-CF datasets with one function now.

[lee1043]

@pochedls good to know, thank you for clarifying.

[tomvothecoder]

Renaming decode_non_cf_time() to decode_time() is technically a breaking change that warranted a major release.
However, I assumed direct usage of this function was low and we haven't released a v1.0.0 yet.

Based on this discussion, we should consider expediting the v1.0.0 release soon (#375) if there are any more breaking changes and/or when #388 is merged.

[lee1043]

Following code was working okay in v0.3.3 but reporting the following error in v0.4.0, which I am curious if the following behavior is expected / desired or if it is a bug.

mod_file = "cmip6.historical.E3SMv2.r1i1p1.mon.pr.198101-200512.AC.v20221027_test2-regridded.nc"  # xCDAT calculated AC and regridded
dm_x = xc.open_dataset(mod_file, data_var="pr")
dm_x.temporal.average(var)

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File /opt/anaconda3/envs/pcmdi_metrics_dev_5/lib/python3.9/site-packages/xcdat/temporal.py:733, in TemporalAccessor._set_data_var_attrs(self, data_var)
    732 try:
--> 733     self.calendar = dv[self.dim].encoding["calendar"]
    734     self.date_type = get_date_type(self.calendar)

KeyError: 'calendar'

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
Cell In [26], line 1
----> 1 dm_x.temporal.average(var)

File /opt/anaconda3/envs/pcmdi_metrics_dev_5/lib/python3.9/site-packages/xcdat/temporal.py:194, in TemporalAccessor.average(self, data_var, weighted, keep_weights)
    149 def average(self, data_var: str, weighted: bool = True, keep_weights: bool = False):
    150     """
    151     Returns a Dataset with the average of a data variable and the time
    152     dimension removed.
   (...)
    192     >>> ds_month.ts
    193     """
--> 194     self._set_data_var_attrs(data_var)
    196     freq = self._infer_freq()
    198     return self._averager(data_var, "average", freq, weighted, keep_weights)

File /opt/anaconda3/envs/pcmdi_metrics_dev_5/lib/python3.9/site-packages/xcdat/temporal.py:736, in TemporalAccessor._set_data_var_attrs(self, data_var)
    734     self.date_type = get_date_type(self.calendar)
    735 except KeyError:
--> 736     raise KeyError(
    737         f"The 'calendar' encoding attribute is not set on the '{data_var}' "
    738         f"time coordinate variable ({self.dim}). This might indicate that the "
    739         "time coordinates were not decoded, which is required for temporal "
    740         "averaging operations. "
    741     )

KeyError: "The 'calendar' encoding attribute is not set on the 'pr' time coordinate variable (time). This might indicate that the time coordinates were not decoded, which is required for temporal averaging operations. "

[pochedls]

It seems like in v0.4 when setting the time attributes an explicit error is thrown if the calendar is missing.

I would have thought that xcdat would put standard in for calendar on open, but maybe it simply guesses the calendar when decoding the time. I think either way, we don't want it to fail here if the time axis is already decoded (it doesn't need the calendar to do temporal averaging).

[tomvothecoder]

decode_time() defaults to calendar="standard" if the calendar attr is missing.

xcdat/xcdat/dataset.py

Lines 318 to 324 in d5d4fd7

	if _is_decodable(coords) and not _is_decoded(coords):
	if coords.attrs.get("calendar") is None:
	coords.attrs["calendar"] = "standard"
	logger.warning(
	f"'{coords.name}' does not have a calendar attribute set. "
	"Defaulting to CF 'standard' calendar."
	)

In your Dataset, I'm curious as to why the calendar attribute is missing after opening and decoding.

Also, I should replace this check to see if time coordinates are decoded:

xcdat/xcdat/temporal.py

Lines 732 to 741 in d5d4fd7

	try:
	self.calendar = dv[self.dim].encoding["calendar"]
	self.date_type = get_date_type(self.calendar)
	except KeyError:
	raise KeyError(
	f"The 'calendar' encoding attribute is not set on the '{data_var}' "
	f"time coordinate variable ({self.dim}). This might indicate that the "
	"time coordinates were not decoded, which is required for temporal "
	"averaging operations. "
	)

with this function:

xcdat/xcdat/dataset.py

Lines 491 to 509 in d5d4fd7

	def _is_decoded(da: xr.DataArray) -> bool:
	"""Check if a time-based DataArray is decoded.
	This is determined by checking if the `encoding` dictionary has "units" and
	"calendar" attributes set.
	Parameters
	----------
	da : xr.DataArray
	A time-based DataArray (e.g,. coordinates, bounds)
	Returns
	-------
	bool
	"""
	units = da.encoding.get("units")
	calendar = da.encoding.get("calendar")
	return calendar is not None and units is not None

Can you provide this file for us to debug?

[lee1043]

@tomvothecoder @pochedls Thank you for your comment. It looks like I have made a mistake when handling the data. In the original code I was opening two files: one that cdms written another written using xcdat. And I was manually copying the time dimension from cdms written dataset to xcdat written dataset, then applied temporal average to the xcdat written dataset. My intention was to correct time mismatching in annual cycle. The issue was resolved when I copied time dimension in the opposite way; copy time from xcdat-written dataset to the cdms-written dataset. My apology for the confusion.

[tomvothecoder]

@lee1043 No worries, that clears things up

[lee1043]

Just confirming that v0.4.0 update resulted no change in the PMP mean climate statistics (bias, rmse, correlation, standard deviation, etc), which is GREAT because we don't want these numbers changing with the newer version!

[tomvothecoder]

Awesome, I'm glad to hear things are identical between versions! Thanks for testing Jiwoo!

[tomvothecoder]

Also thanks for opening up this new thread since #296 is outdated.

[arfriedman]

Hi all,

I am a recent xcdat user. I've encountered a specific problem temporal averaging a dataset with a fixed-frequency cftime axis which seems related to the calendar issue discussed here.

I am examining monthly data from ERSST:

import cftime
import xarray as xr
import xcdat as xc
ERSST = xc.open_dataset("http://psl.noaa.gov/thredds/dodsC/Datasets/noaa.ersst.v5/sst.mnmean.nc")

I change the time to a fixed-frequency cftime range (to allow for comparison with other datasets in my project):
ERSST["time"] = xr.cftime_range(start=cftime.datetime(1854, 1, 1), end=cftime.datetime(2022, 12, 31), freq="M")

Sseasonal averaging works fine with the original time axis (not shown), but attempting to take a seasonal average of the modified fixed-cftime-frequency dataset results in an error:

ERSST_seas = ERSST.temporal.group_average("sst", "season")
KeyError: "The 'calendar' encoding attribute is not set on the 'sst' time coordinate variable (time). This might indicate that the time coordinates were not decoded, which is required for temporal averaging operations. "

I just wanted to check if the error with the modified axis is a bug, or if I am pushing the software to the limit.

Thanks much,
Andrew

[tomvothecoder]

I'll include this information in #414.

[arfriedman]

Thank you @tomvothecoder and @pochedls

[pochedls]

@arfriedman – I realize that the workaround that I gave you is problematic because of this issue. Basically – since the original time bounds were dropped (and then generated with .add_missing_bounds()) they will not correspond to the calendar correctly. I think the effect of this issue is that your seasonal averages will not receive correct weights (so instead of February receiving 28 days of "weight" it probably is receiving 30 days).

[arfriedman]

Thank you @pochedls for letting me know. I see that #412 would definitely be helpful.

[tomvothecoder]

#414 improves the usage of custom time coordinates with the temporal accessor APIs.

Instead of raising an error, the temporal accessor APIs will raise a warning if a "calendar" attribute is not set and defaults it to "standard". The dataset still needs time bounds before the APIs are called.

[jypeter]

That's a good example where some equivalent of setTimeBoundsMonthly would be useful :-)