Unable to identify time when input files with the same time but different calendars

[KianHuangKimeow]

Issue: Input files with the same time but different calendars cannot be identified by TempestExtremes (TE).

Description: Under some circumstances (e.g., WRF), users may use xarray and other Python tools to interpolate model states from model levels to standard pressure levels, then feed TE applications with processed (standard pressure levels) and original WRF (surface) data. Because xarray will read time as numpy.datetime64, the time dimension will be saved with an attribute of “proleptic_gregorian” calendar by default. In this case, time in WRF outputs (standard calendar) and processed data (proleptic_gregorian) have different calendar attributes. When TE applications read files, time in the following files is compared with the one in the first file. In src/base/TimeObj.cpp, operator Time::operator== identifies two time objects by comparing their calendar types and then other member variables, which will result in a time identification error even if the timestamps in different files are the same.

Suggestion: It might be better to add a warning or exception when calendars in files are different or standardize time (i.e., converting to the same calendar) when reading files.

Temporary Solution: The current solution for this issue is to keep the same calendar when processing model data. If choosing xarray, one might use the following command to convert the calendar.

ds = ds.convert_calendar('standard', dim='Time', use_cftime=True)

[paullric]

That seems like an xarray issue, honestly. Maybe we should change the TimeObj::operator== to recognize proleptic_gregorian and standard as the same?

[KianHuangKimeow]

This issue was not easy to find out in the first place, so I left a description for others who might have the same problem.
It looks like the differences between proleptic_gregorian and standard calendars are mainly when the datetime is before 1582-10-15 (see https://cfconventions.org/cf-conventions/cf-conventions#calendar). They should be the same for modern studies. If we want to change the TimeObj::operator== to recognize them as the same, it might be better to assign CalendarType to the same one when reading from inputs (i.e., in function Time::CalendarTypeFromString).

[paullric]

It’s better to keep different CalendarTypes so that when the data is re-written there is a consistency in the calendar. The fact that xarray doesn’t preserve that consistency is arguably a bug.

…

On Feb 18, 2025, at 9:52 AM, Kian Huang ***@***.***> wrote: KianHuangKimeow left a comment (ClimateGlobalChange/tempestextremes#95) This issue was not easy to find out in the first place, so I left a description for others who might have the same problem. It looks like the differences between proleptic_gregorian and standard calendars are mainly when the datetime is before 1582-10-15 (see https://cfconventions.org/cf-conventions/cf-conventions#calendar). They should be the same for modern studies. If we want to change the TimeObj::operator== to recognize them as the same, it might be better to assign CalendarType to the same one when reading from inputs (i.e., in function Time::CalendarTypeFromString). — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented. <https://cfconventions.org/cf-conventions/cf-conventions#calendar> <#95 (comment)> <https://github.com/notifications/unsubscribe-auth/ABIVPZFDUFCMHJJPVXCLHSD2QNXPLAVCNFSM6AAAAABXKU3D2GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRWGQ3DEOJTHE> KianHuangKimeow left a comment (ClimateGlobalChange/tempestextremes#95) <#95 (comment)> This issue was not easy to find out in the first place, so I left a description for others who might have the same problem. It looks like the differences between proleptic_gregorian and standard calendars are mainly when the datetime is before 1582-10-15 (see https://cfconventions.org/cf-conventions/cf-conventions#calendar). They should be the same for modern studies. If we want to change the TimeObj::operator== to recognize them as the same, it might be better to assign CalendarType to the same one when reading from inputs (i.e., in function Time::CalendarTypeFromString). — Reply to this email directly, view it on GitHub <#95 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABIVPZFDUFCMHJJPVXCLHSD2QNXPLAVCNFSM6AAAAABXKU3D2GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRWGQ3DEOJTHE>. You are receiving this because you commented.