Annual cycling on decimal day

[lhuggett]

When setting the initial cycle point as a decimal day of year, the subsequent cycle points will be on the same date, regardless of whether or not it is actually the same decimal day.
E.g. using day 101, if the ICP is in the year 2000 (a leap year), the date will be 2000-04-10, which is correct, but then in 2001, the date will be 2001-04-10, even though the 101st day of 2001 was 2001-04-11. Similarly if the ICP year is not a leap year, the subsequent leap years will not be taken into account.

[wxtim]

Attempt to replicate issue

I tried initial cycle point = 2000-060 (29 Feb) and got fascinating output...

Worse - because 29 and 28 are clearly not exactly a year apart

graph = "model[-P1Y] => model"

does not create a requirement for the task on the 2000-02-29 to complete prior to 2001-02-28 to start.

adding a print statement to cylc.lib.isodatetime.parsers (~line 310 at time of writing) in method TimePointParser._create_timepoint_from_info

        info.update({"truncated_dump_format": truncated_dump_format})
        print(info)
        return data.TimePoint(**info)

produced interesting results:
screendump.txt

This looks to me like the three digit day number is parsed as a date prior to being used as a base time for cycling, and therefore Cylc has no idea that you wanted it to start on the same day of each year. To do this I think we'd need to add some sort of Boolean variable to store the input format of the initial cycle time in the and make this part of the Timepoint object This would then hopefully allow us to change how the cycling works.

Is this a bug which needs fixing or a feature which requires documentation?

[matthewrmshin]

🐛

[hjoliver]

(These cycling edge cases can be a nightmare!)

[wxtim]

Note

There's already a method for determining whether the date is one we want to apply the fix for this to:

if TimePoint.get_is_ordinal_date() == True:

The conversion of the YYYY-DD format to YYYY-MM-DD seems to be handled by cycling.iso8601.iso8601point.standardise(). I tried commenting most of the logic out of this and the suite still works, and still displays the same behaviour, so it looks like the problem will be in the annual cycling logic.

[wxtim]

Proposed Solution

There is a solution: Add cycle point format = %Y-%j to the [cylc] section of the suite.rc. The only issue with this is that you don't see the dates calculated in the suite - each cycle is referred to as YYYY-DDD.

We could make cylc/isodatetime check for initial cycle point being given in this format and set the cycle point format. I do not think that there is a clear case for either behaviour being set to default.

I think we should change the documentation and/or get cylc to return a warning if dates are given in this format.

Responses @lhuggett @oliver-sanders @hjoliver @matthewrmshin @sadielbartholomew @kinow

[wxtim]

Test Suite Code

[cylc]
    cycle point format = %Y-%j
[scheduling] 
    initial cycle point = 2000-060
    final cycle point = 2005-060
    [[dependencies]] 
        [[[P1Y]]] 
            graph = "model[-P1Y] => model" 
[runtime]
    [[root]]
        script = sleep 5

[hjoliver]

So is the problem here essentially that the ordinal day initial point gets converted to the default cycle point format, which is not really compatible with ordinal counting?

I wonder if we should (in general, not just for this case) infer the cycle point format from the initial cycle point format?

[matthewrmshin]

I wonder if we should (in general, not just for this case) infer the cycle point format from the initial cycle point format?

That's also our first reaction. However, we currently allow this to be abbreviated. E.g. initial cycle point = 2000 above would imply 2000-001, so I am not sure how best to do this.

Either way, we need to document this - now that we know P1Y has different meaning depending on the format of the date!

[hjoliver]

Good point (abbreviation). OK, I guess clear documentation will have to do for now.

[oliver-sanders]

So is the problem here essentially that the ordinal day initial point gets converted to the default cycle point format, which is not really compatible with ordinal counting?

I would guess so. Isodatetime has different calendar modes, calculcations performed within a calendar mode make some sort of sense, once you convert to a different mode things start to go astray. We should keep TimePoints in the mode they were created in for all calculations for safety?

I think the issue here is largely that ISO8601 standardises only the representation of dates and times and intervals (not durations) so has no specification for how arithmetic should be performed between them.

• How should leap years (or leap days as I prefer to call them) behave when we add one year to a decimal date (e.g. decimal date representing the 29th of Feb + P1Y)?
• What happens if we add one month to a decimal date?

I believe what we are doing is beyond ISO8601 so we are left to make up the rule book as we go along. Perhaps it isn't really sensible to attempt this and we need to make decimal day cycling a different cycling mode all to itself and ban calculations which cross cycling modes. Perhaps we can come up with some safe sensible solutions.

To add to all this fun ISO8601-1 specifies a new format

• week of year - day of week YYYY-Www-D

And ISO8601-2 specifies a few [optional] new formats including:

• Seasons YYYY-SS (this kinda makes sense)
• Date-time globs e.g. YYYY-XX-XX (this could be usefull)
• Approximations e.g. YYYY-MM?-DD (ok this is too far!)