Fix kerchunk JSON file filtering#54
Conversation
eb594f1 to
6360f07
Compare
|
I hope I didn't miss some reason why the start/end dates were the way they were. Also let me know if the changes in |
|
Ok I see what you are doing here. I remember working on this but clearly didn't leave it working correctly! Can you add a test to demonstrate this working correctly? Be sure to include the edge case of starting a test right at the end of the year that will need simulation output from both that year and the following year (ideally it would work if the simulation was running backward in time too, though I haven't been testing that behavior yet). |
- refactor to deduplicate code - simplify how dates are passed around for the kerchunk functions - this was originally done to fix the date format mismatch when filtering the JSON files but this bug got fixed while this PR was waiting for review so it remains as mostly just a refactor after rebasing and resolving conflicts
9d557bd to
315fa48
Compare
|
Rebased onto main and resolved conflicts. Since the bug was fixed in main, this is now just more of a refactor and adding of tests. |
|
@honzaflash Ok I took a look and also added a comment to whats new since I forgot for the last PR. Looks good! |
Changes
datetimeand dates from the filenames get converted to it to compareThe issue before:
make_ciofs_kerchunkwould receive dates as string formatted likeYYYY_0DDDorYYYY-MM-DDdatetime.strptime(Path(j).stem, date_format).isoformat()gettingYYYY-MM-DDT...CIOFSthis would lead to comparison like this'2022_0213' <= '2022-09-01'which is false'2022_0213' == datetime(2022,8,1).strftime('%Y_0%j')and 2022-08-01 is before 2022-09-01'_' <= '-'is falseThe fixed issue after:
make_ciofs_kerchunknow receives start/end dates asdatetimeobjectsdatetimeobjects as well for the comparisonbase_strcould always be refactored to also accept adatetimeobject instead of the year 'YYYY' stringTesting