Currently the outputs generated from the eval script create unnecessary friction when performing analysis, and may lead to unintended errors. I am including an example notebook below, which highlights several ways in which the analysis process is hindered.
-
The data is separated by level for ease of ML training, however, given that the eval outputs are not used in this context, we introduce the step of concatenating variables back to their respective depths, while this is not a big lift, and mistakes in doing this could an additional place for errors.
-
The horizontal coordinates of the eval outputs do not align with the true data (['lat','lon'] vs ['y','x']) this is unnecessary and requires manually renaming axes
-
We are missing most of the grid information necessary for performing calculations. At the moment, I address this by recomputing them from the same grid files used in processing the data; however, for future proofing, I think that this grid information should either live in the eval datasets, or we should create a grid zarr for each resolution and store a path to that grid in the attrs. These differences, have an impact on conservation as slight changes to depth, area, or lat/lon locations can make a difference
Sample_Eval_Notebook_OM4.ipynb
Currently the outputs generated from the eval script create unnecessary friction when performing analysis, and may lead to unintended errors. I am including an example notebook below, which highlights several ways in which the analysis process is hindered.
The data is separated by level for ease of ML training, however, given that the eval outputs are not used in this context, we introduce the step of concatenating variables back to their respective depths, while this is not a big lift, and mistakes in doing this could an additional place for errors.
The horizontal coordinates of the eval outputs do not align with the true data (['lat','lon'] vs ['y','x']) this is unnecessary and requires manually renaming axes
We are missing most of the grid information necessary for performing calculations. At the moment, I address this by recomputing them from the same grid files used in processing the data; however, for future proofing, I think that this grid information should either live in the eval datasets, or we should create a grid zarr for each resolution and store a path to that grid in the attrs. These differences, have an impact on conservation as slight changes to depth, area, or lat/lon locations can make a difference
Sample_Eval_Notebook_OM4.ipynb