Converting notebooks from COSIMA Cookbook to ACCESS-NRI intake catalog

[navidcy]

Why we need to have two ways for people to load output? At the moment, after #298, we have tutorials both for the cookbook and for the ACCESS-NRI Intake catalog

https://cosima-recipes.readthedocs.io/en/latest/tutorials.html

Let's just leave the best of the two. Is ACCESS-NRI Intake catalog the future? Let's make that the default in all examples and drop the cookbook then?

cc @angus-g, @aidanheerdegen, @AndyHoggANU, @aekiss, @adele-morrison, @anton-seaice, @PaulSpence, @rmholmes, @edoddridge, @micaeljtoliveira

Recipes

• Along_slope_velocities.ipynb (Migrate alongslope-velocities to intake #498 )
• Along_Isobath_Average.ipynb (Update the along isobath average recipe to use intake #499 )
• Apply_function_to_every_gridpoint.ipynb (Convert apply_function_to_every_gridpoint to Intake #461 )
• Atlantic_IndoPacific_Basin_Overturning_Circulation.ipynb (Combined Overturning Circulation notebook using Intake #533)
• Barotropic_Streamfunction.ipynb (Intake conversion Barotropic_Streamfunction_model_agnostic #351)
• Bathymetry.ipynb (Intake conversion Bathymetry #352)
• Compare_SSH_model_obs.ipynb (Intake conversion Compare_SSH_model_obs #354)
• Compare_SST_SSS_TemperatureSalinity_to_WOA13.ipynb (Intake conversion Compare_SST_SSS_TemperatureSalinity_to_WOA13 #355)
• Cross-contour_transport.ipynb (Intake conversion Cross-contour_transport #356)
• Cross-slope_section.ipynb Update cross-slope section recipe to use intake #512
• Eddy-Mean_Kinetic_Energy_Decomposition.ipynb (Intake conversion Decomposing_kinetic_energy_into_mean_and_transient #358)
• Equatorial_thermal_and_zonal_velocity_structure.ipynb (Intake conversion Equatorial_thermal_and_zonal_velocity_structure #359)
• Extract_Variables_at_Ocean_Bottom.ipynb (Intake conversion Extract_variables_at_ocean_bottom #349)
• Geostrophic_Velocities_from_Sea_Level.ipynb (Intake conversion Geostrophic_Velocities_from_Sea_Level #360)
• Horizontal_Regridding.ipynb (Intake conversion Regridding example; take #2 #413)
• Hovmoller_Temperature_Depth.ipynb (Fixing bug in Hovmoller_Temperature_Depth #466)
• Meridional_Overturning_Circulation.ipynb (Combined Overturning Circulation notebook using Intake #533)
• Meridional_heat_transport.ipynb (Convert Meridional Heat Transport to Intake #517)
• Model_Resolution_Comparison.ipynb (Intake conversion Model_Resolution_Comparison #362)
• Nearest_Neighbour_Distance.ipynb (Intake conversion NearestNeighbourDistance #363)
• Neutral_density.ipynb (Update Neutral Density recipe to use Intake #536)
• Particle_tracking_with_Parcels.ipynb (Convert Particle_tracking_with_Parcels notebook to Intake #534)
• Querying_Scalar_Quantities_and_Annually_Averaged_Timeseries.ipynb (Intake conversion Querying_Scalar_Quantities_and_Annually_Averaged_Timeseries #365)
• RelativeVorticity.ipynb (Intake conversion of Relative_Vorticity; take #2 #426)
• Sea_Ice_Area_Concentration_Volume_with_Obs.ipynb
• Sea_Ice_Coordinates.ipynb (Intake conversion SeaIce_Plot_Example #369)
• Sea_Ice_Seasonality_Statistics.ipynb Intake sea ice seasonality #518
• Surface_Water_Mass_Transformation.ipynb (Update Surface_Water_Mass_Transformation to use access_nri_intake #520)
• Temperature_Salinity_Diagram.ipynb (Intake conversion TemperatureSalinityDiagrams #371)
• Transformation_from_Depth_to_Potential_Density.ipynb (Update notebook #522)
• Transport_Through_Straits.ipynb (Intake conversion Transport_Through_Straits #372)
• True_Zonal_Mean.ipynb (Intake conversion True_Zonal_Mean #373)

Tutorials

• Animations_with_xmovie.ipynb
• Making_Maps_with_Cartopy.ipynb (Intake conversion Making_Maps_with_Cartopy #377)
• Model_Agnostic_Analysis.ipynb (Intake conversion Model_Agnostic_Analysis #378)
• Spatial_selection.ipynb (Intake conversion Spatial_selection #379)
• Submitting_analysis_jobs_to_gadi.ipynb (Intake conversion Submitting_analysis_jobs_to_gadi #380)
• Template_For_Notebooks.ipynb (Intake conversion Template_For_Notebooks #381)
• Using_Explorer_tools.ipynb (Intake conversion Using_Explorer_tools #382)

ACCESS-OM2-GMD-Paper-Figs

• Fig12-ZonalTempSaltBias.ipynb (INTAKE_Fig12-ZonelTempSalt #524)
• Fig3-GlobalTimeseries.ipynb (Intake conversion Fig3-GlobalTimeseries #344)
• Fig4-DrakePassageTransport.ipynb (Intake conversion Fig4-DrakePassageTransport #345)
• Figs14-20-23-24-PlotWOCETransects.ipynb (Intake conversion Figs14-20-23-24-PlotWOCETransects #346)

Admin

• Remove references to COSIMA Cookbook from README, documentation, etc. (Final documentation updates for Intake conversion #539)

[navidcy]

#310 seems relevant

[aidanheerdegen]

Largely this is up to the COSIMA community, but it is likely a choice you will be forced to make for a few reasons:

1. The Cookbook Database does not scale well, particularly in the time taken to index data
2. The infrastructure which runs the indexing will likely cease to exist by the end of the first quarter of 2024 (the accessdev Virtual Machine is scheduled to be shuttered, and this is what provides the outward facing web address for the jenkins service)
3. There are no resources to do more than just maintain the Cookbook DB (@micaeljtoliveira does what he can, but he has many other high priority tasks)
4. ACCESS-NRI has chosen to support the Intake Catalogue (in part for some of the reasons above)

So I'd suggest taking this to a COSIMA meeting and making the community aware that a decision needs to be made, and probably the sooner the better.

One downside of dropping the Cookbook is there is currently no GUI interface for searching the Intake Database. However there are some powerful methods for searching (filtering) the intake catalogue

https://access-nri-intake-catalog.readthedocs.io/en/latest/usage/quickstart.html#data-discovery

Also there are plans for some tools to assist with data discovery from ACCESS-NRI, and we're working on prototypes we hope to share soon.

[navidcy]

Thanks @aidanheerdegen!
I'm happy to hear also voices from others here and then take it to a COSIMA meeting -- sounds good!

[adele-morrison]

The advice we had from ACCESS-NRI on this back around the workshop last year was that at that time they didn’t advise switching everything to intake catalogue, because it was so new and untested. But @aidanheerdegen @dougiesquire if you think the intake catalogue is ready for us to switch everything over completely, i think we should do that. We discussed this previously at a COSIMA meeting and agreed we should switch when the time was right (which is maybe now?).

[anton-seaice]

Not much to add from me:

• There hasn't been much uptake of the intake catalogue, so there may still be bugs/missing datasets etc that will need attention.
• We should keep the cookbook going in its current state (i.e. without adding new data or updates) for existing users, and keep the cookbook tutorials available for people who are halfway through projects for some time. This could be a simple as a tag on the repository, or a fork/branch of the repo, or a seperate folder in the main branch (depending if we want the tutorials on the read the docs version or are happy with them only in github).
• It would be nice to have the test pipeline replaced prior to the switch (which also relies on accessdev), which could make reviewing notebooks more robust too.

[dougiesquire]

I think @max-anu has been modifying the existing recipes to use the catalog. It would be good to get an update from him. There are definitely datasets that are in the cookbook but not yet in the catalog. I think @max-anu will probably have the best idea of what these are.

[aidanheerdegen]

@anton-seaice makes a good point about retaining existing capability for those who rely on it, identifying deficiencies and setting up a new testing infrastructure.

@dougiesquire is also correct that datasets that are missing from intake should be identified.

@adele-morrison I'm not in a position to say if the intake catalogue is ready or not, but we're rapidly approaching a point where there isn't really an option, so we need to make sure it is.

So, synthesising from above, some steps to consider:

• Decide on deprecation plan for cookbook DB (timing to stop ingesting new data and freeze DB, where/how to archive CC DB recipes for medium/long term support)
• Set up testing infrastructure for intake recipes
• Document datasets missing from intake
• Decide which datasets need to be added, particularly if there are any that must be added before swapping to intake
• Set proposed time-frame for replacing CC DB recipes with intake versions
• Communicate plan to community, and get users testing out intake catalogue to find weak points
• Deprecate CC DB and replace with intake recipes in main and docs

Might be good to break this out into a GitHub project, make some issues etc. I would do this, but I'm not in a position to assist much I'm afraid.

[Thomas-Moore-Creative]

Could the culture be shifted going forward so that when someone / a group runs an experiment that is meant to be open to the community that creating an Intake-ESM datastore is part of the process of making it available and documented? My view is that it's the "authors" of specific experiments that are best placed to create and update datastores for the catalog?

@dougiesquire appears to have done a great job documenting how that can be done for those who've never built a catalog before > I want to catalog my own data to make it easy to find and load

This new catalog is a big step forward, IMO, but will require a community effort to maintain, in my view.

[rbeucher]

Hi All,

So from the MED team perspective, I asked @max-anu to look at converting the recipes to use the intake catalog.
The main reason for going ahead with this was precisely to identify the gaps that are mentioned above (missing functionalities, datasets etc.) and to push for a broader use of the intake-catalog. @max-anu should be able to give us an update.

#310 is supposed to be a draft and everyone is welcome to participate and/or comment on it.

Happy to discuss the plan going ahead.

[rbeucher]

I have started a repo on the ACCESS-NRI organisation to automate testing of the recipes:
https://github.com/ACCESS-NRI/COSIMA-recipes-workflow

This is based on what I have set up for the ESMValTool recipes:

https://github.com/ACCESS-NRI/ESMValTool-workflow?tab=readme-ov-file#recipes-current-status

Will keep you posted.