Refresh notebook template for `odc-stac` adoption and improved user guidance

[cbur24]

As we begin transitioning notebooks to use odc-stac instead of datacube, we need to update the standard notebook template. This includes adding a new top-of-notebook information block to inform users about the change, as well as refreshing the overall notebook introduction to include updated guidance and resources. Ensuring consistent standards and clear user guidance will make it easier for users to adapt to the change from datacube to odc-stac.

Proposed changes:

• Add a prominent Important Note at the top of each notebook, e.g.:

  ⚠️ Important Note
  This notebook loads data into xarray using odc-stac. This means:

  • The notebook can be run either within the sandbox, or on any computer with an internet connection and the required Python packages installed (e.g. by running pip install dea-tools in your Python environment).
  • Data loading performance will depend on the available internet bandwidth.

• Consider refreshing the entire top block of the notebook template to:

  • Point to resources like the Knowledge Hub.
  • Mention options for running notebooks on platforms like Google Colab or Binder (or include magic links?).
  • Include links to relevant DEA documentation and support channels.

Priority: Needed before large-scale notebook updates. e.g. #1370

[GL-S]

I don't recall the technical details, but wasn't there a discussion in the recent past about possible issues in accessing lots of data at the same time using STAC? Was that resolved?

[cbur24]

Somewhat resolved I believe. The issue was the explorer page crashing from too many bot requests

[cbur24]

Perhaps something like this at the top each notebook? I think we need to organise the dea-notebooks repo to contain a simple environment.yaml file or requirement.txt file at the top level of the directory for binder to work. And for codespaces we might need a .devcontainer folder

• Using binder exampes: https://mybinder.readthedocs.io/en/latest/examples/sample_repos.html
• CodeSpaces example: https://github.com/ceos-org/cloud-native-geospatial-eo-workshop

Branch with draft update notebooks template: https://github.com/GeoscienceAustralia/dea-notebooks/blob/template/DEA_notebooks_template.ipynb

---

How you can run this notebook:

• DEA's Sandbox: Some users may be eligible to sign-up for DEA's Sandbox environment which provides a small amount of free compute and an ODC-compatible Jupyter-lab python environment.
• Personal computer: This notebook loads DEA data using odc-stac. This means data is streamed across an internet connection so can be run on your personal computer provided you have the the required Python packages installed. Running pip install dea-tools>0.4 will usually suffice.
• Run this notebook online: You can also use the links below to launch either a binder or GitHub Codespaces cloud compute instance to run this notebook:

[robbibt]

Yeah, I think something like that would be really nice! We can workshop the wording a little to keep it as streamlined as possible - I'm also interested in experimenting with some nice HTML styling to make it quite visual and nice to look at, but that's kind of an optional extra.

For Binder, we have a pyproject.toml at the top level which theoretically should be everything we need and more modern than requirements.txt; I'd be suprised if that's not supported but it's possible. I also think services like Google Colab might need any package installation to happen in the notebook which would be a pain - would be far nicer if we could get the environment to set up automatically.

Edit: at least at a first attempt, doesn't seem like Binder is installing from the pyproject.toml. I guess we could include some kind of automated step to export a requirements.txt file.

[robbibt]

Another issue is the giant size of the repo - Binder takes ages just to transfer the required data. We might need to consider archiving some of our older branches and history and starting fresh to reduce the repo size:

[robbibt]

Oh neat, this looks really nice! I think the alert is great for getting attention - I like it. I do think we can probably think about streamlining the original top section - some of that material may no longer be as important as it once was and it would be good to keep things as minimal as possible with the new "access" text.

We should be able to get multiple badges onto one line too.

[robbibt]

OK, shameless AI assistance for this, but remarkably something like this renders fine in a notebook cell!

<!-- DEA Notebook Header -->
<table style="width:100%; border:1px solid #e2e8f0; border-radius:12px; background:#f8fafc; font-family:'Segoe UI',Roboto,sans-serif; line-height:1.6;">
<tr>
<td style="padding:20px; vertical-align:top; width:80%;">

<h1 style="margin-top:0; color:#004f7c;">Downloading data with STAC</h1>

<p>
🔎 Visit <a href="https://knowledge.dea.ga.gov.au">DEA's Knowledge Hub</a> for detailed technical information including methods, quality, and data access.
</p>

<!-- Vertical bubbles -->
<div style="margin-bottom:12px; padding:12px; background:#ffffff; border:1px solid #e5e7eb; border-radius:8px;">
<b>📦 Products used</b><br>
<a href="https://knowledge.dea.ga.gov.au/products/ga_s2am_ard_3">ga_s2am_ard_3</a>, 
<a href="https://knowledge.dea.ga.gov.au/products/ga_s2bm_ard_3">ga_s2bm_ard_3</a>, 
<a href="https://knowledge.dea.ga.gov.au/products/ga_s2cm_ard_3">ga_s2cm_ard_3</a>, 
<a href="https://knowledge.dea.ga.gov.au/products/ga_ls5t_ard_3">ga_ls5t_ard_3</a>, 
<a href="https://knowledge.dea.ga.gov.au/products/ga_ls7e_ard_3">ga_ls7e_ard_3</a>, 
<a href="https://knowledge.dea.ga.gov.au/products/ga_ls8c_ard_3">ga_ls8c_ard_3</a>
</div>

<div style="margin-bottom:12px; padding:12px; background:#ffffff; border:1px solid #e5e7eb; border-radius:8px;">
<b>⚙️ Special requirements</b><br>
Optional notes, e.g. If running on the NCI, ensure that <code>module load otps</code> is run prior to launching this notebook.
</div>

<div style="margin-bottom:12px; padding:12px; background:#ffffff; border:1px solid #e5e7eb; border-radius:8px;">
<b>📖 Prerequisites</b><br>
Optional list of prerequisite notebooks or background knowledge.
</div>

<div style="margin-bottom:12px; padding:14px; background:#ecfdf5; border:1px solid #34d399; border-radius:8px;">
<b>🚀 How you can run this notebook</b><br>

- <b>DEA Sandbox:</b> Some users may be eligible to sign up for DEA's 
  <a href="https://sandbox.dea.ga.gov.au/">Sandbox environment</a> which provides 
  free compute and an ODC-compatible Python environment.<br>

- <b>Personal computer:</b> This notebook loads DEA data using <code>odc-stac</code>. Data streams across an internet connection, so it can be run locally provided you install the required Python packages, e.g. <code>pip install dea-tools[notebooks]</code><br>

- <b>Run this notebook online:</b> Launch a Binder or GitHub Codespaces instance to run the notebook from your browser:<br>

  <a href="https://mybinder.org/v2/gh/GeoscienceAustralia/dea-notebooks/develop?labpath=How_to_guides/Downloading_data_with_STAC.ipynb" target="_blank">
    <img src="https://mybinder.org/badge_logo.svg" alt="Launch Binder" style="height:20px;">
  </a>

  <a href="https://github.com/codespaces/badge.svg)](https://github.com/codespaces/new?hide_repo_select=true&ref=develop&repo=GeoscienceAustralia%2Fdea-notebooks&location=How_to_guides/Downloading_data_with_STAC.ipynb" target="_blank">
    <img src="https://img.shields.io/badge/Codespaces-Open-blue?logo=github" alt="Open in GitHub Codespaces" style="height:20px;">
  </a>
</div>

</td>

<!-- Logo on the right -->
<td style="padding:20px; text-align:right; vertical-align:top; width:20%;">
  <img src="./Supplementary_data/dea_logo.jpg" alt="DEA logo" style="width:220px; max-width:100%; height:auto;">
</td>
</tr>
</table>

[cbur24]

Oooh looks nice @robbibt!

I also haven't been able to get binder to run with the packages installed yet, partly I think because it caches the first image that you try - and if that's wrong and you make a change, it doesn't then seem to reset. Also, is the binder compute size very small? I couldn't find much information on that - this page suggests the data should be less than 10mb. Perhaps instead we should only provide support for github codespaces? In that case, the user can request fairly decent sized machines (and its user configurable). Basically, Codespaces its more of a one-to-one replacement of the Sandbox. If we rely on binder, some (many?) of our notebooks might not run with memory errors or will run slowly.

On that template branch I also includes a .devcontainer file following the pattern from the cloud-native-geospatial workshop repo. Hasn't worked for me yet though. https://github.com/GeoscienceAustralia/dea-notebooks/tree/template/.devcontainer

[robbibt]

I also haven't been able to get binder to run with the packages installed yet, partly I think because it caches the first image that you try - and if that's wrong and you make a change, it doesn't then seem to reset. Also, is the binder compute size very small? I couldn't find much information on that - this page suggests the data should be less than 10mb. Perhaps instead we should only provide support for github codespaces?

Yeah, seems like a 2gb memory limit, which is crazy small. Codespaces and perhaps Google Colab feel like a good option, as I think they at least give you about 16gb of memory. Will have to check how installation works in Colab though.

For the dev container, I think this line will need to be updated. We can likely set it up to install via uv instead of needing a requirements.txt - there's an example here from the template project I used for eo-tides:

• https://github.com/fpgmaas/cookiecutter-uv/blob/main/%7B%7Bcookiecutter.project_name%7D%7D/.devcontainer/devcontainer.json#L14
• https://github.com/fpgmaas/cookiecutter-uv/blob/main/%7B%7Bcookiecutter.project_name%7D%7D/.devcontainer/postCreateCommand.sh

Ping @caitlinadams and @mickwelli as they might have some recent experience with best practice for this stuff. It would be a great thing to discuss at Innovation Day next week too!

[cbur24]

@robbibt @caitlinadams I believe I got an example of codespaces working nicely for the S1-Hackathon, including installing dea-tools. Might be a foundation to build upon for dea-notebooks

docs: https://github.com/cbur24/sentinel-1-hackathon/blob/main/README.md#getting-started-and-python-environment-set-up

devcontainer json set-up: https://github.com/cbur24/sentinel-1-hackathon/tree/main/.devcontainer

[robbibt]

This looks super nice @cbur24 - unfortunately I've already maxed out my quota for codespaces this month so I need to wait another week before I can run it! Pip installing from the requirements.txt looks perfect for the hackathon - during Innovation Day I might have a go at seeing if we can install from the pyproject.toml directly for DEA Tools, but we could always write out a requirements.txt for this purpose there too.

For this, pretty sure you can go:
https://github.com/cbur24/sentinel-1-hackathon/blob/main/requirements.txt#L1-L2C19

dea-tools[jupyter]>=0.4

But it should default to the latest version anyway.

[cbur24]

So if I run only dea-tools[jupyter]>0.4 it will install everything in base dea-tools + the jupyter submodules? I wasn't sure if that was how it works or not.

I think if you use this command in the devcontainer.json, "postCreateCommand": ... you can basically run whatever you want - seems to just pass this command to the terminal window - so should be fairly straightforward to install from the .toml instead