|
2 | 2 | { |
3 | 3 | "objectID": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html", |
4 | 4 | "href": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html", |
5 | | - "title": "Store CRS information in a auxiliary variable specified in grid_mapping", |
| 5 | + "title": "Create a GeoZarr with multi-scales containing the WebMercatorQuad TMS", |
6 | 6 | "section": "", |
7 | | - "text": "import json\n\nimport cf_xarray # noqa\nimport dask.array as da\nimport matplotlib.pyplot as plt\nimport morecantile\nimport numpy as np\nimport panel\nimport rasterio\nimport rioxarray # noqa\nimport xarray as xr\nimport zarr\nfrom rio_tiler.io.xarray import XarrayReader\n\n\nfp_base = \"20020601090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1\"\ninput = f\"../data/{fp_base}.nc\"\nv3_output = f\"../output/v3/{fp_base}_multiscales.zarr\"\n\n\nds = xr.open_dataset(input)", |
| 7 | + "text": "While GeoZarr v0.4 is Zarr V2 specific, let’s write a Zarr V3 store to get an idea about how GeoZarr could be adapted for Zarr format 3.", |
8 | 8 | "crumbs": [ |
9 | 9 | "Examples", |
10 | 10 | "WebMercatorQuad overviews (Zarr V3)" |
|
13 | 13 | { |
14 | 14 | "objectID": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#load-example-dataset-from-netcdf-into-xarray", |
15 | 15 | "href": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#load-example-dataset-from-netcdf-into-xarray", |
16 | | - "title": "Store CRS information in a auxiliary variable specified in grid_mapping", |
17 | | - "section": "", |
18 | | - "text": "import json\n\nimport cf_xarray # noqa\nimport dask.array as da\nimport matplotlib.pyplot as plt\nimport morecantile\nimport numpy as np\nimport panel\nimport rasterio\nimport rioxarray # noqa\nimport xarray as xr\nimport zarr\nfrom rio_tiler.io.xarray import XarrayReader\n\n\nfp_base = \"20020601090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1\"\ninput = f\"../data/{fp_base}.nc\"\nv3_output = f\"../output/v3/{fp_base}_multiscales.zarr\"\n\n\nds = xr.open_dataset(input)", |
| 16 | + "title": "Create a GeoZarr with multi-scales containing the WebMercatorQuad TMS", |
| 17 | + "section": "Load example dataset from NetCDF into Xarray", |
| 18 | + "text": "Load example dataset from NetCDF into Xarray\n\nimport json\n\nimport cf_xarray # noqa\nimport dask.array as da\nimport matplotlib.pyplot as plt\nimport morecantile\nimport numpy as np\nimport panel\nimport rasterio\nimport rioxarray # noqa\nimport xarray as xr\nimport zarr\nfrom rio_tiler.io.xarray import XarrayReader\n\n\nfp_base = \"20020601090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1\"\ninput = f\"../data/{fp_base}.nc\"\nv3_output = f\"../output/v3/{fp_base}_multiscales.zarr\"\n\n\nds = xr.open_dataset(input)", |
19 | 19 | "crumbs": [ |
20 | 20 | "Examples", |
21 | 21 | "WebMercatorQuad overviews (Zarr V3)" |
|
24 | 24 | { |
25 | 25 | "objectID": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#check-that-all-variables-have-a-cf-compliant-standard-name", |
26 | 26 | "href": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#check-that-all-variables-have-a-cf-compliant-standard-name", |
27 | | - "title": "Store CRS information in a auxiliary variable specified in grid_mapping", |
| 27 | + "title": "Create a GeoZarr with multi-scales containing the WebMercatorQuad TMS", |
28 | 28 | "section": "Check that all variables have a CF-compliant standard name", |
29 | 29 | "text": "Check that all variables have a CF-compliant standard name\n\nstandard_names = ds.cf.standard_names\nvars_with_standard_names = [v[0] for v in ds.cf.standard_names.values()]\ncompliant_vars = []\nnon_complaint_vars = []\nfor var in ds.variables:\n if var not in vars_with_standard_names:\n non_complaint_vars.append(var)\n else:\n compliant_vars.append(var)\n assert ds[var].attrs[\"standard_name\"]\n\nprint(f\"These variables do NOT have a CF-compliant standard name: {non_complaint_vars}\")\nprint(f\"These variables have a CF-compliant standard name: {compliant_vars}\")\n\nThese variables do NOT have a CF-compliant standard name: ['analysis_error', 'mask']\nThese variables have a CF-compliant standard name: ['time', 'lat', 'lon', 'analysed_sst', 'sea_ice_fraction']\n\n\nNot all the variables in this dataset have a CF-compliant standard name. See https://github.com/zarr-developers/geozarr-spec/issues/60 for a recommendation that CF-compliant standard names should be a “SHOULD” rather than a “MUST” condition in the GeoZarr spec. For now, let’s subset to the variables that do use CF-compliant standard names.\n\nds = ds[compliant_vars]", |
30 | 30 | "crumbs": [ |
|
35 | 35 | { |
36 | 36 | "objectID": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#assign-crs-information-to-an-auxiliary-variable-using-rioxarray", |
37 | 37 | "href": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#assign-crs-information-to-an-auxiliary-variable-using-rioxarray", |
38 | | - "title": "Store CRS information in a auxiliary variable specified in grid_mapping", |
| 38 | + "title": "Create a GeoZarr with multi-scales containing the WebMercatorQuad TMS", |
39 | 39 | "section": "Assign CRS information to an auxiliary variable using rioxarray", |
40 | 40 | "text": "Assign CRS information to an auxiliary variable using rioxarray\n\nds = ds.rio.write_crs(\"epsg:4326\")\n# Specify which variable contains CRS information using grid_mapping\nfor var in ds.data_vars:\n ds[var].attrs[\"grid_mapping\"] = \"spatial_ref\"", |
41 | 41 | "crumbs": [ |
|
46 | 46 | { |
47 | 47 | "objectID": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#specify-that-the-analysed_sst-variable-will-contain-multiscales", |
48 | 48 | "href": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#specify-that-the-analysed_sst-variable-will-contain-multiscales", |
49 | | - "title": "Store CRS information in a auxiliary variable specified in grid_mapping", |
| 49 | + "title": "Create a GeoZarr with multi-scales containing the WebMercatorQuad TMS", |
50 | 50 | "section": "Specify that the analysed_sst variable will contain multiscales", |
51 | 51 | "text": "Specify that the analysed_sst variable will contain multiscales\n\nds[\"analysed_sst\"].attrs[\"multiscales\"] = {\n \"tile_matrix_set\": \"WebMercatorQuad\",\n \"resampling_method\": \"nearest\",\n \"tile_matrix_limits\": {\"0\": {}, \"1\": {}, \"2\": {}},\n}", |
52 | 52 | "crumbs": [ |
|
57 | 57 | { |
58 | 58 | "objectID": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#specify-encoding-and-write-to-zarr-v3-format", |
59 | 59 | "href": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#specify-encoding-and-write-to-zarr-v3-format", |
60 | | - "title": "Store CRS information in a auxiliary variable specified in grid_mapping", |
| 60 | + "title": "Create a GeoZarr with multi-scales containing the WebMercatorQuad TMS", |
61 | 61 | "section": "Specify encoding and write to Zarr V3 format", |
62 | 62 | "text": "Specify encoding and write to Zarr V3 format\n\nspatial_chunk = 4096\ncompressor = zarr.codecs.ZstdCodec(level=1)\nencoding = {\n \"analysed_sst\": {\n \"chunks\": (1, spatial_chunk, spatial_chunk),\n \"compressors\": compressor,\n },\n \"sea_ice_fraction\": {\n \"chunks\": (1, spatial_chunk, spatial_chunk),\n \"compressors\": compressor,\n },\n}\nds.to_zarr(v3_output, mode=\"w\", consolidated=True, zarr_format=3, encoding=encoding)\n\n/Users/max/Documents/Code/developmentseed/geozarr-examples/.pixi/envs/test/lib/python3.13/site-packages/zarr/api/asynchronous.py:203: UserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future.\n warnings.warn(\n\n\n<xarray.backends.zarr.ZarrStore at 0x1655de710>", |
63 | 63 | "crumbs": [ |
|
68 | 68 | { |
69 | 69 | "objectID": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#create-an-empty-xarray-dataset-for-each-zoom-level", |
70 | 70 | "href": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#create-an-empty-xarray-dataset-for-each-zoom-level", |
71 | | - "title": "Store CRS information in a auxiliary variable specified in grid_mapping", |
| 71 | + "title": "Create a GeoZarr with multi-scales containing the WebMercatorQuad TMS", |
72 | 72 | "section": "Create an empty xarray Dataset for each zoom level", |
73 | 73 | "text": "Create an empty xarray Dataset for each zoom level\n\ntms = morecantile.tms.get(\"WebMercatorQuad\")\ntileWidth = 256\nvar = \"analysed_sst\"\ndataset_length = ds[var].sizes[\"time\"]\nzoom_levels = [0, 1, 2]\n\n\ndef create_overview_template(var, standard_name, *, tileWidth, dataset_length, zoom):\n width = 2**zoom * tileWidth\n overview_da = xr.DataArray(\n da.empty(\n shape=(dataset_length, width, width),\n dtype=np.float32,\n chunks=(1, tileWidth, tileWidth),\n ),\n dims=ds[var].dims,\n )\n template = overview_da.to_dataset(name=var)\n template = template.rio.write_crs(\"epsg:3857\")\n # Convert transform to GDAL's format\n transform = rasterio.transform.from_bounds(*tms.xy_bbox, width, width)\n transform = transform.to_gdal()\n # Convert transform to space separated string\n transform = \" \".join([str(i) for i in transform])\n # Save as an attribute in the `spatial_ref` variable\n template[\"spatial_ref\"].attrs[\"GeoTransform\"] = transform\n template[var].attrs[\"grid_mapping\"] = \"spatial_ref\"\n template[var].attrs[\"standard_name\"] = standard_name\n return template", |
74 | 74 | "crumbs": [ |
|
79 | 79 | { |
80 | 80 | "objectID": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#write-overview-template-with-no-data-to-zarr-store", |
81 | 81 | "href": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#write-overview-template-with-no-data-to-zarr-store", |
82 | | - "title": "Store CRS information in a auxiliary variable specified in grid_mapping", |
| 82 | + "title": "Create a GeoZarr with multi-scales containing the WebMercatorQuad TMS", |
83 | 83 | "section": "Write overview template (with no data) to zarr store", |
84 | 84 | "text": "Write overview template (with no data) to zarr store\n\nfor zoom in zoom_levels:\n template = create_overview_template(\n var,\n ds[var].attrs[\"standard_name\"],\n tileWidth=tileWidth,\n dataset_length=dataset_length,\n zoom=zoom,\n )\n template.to_zarr(\n v3_output,\n group=str(zoom),\n compute=False,\n consolidated=False,\n mode=\"w\",\n zarr_format=3,\n )", |
85 | 85 | "crumbs": [ |
|
90 | 90 | { |
91 | 91 | "objectID": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#populate-zarr-array-with-overview-data", |
92 | 92 | "href": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#populate-zarr-array-with-overview-data", |
93 | | - "title": "Store CRS information in a auxiliary variable specified in grid_mapping", |
| 93 | + "title": "Create a GeoZarr with multi-scales containing the WebMercatorQuad TMS", |
94 | 94 | "section": "Populate Zarr array with overview data", |
95 | 95 | "text": "Populate Zarr array with overview data\n\ndef populate_tile_data(dst: XarrayReader, za: zarr.Array, x: int, y: int, zoom: int):\n x_start = x * tileWidth\n x_stop = (x + 1) * tileWidth\n y_start = y * tileWidth\n y_stop = (y + 1) * tileWidth\n tile = dst.tile(x, y, zoom).data\n za[:, y_start:y_stop, x_start:x_stop] = tile\n\n\nmatrices = tms.tileMatrices\n\nwith XarrayReader(ds[var]) as dst:\n for zoom in zoom_levels:\n tm = matrices[zoom]\n za = zarr.open_array(v3_output, path=f\"{zoom}/{var}\", zarr_version=3)\n for x in range(tm.matrixWidth):\n for y in range(tm.matrixHeight):\n populate_tile_data(dst, za, x, y, zoom)", |
96 | 96 | "crumbs": [ |
|
101 | 101 | { |
102 | 102 | "objectID": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#inspect-zarr-v3-store", |
103 | 103 | "href": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#inspect-zarr-v3-store", |
104 | | - "title": "Store CRS information in a auxiliary variable specified in grid_mapping", |
| 104 | + "title": "Create a GeoZarr with multi-scales containing the WebMercatorQuad TMS", |
105 | 105 | "section": "Inspect Zarr V3 store", |
106 | 106 | "text": "Inspect Zarr V3 store\nFirst, let’s look at the structure of Zarr arrays using zarr’s Group.tree() method\n\nroot = zarr.open_group(v3_output, mode=\"r\")\nroot.tree()\n\n/\n├── 0\n│ ├── analysed_sst (1, 256, 256) float32\n│ └── spatial_ref () int64\n├── 1\n│ ├── analysed_sst (1, 512, 512) float32\n│ └── spatial_ref () int64\n├── 2\n│ ├── analysed_sst (1, 1024, 1024) float32\n│ └── spatial_ref () int64\n├── analysed_sst (1, 17999, 36000) float64\n├── lat (17999,) float32\n├── lon (36000,) float32\n├── sea_ice_fraction (1, 17999, 36000) float64\n├── spatial_ref () int64\n└── time (1,) int32\n\n\n\nSecond, let’s look at what’s actually recorded in the Zarr metadata using the consolidated metadata at the root of the Zarr store.\nIn order to match valid JSON, we convert the nan fill_value entries to “nan”.\n\nKey observations\n\nFor each group and array, metadata is stored under the ‘attributes’ key in ‘zarr.json’\nAll arrays contain a attributes/standard_name\nThe dimensions associated with an array are stored under zarr.json/dimension_names (separately from the attributes) rather than _ARRAY_DIMENSIONS\nthe node_type specifies whether the key holds a Zarr Array or a Zarr Group\nThe coordinates associated with an array are still specified within the array metadata. Currently this is an Xarray implementation detail rather than a part of the GeoZarr specification.\nThe fill_value for sea_ice_fraction and analysed_sst is 0 instead of nan. This is likely an error with the fill value not being explicitly specified.\nmetadata/multiscales for analysed_sst contains information about the multiscales, specifying that they are a WebMercatorQuad TMS created using nearest resampling\nThe 0, 1, and 2 groups contain GeoZarr compliant overviews for the analysed_sst variable, including the required standard_name and grid_mapping attributes.\n\n\npanel.extension()\nconsolidated_metadata_file = f\"{v3_output}/zarr.json\"\nwith open(consolidated_metadata_file) as f:\n metadata = json.load(f)[\"consolidated_metadata\"][\"metadata\"]\npanel.pane.JSON(metadata, name=\"JSON\")", |
107 | 107 | "crumbs": [ |
|
112 | 112 | { |
113 | 113 | "objectID": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#plot-one-of-the-zoom-levels", |
114 | 114 | "href": "examples/04_multiscales_as_WebMercatorQuad_ZarrV3.html#plot-one-of-the-zoom-levels", |
115 | | - "title": "Store CRS information in a auxiliary variable specified in grid_mapping", |
| 115 | + "title": "Create a GeoZarr with multi-scales containing the WebMercatorQuad TMS", |
116 | 116 | "section": "Plot one of the zoom levels", |
117 | 117 | "text": "Plot one of the zoom levels\n\nvar = \"analysed_sst\"\nzoom = 2\narr = zarr.open_array(v3_output, path=f\"{zoom}/{var}\")\narr = arr[:]\nplt.imshow(arr.squeeze())", |
118 | 118 | "crumbs": [ |
|
0 commit comments