Merge pull request #10 from xcube-dev/konstntokas-xxx-small_things

Enhance small things

Author: konstntokas

CHANGES.md

@@ -1,9 +1,18 @@
 ## Changes in 0.3.0 (in development)
 
-* Switched to the new unified `xcube-resampling` library from 
-  `xcube.core.resampling`.
-* Added `spatial_resample_params` support and updated the config visualization to
-  display the `Spatial Resample Params`.
+* **Unified spatial resampling**: Switched to the new
+  `xcube_resampling.resample_in_space` library, replacing
+  `xcube.core.resampling.resample_in_space`.
+* **Spatial resample parameters**: Added support for `spatial_resample_params`
+  and updated the configuration visualization to display these parameters.
+* **Temporal resampling**: Introduced `xcube_resampling.resample_in_time` with
+  `temporal_resample_params` added to the configuration to cover temporal resampling.
+* **Controlled cube generation**: `xcube_multistore.MultiSourceDataStore` is now only
+  initialized in the constructor without automatically starting cube generation.
+  A new `generate()` method is provided to explicitly start cube creation.
+* **Geolocation visualization**: Added `display_geolocations()` to visualize
+  all requested geolocations of the datacubes to be generated.
+* Fixed visualization of preload handle
 
 
 ## Changes in 0.2.0
@@ -13,6 +22,7 @@
 * Added auxiliary functions to support setting up configurations for cube generation. 
   Examples of all functionalities are provided in the notebook [setup_config.ipynb](examples/setup_config.ipynb).
 
+
 ## Changes in 0.1.0
 
 * The initial version of the Multi-Source Data Store has been developed in 

docs/config.md

@@ -56,6 +56,7 @@ This configures a dataset object from a single data source.
 * **[format_id](#format_id)**
 * **[custom_processing](#custom_processing)**
 * **[spatial_resample_params](#spatial_resample_params)**
+* **[temporal_resample_params](#temporal_resample_params)**
 
 
 #### multi dataset object
@@ -83,6 +84,7 @@ This configures a variable object, which is stored as a data variable within a s
 * **[open_params](#open_params)**
 * **[custom_processing](#custom_processing)**
 * **[spatial_resample_params](#spatial_resample_params)**
+* **[temporal_resample_params](#temporal_resample_params)**
 
 ### store object
 This configures a store object representing one xcube data store instance.
@@ -132,7 +134,6 @@ generation process.
 
 * **[visualize](#visualize)**
 * **[force_preload](#force_preload)**
-* **[dask_scheduler](#dask_scheduler)**
 * **[gdal_http_params](#gdal_http_params)**
 
 
@@ -162,7 +163,7 @@ datacube is required. For further information view the [xarray.merge documentati
 Desired format of the saved datacube.  
 
 **Default:** `zarr`  
-**Allowed values:** `netcdf`, `zarr`
+**Allowed values:** `netcdf`, `zarr`, [`levels`](https://xcube.readthedocs.io/en/latest/mldatasets.html#the-xcube-levels-format)
 
 ### custom_processing
 This section enables users to define a Python function that is executed after opening 
@@ -176,9 +177,18 @@ the dataset. For a full list of supported parameters, refer to the
 [`xcube_resampling.resample_in_space` Python API documentation](https://xcube-dev.github.io/xcube-resampling/api/#xcube_resampling.resample_in_space).
 
 
-_NOTE_: The first 3 arguments `(source_ds, target_gm, source_gm)` in the `xcube_resampling.resample_in_space` are not
-required and will be inferred automatically from the config file's dataset and 
-gridmapping section.
+_NOTE_: The first 3 arguments `(source_ds, target_gm, source_gm)` in the
+`xcube_resampling.resample_in_space` are not required and will be inferred
+automatically from the config file's dataset and gridmapping section.
+
+### temporal_resample_params
+This section enables user to define the parameters for temporal resampling of
+the dataset. For a full list of supported parameters, refer to the
+[`xcube_resampling.resample_in_time` Python API documentation](https://xcube-dev.github.io/xcube-resampling/api/#xcube_resampling.resample_in_time).
+
+_NOTE_: The argument `frequency` is required; the remaining keyword arguments are
+optional.
+
 
 **Properties**:  
 
@@ -202,7 +212,7 @@ below.
 * [file, https, memory, s3](https://xcube.readthedocs.io/en/latest/dataaccess.html#filesystem-based-data-stores)   
 * [sentinelhub, sentinelhub-cdse](https://github.com/xcube-dev/xcube-sh)  
 * [smos](https://github.com/xcube-dev/xcube-smos)  
-* [stac, stac-cdse, stac-xcube](https://github.com/xcube-dev/xcube-stac)
+* [stac, stac-cdse, stac-cdse-ardc, stac-pc, stac-pc-ardc, stac-xcube](https://github.com/xcube-dev/xcube-stac)
 * [zenodo](https://github.com/xcube-dev/xcube-zenodo)  
 
 ---
@@ -243,12 +253,6 @@ only non-preloaded datasets will be preloaded.
 
 **Default:** `True`
 
-### dask_scheduler
-Scheduler mode put into `dask.config.set(scheduler=<scheduler_mode>)`  
-  
-**Default:** `threads`  
-**Allowed values:** `threads`, `processes`, `single-threaded`, `sync`, `distributed`
-
 ### gdal_http_params
 GDAL http environment variables which are used when opening a tif file with 
 `rioxarray.open_rasterio`, which uses GDAL driver under the hood, from a remote source.  

docs/examples/example2/scs2_config.yml

@@ -119,6 +119,3 @@ grid_mappings:
     crs: EPSG:4326
     tile_size: 1000
 
-general:
-  dask_scheduler: "single-threaded"
-

test/accessors/__init__.py

@@ -0,0 +1,21 @@
+# MIT License
+#
+# Copyright (c) 2025 Brockmann Consult GmbH
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NON INFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.

test/accessors/test_cds.py

@@ -0,0 +1,59 @@
+# MIT License
+#
+# Copyright (c) 2025 Brockmann Consult GmbH
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NON INFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from xcube.core.store import new_data_store
+from xcube.util.jsonschema import JsonObjectSchema,JsonNumberSchema
+import xarray as xr
+from unittest.mock import patch
+
+from xcube_multistore.accessors.cds import CdsAccessor
+from ..sample_data import get_sample_data_3d
+
+import unittest
+
+
+class CdsAccessorTest(unittest.TestCase):
+
+    def setUp(self):
+        ds_3d = get_sample_data_3d()
+        memory_store = new_data_store("memory", root="datasource")
+        memory_store.write_data(ds_3d, "era5_dataset3.zarr", replace=True)
+        self.accesor = CdsAccessor(memory_store)
+
+    def test_open_data(self):
+        ds = self.accesor.open_data("era5_dataset3.zarr")
+        self.assertIsInstance(ds, xr.Dataset)
+        self.assertCountEqual(["band_1"], ds.data_vars)
+        self.assertEqual([10, 3, 3], [ds.sizes["time"], ds.sizes["lat"], ds.sizes["lon"]])
+
+    @patch("xcube.core.store.fs.store.BaseFsDataStore.open_data")
+    @patch("xcube.core.store.fs.store.BaseFsDataStore.get_open_data_params_schema")
+    def test_open_data_spatial_res(self, mock_open_params_schema, mock_open_data):
+        mock_open_params_schema.return_value = JsonObjectSchema(
+            properties=dict(spatial_res=JsonNumberSchema(minimum=0.1 ))
+        )
+        mock_open_data.return_value = get_sample_data_3d()
+        ds = self.accesor.open_data("era5_dataset3.zarr", point=(5.0, 40.0))
+        self.assertIsInstance(ds, xr.Dataset)
+        self.assertCountEqual(["band_1"], ds.data_vars)
+        self.assertCountEqual(("time",), ds.dims)
+        self.assertEqual([10], [ds.sizes["time"]])

test/accessors/test_clms.py

@@ -0,0 +1,47 @@
+# MIT License
+#
+# Copyright (c) 2025 Brockmann Consult GmbH
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NON INFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from xcube.core.store import new_data_store, PreloadedDataStore
+from xcube.util.jsonschema import JsonObjectSchema,JsonNumberSchema
+import xarray as xr
+from unittest.mock import patch
+
+from xcube_multistore.accessors.clms import ClmsAccessor
+from ..sample_data import get_sample_data_3d
+
+import unittest
+
+
+class ClmsAccessorTest(unittest.TestCase):
+
+    def setUp(self):
+        self.ds_3d = get_sample_data_3d()
+        memory_store = new_data_store("memory", root="datasource")
+        memory_store.cache_store = new_data_store("memory", root="cache_datadource")
+        memory_store.cache_store.write_data(self.ds_3d, "clms_storage|clms_dataset.zarr", replace=True)
+        self.accesor = ClmsAccessor(memory_store)
+
+    def test_open_data_cache_store(self):
+        ds = self.accesor.open_data("clms_storage|clms_dataset.zarr")
+        self.assertIsInstance(ds, xr.Dataset)
+        self.assertCountEqual(["band_1"], ds.data_vars)
+        self.assertEqual([10, 3, 3], [ds.sizes["time"], ds.sizes["lat"], ds.sizes["lon"]])

test/accessors/test_zenodo.py

@@ -0,0 +1,52 @@
+# MIT License
+#
+# Copyright (c) 2025 Brockmann Consult GmbH
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NON INFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from xcube.core.store import new_data_store
+import xarray as xr
+
+from xcube_multistore.accessors.zenodo import ZenodoAccessor
+from ..sample_data import get_sample_data_3d
+
+import unittest
+
+
+class ZenodoAccessorTest(unittest.TestCase):
+
+    def setUp(self):
+        self.ds_3d = get_sample_data_3d()
+        memory_store = new_data_store("memory", root="datasource")
+        memory_store.write_data(self.ds_3d, "dataset.zarr", replace=True)
+        memory_store.cache_store = new_data_store("memory", root="cache_datadource")
+        memory_store.cache_store.write_data(self.ds_3d, "zenodo_cache/dataset.zarr", replace=True)
+        self.accesor = ZenodoAccessor(memory_store)
+
+    def test_open_data(self):
+        ds = self.accesor.open_data("dataset.zarr")
+        self.assertIsInstance(ds, xr.Dataset)
+        self.assertCountEqual(["band_1"], ds.data_vars)
+        self.assertEqual([10, 3, 3], [ds.sizes["time"], ds.sizes["lat"], ds.sizes["lon"]])
+
+    def test_open_data_cache_store(self):
+        ds = self.accesor.open_data("zenodo_cache/dataset.zarr")
+        self.assertIsInstance(ds, xr.Dataset)
+        self.assertCountEqual(["band_1"], ds.data_vars)
+        self.assertEqual([10, 3, 3], [ds.sizes["time"], ds.sizes["lat"], ds.sizes["lon"]])