Merge pull request #12 from xcube-dev/konstntokas-xxx-large_datarequest

Support large dataset requests for stac and cds

Author: konstntokas

CHANGES.md

@@ -1,4 +1,4 @@
-## Changes in 0.3.0 (in development)
+## Changes in 0.3.0
 
 * **Unified spatial resampling**: Switched to the new
   `xcube_resampling.resample_in_space` library, replacing
@@ -14,6 +14,9 @@
   all requested geolocations of the datacubes to be generated.
 * Fixed visualization of preload handle
 * Experimental configuration setup GUI added using panel.
+* Implemented a new access handler that supports large dataset requests by
+  automatically splitting and retrieving data from the `"cds"`, `"stac-cdse-ardc"`,
+  and `"stac-pc-ardc"` data stores.
 
 
 ## Changes in 0.2.0

README.md

@@ -16,4 +16,4 @@
 xcube Multi-Source Data Store: Seamlessly Integrating and Harmonizing Data from 
 Multiple Sources.
 
-Find out more in the [xcube Multi-Source Data Store Documentation](https://xcube-dev.github.io/xcube-multistore/).
+Find out more in the [xcube Multi-Source Data Store Documentation](https://xcube-dev.github.io/xcube-multistore/).

docs/config.md

@@ -54,6 +54,7 @@ This configures a dataset object from a single data source.
 * **[data_id](#data_id)**
 * **[open_params](#open_params)**
 * **[format_id](#format_id)**
+* **[chunksize](#chunksize)**
 * **[custom_processing](#custom_processing)**
 * **[spatial_resample_params](#spatial_resample_params)**
 * **[temporal_resample_params](#temporal_resample_params)**
@@ -69,6 +70,7 @@ opened individually, and once all datasets are loaded, they are merged using
 * **[identifier](#identifier)**: This identifier defines the name of the final data cube.
 * **[grid_mapping](#grid_mapping)**
 * **[format_id](#format_id)**
+* **[chunksize](#chunksize)**
 * **[xr_merge_params](#xr_merge_params)**
 * **variables**: List of [variable object](#variable-object) objects.
 
@@ -152,7 +154,7 @@ Identifier that assigns a grid mapping to the final dataset for reprojection.
 Unique identifier for the dataset's data source within the assigned data store.
 
 ### open_params
-Open data parameters related to the data store and data_id
+Open data parameters related to the data store and data_id.
 
 ### xr_merge_params
 Parameters of `xarry.merge` needed if harmonization of multiple datasets into one
@@ -165,6 +167,9 @@ Desired format of the saved datacube.
 **Default:** `zarr`  
 **Allowed values:** `netcdf`, `zarr`, [`levels`](https://xcube.readthedocs.io/en/latest/mldatasets.html#the-xcube-levels-format)
 
+### chunksize
+Optional mapping from dimension to chunk size.
+
 ### spatial_resample_params
 This section enables user to define the parameters for spatial resampling of 
 the dataset. For a full list of supported parameters, refer to the

docs/index.md

@@ -62,13 +62,34 @@ For further examples please view the [examples folder](https://github.com/xcube-
 * support preload API for [xcube-clms](https://github.com/xcube-dev/xcube-clms) and 
   [xcube-zendoo](https://github.com/xcube-dev/xcube-zenodo)
 * allow to write to netcdf and zarr
+* some auxiliary functionalities which shall help to setup a config YAML file.
+* interpolate along the time axis
 
-> The following features will be implemented in the future:
+### Configuration Generator GUI
 
-* some auxiliary functionalities which shall help to setup a config YAML file. 
-* interpolate along the time axis 
+The **Configuration Generator GUI** provides an interactive interface for creating and
+editing the configuration YAML, making the setup process more intuitive and less
+error-prone.
 
-### License
+**Key features (in development):**
+
+- Display of all available fields for each configuration section
+- Dynamic fetching and updating of valid parameters and inputs
+- Dropdown menus that show only supported options
+- Autofill assistance for large option sets (e.g., thousands of data IDs)
+- Built-in configuration validator/checker
+- Geolocation visualization to help define bounding boxes
+
+> **Note:** This feature is under active development, and only a minimal working
+> example is currently available.
+
+To launch the GUI, run the following command from the package root:
+
+```bash
+panel serve xcube_multistore/gui/app.py --dev
+```
+
+## License
 
 The package is open source and released under the 
 [MIT license](https://opensource.org/license/mit). :heart:

environment.yml

@@ -13,13 +13,13 @@ dependencies:
   - tabulate
   - xarray
   - xcube >=1.9.0
-  - xcube-resampling
+  - xcube-resampling >=0.2.3
   - yaml
   # data store plugins
   - xcube-cci >=0.11.2
   - xcube-cds >=1.0.0
   - xcube-clms >=0.2.2
-  - xcube-stac >=1.1.0
+  - xcube-stac >=1.1.2
   - xcube-zenodo >=1.0.0
   # Development Dependencies - Tools
   - black

test/accessors/test_cds.py

@@ -20,40 +20,66 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from xcube.core.store import new_data_store
-from xcube.util.jsonschema import JsonObjectSchema,JsonNumberSchema
+import unittest
+from unittest.mock import MagicMock
+
 import xarray as xr
-from unittest.mock import patch
+from xcube.core.store import DataStoreError
 
 from xcube_multistore.accessors.cds import CdsAccessor
-from ..sample_data import get_sample_data_3d
 
-import unittest
+from ..sample_data import get_sample_data_3d
 
 
 class CdsAccessorTest(unittest.TestCase):
 
     def setUp(self):
-        ds_3d = get_sample_data_3d()
-        memory_store = new_data_store("memory", root="datasource")
-        memory_store.write_data(ds_3d, "era5_dataset3.zarr", replace=True)
-        self.accesor = CdsAccessor(memory_store)
+        self.ds_3d = get_sample_data_3d()
+        self.accessor = CdsAccessor(MagicMock(), MagicMock(), "era5-land", MagicMock())
 
     def test_open_data(self):
-        ds = self.accesor.open_data("era5_dataset3.zarr")
-        self.assertIsInstance(ds, xr.Dataset)
-        self.assertCountEqual(["band_1"], ds.data_vars)
-        self.assertEqual([10, 3, 3], [ds.sizes["time"], ds.sizes["lat"], ds.sizes["lon"]])
+        self.accessor.store.open_data.return_value = self.ds_3d
+        ds = self.accessor.open_data(
+            "era5-land", time_range=("2025-01-01", "2025-01-10")
+        )
+        xr.testing.assert_equal(self.ds_3d, ds)
+        self.accessor.store.open_data.assert_called_once()
 
-    @patch("xcube.core.store.fs.store.BaseFsDataStore.open_data")
-    @patch("xcube.core.store.fs.store.BaseFsDataStore.get_open_data_params_schema")
-    def test_open_data_spatial_res(self, mock_open_params_schema, mock_open_data):
-        mock_open_params_schema.return_value = JsonObjectSchema(
-            properties=dict(spatial_res=JsonNumberSchema(minimum=0.1 ))
+    def test_open_data_spatial_res(self):
+        self.accessor.store.open_data.return_value = self.ds_3d
+        ds = self.accessor.open_data(
+            "era5-land",
+            time_range=("2025-01-01", "2025-01-10"),
+            point=(5.0, 40.0),
         )
-        mock_open_data.return_value = get_sample_data_3d()
-        ds = self.accesor.open_data("era5_dataset3.zarr", point=(5.0, 40.0))
         self.assertIsInstance(ds, xr.Dataset)
         self.assertCountEqual(["band_1"], ds.data_vars)
         self.assertCountEqual(("time",), ds.dims)
         self.assertEqual([10], [ds.sizes["time"]])
+
+    def test_open_data_splits(self):
+        # Fail first call, succeed sequentially on split halves
+        def side_effect(*args, **kwargs):
+            time_range = kwargs.get("time_range")
+            if time_range == ("2025-01-01", "2025-01-10"):
+                raise Exception("Too large request")
+            if time_range == ("2025-01-01", "2025-01-05"):
+                return self.ds_3d.isel(time=slice(0, 5))
+            if time_range == ("2025-01-06", "2025-01-10"):
+                return self.ds_3d.isel(time=slice(5, 10))
+            raise AssertionError("Unexpected time_range")
+
+        self.accessor.store.open_data.side_effect = side_effect
+        ds = self.accessor.open_data(
+            "era5-land", time_range=("2025-01-01", "2025-01-10")
+        )
+        xr.testing.assert_equal(self.ds_3d, ds)
+
+    def test_open_with_split_base_case_error(self):
+        self.accessor.store.open_data.side_effect = Exception("fail always")
+
+        with self.assertRaises(DataStoreError) as cm:
+            _ = self.accessor.open_data(
+                "era5-land", time_range=("2025-01-01", "2025-01-10")
+            )
+        self.assertIn("Cannot further split time range", str(cm.exception))

test/accessors/test_clms.py

@@ -20,15 +20,15 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from xcube.core.store import new_data_store, PreloadedDataStore
-from xcube.util.jsonschema import JsonObjectSchema,JsonNumberSchema
+import unittest
+from unittest.mock import MagicMock
+
 import xarray as xr
-from unittest.mock import patch
+from xcube.core.store import new_data_store
 
 from xcube_multistore.accessors.clms import ClmsAccessor
-from ..sample_data import get_sample_data_3d
 
-import unittest
+from ..sample_data import get_sample_data_3d
 
 
 class ClmsAccessorTest(unittest.TestCase):
@@ -37,11 +37,16 @@ def setUp(self):
         self.ds_3d = get_sample_data_3d()
         memory_store = new_data_store("memory", root="datasource")
         memory_store.cache_store = new_data_store("memory", root="cache_datadource")
-        memory_store.cache_store.write_data(self.ds_3d, "clms_storage|clms_dataset.zarr", replace=True)
-        self.accesor = ClmsAccessor(memory_store)
+        memory_store.cache_store.write_data(
+            self.ds_3d, "clms_storage|clms_dataset.zarr", replace=True
+        )
+        storage_store = new_data_store("file", root="data")
+        self.accesor = ClmsAccessor(memory_store, storage_store, "test", MagicMock())
 
     def test_open_data_cache_store(self):
         ds = self.accesor.open_data("clms_storage|clms_dataset.zarr")
         self.assertIsInstance(ds, xr.Dataset)
         self.assertCountEqual(["band_1"], ds.data_vars)
-        self.assertEqual([10, 3, 3], [ds.sizes["time"], ds.sizes["lat"], ds.sizes["lon"]])
+        self.assertEqual(
+            [10, 3, 3], [ds.sizes["time"], ds.sizes["lat"], ds.sizes["lon"]]
+        )

test/accessors/test_stac.py

@@ -0,0 +1,71 @@
+# MIT License
+#
+# Copyright (c) 2025 Brockmann Consult GmbH
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NON INFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import unittest
+from unittest.mock import MagicMock
+
+import xarray as xr
+from xcube.core.store import new_data_store
+
+from xcube_multistore.accessors.stac import StacAccessor
+
+from ..sample_data import get_sample_data_2d
+
+
+class StacAccessorTest(unittest.TestCase):
+
+    def setUp(self):
+        self.ds_2d = get_sample_data_2d()
+        self.storage = new_data_store("memory", root="data")
+        self.accessor = StacAccessor(MagicMock(), self.storage, "test", MagicMock())
+
+    def test_open_data(self):
+        self.accessor.store.open_data.return_value = self.ds_2d
+        ds = self.accessor.open_data(
+            "sentinel-2-l2a",
+            bbox=[9, 54, 11, 56],
+            time_range=("2025-01-01", "2025-01-31"),
+            spatial_res=0.0001,
+        )
+        self.assertIsInstance(ds, xr.Dataset)
+        self.assertCountEqual(["band_1"], ds.data_vars)
+        self.assertCountEqual(("time", "lat", "lon"), ds.dims)
+        self.assertEqual(
+            [9, 9, 9], [ds.sizes["time"], ds.sizes["lat"], ds.sizes["lon"]]
+        )
+
+    def test_open_data_point_request(self):
+        self.accessor.store.open_data.return_value = self.ds_2d
+        ds = self.accessor.open_data(
+            "sentinel-2-l2a",
+            point=(10, 55),
+            bbox_width=4000,
+            time_range=("2020-01-01", "2020-12-31"),
+            spatial_res=10,
+            asset_names=["B02"],
+        )
+        self.assertIsInstance(ds, xr.Dataset)
+        self.assertCountEqual(["band_1"], ds.data_vars)
+        self.assertCountEqual(("time", "lat", "lon"), ds.dims)
+        self.assertEqual(
+            [3, 9, 9], [ds.sizes["time"], ds.sizes["lat"], ds.sizes["lon"]]
+        )

test/accessors/test_zenodo.py

@@ -20,13 +20,15 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-from xcube.core.store import new_data_store
+import unittest
+from unittest.mock import MagicMock
+
 import xarray as xr
+from xcube.core.store import new_data_store
 
 from xcube_multistore.accessors.zenodo import ZenodoAccessor
-from ..sample_data import get_sample_data_3d
 
-import unittest
+from ..sample_data import get_sample_data_3d
 
 
 class ZenodoAccessorTest(unittest.TestCase):
@@ -36,17 +38,24 @@ def setUp(self):
         memory_store = new_data_store("memory", root="datasource")
         memory_store.write_data(self.ds_3d, "dataset.zarr", replace=True)
         memory_store.cache_store = new_data_store("memory", root="cache_datadource")
-        memory_store.cache_store.write_data(self.ds_3d, "zenodo_cache/dataset.zarr", replace=True)
-        self.accesor = ZenodoAccessor(memory_store)
+        memory_store.cache_store.write_data(
+            self.ds_3d, "zenodo_cache/dataset.zarr", replace=True
+        )
+        storage_store = new_data_store("file", root="data")
+        self.accesor = ZenodoAccessor(memory_store, storage_store, "test", MagicMock())
 
     def test_open_data(self):
         ds = self.accesor.open_data("dataset.zarr")
         self.assertIsInstance(ds, xr.Dataset)
         self.assertCountEqual(["band_1"], ds.data_vars)
-        self.assertEqual([10, 3, 3], [ds.sizes["time"], ds.sizes["lat"], ds.sizes["lon"]])
+        self.assertEqual(
+            [10, 3, 3], [ds.sizes["time"], ds.sizes["lat"], ds.sizes["lon"]]
+        )
 
     def test_open_data_cache_store(self):
         ds = self.accesor.open_data("zenodo_cache/dataset.zarr")
         self.assertIsInstance(ds, xr.Dataset)
         self.assertCountEqual(["band_1"], ds.data_vars)
-        self.assertEqual([10, 3, 3], [ds.sizes["time"], ds.sizes["lat"], ds.sizes["lon"]])
+        self.assertEqual(
+            [10, 3, 3], [ds.sizes["time"], ds.sizes["lat"], ds.sizes["lon"]]
+        )

test/test_config.py

@@ -20,7 +20,6 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
-import sys
 import unittest
 from unittest.mock import patch
 
@@ -123,6 +122,5 @@ def test_is_jupyter_notebook(self):
             self.assertFalse(_is_jupyter_notebook())
 
         with patch("IPython.get_ipython") as mock_get_ipython:
-            mock_get_ipython.return_value = object()   # anything non-None
+            mock_get_ipython.return_value = object()  # anything non-None
             self.assertTrue(_is_jupyter_notebook())
-