Refactor codebase to support a new simplified Parser->ManifestStore model. #601

sharkinsspatial · 2025-05-20T17:29:46Z

Closes
Fragility of url auto-parsing logic #561
Function definition for open_virtual_dataset in next major release #553
Redefine Virtual Readers as func(filepath) -> ManifestStore #498
Should we try to use Obstore for reading files? #476
Support passing configuration options to default_object_store #559
Tests added
Tests passing
Full type hint coverage
Changes are documented in docs/releases.rst
New functions/methods are listed in api.rst
New functionality has documentation

…array

…Store

for more information, see https://pre-commit.ci

…Store

…s://github.com/TomNicholas/VirtualiZarr into refactor-kerchunk-readers-to-use-ManifestStore

…aset.

for more information, see https://pre-commit.ci

chuckwondo · 2025-06-12T14:18:24Z

virtualizarr/parsers/fits.py

+        Instantiate a parser with parser-specific parameters that can be used in the __call__ method.
+        Parameters


Suggested change

Instantiate a parser with parser-specific parameters that can be used in the __call__ method.

Parameters

Instantiate a parser with parser-specific parameters that can be used in the

`__call__` method.

Parameters

Suggested change

Instantiate a parser with parser-specific parameters that can be used in the __call__ method.

Parameters

Instantiate a parser with parser-specific parameters that can be used in the __call__ method.

Parameters

chuckwondo · 2025-06-12T14:21:47Z

virtualizarr/parsers/kerchunk_json.py

+        Instantiate a parser with parser-specific parameters that can be used in the __call__ method.
+        Parameters:
+            group (str): The group within the file to be used as the Zarr root group for the ManifestStore.
+            fs_root (str): The qualifier to be used for kerchunk references containing relative paths.
+            skip_variables (Iterable[str]): Variables in the file that will be ignored when creating the ManifestStore.
+            store_registry (ObjectStoreRegistry): A user defined ObjectStoreRegistry to be used for reading data for kerchunk references contain paths to multiple locations.


Suggested change

Instantiate a parser with parser-specific parameters that can be used in the __call__ method.

Parameters:

group (str): The group within the file to be used as the Zarr root group for the ManifestStore.

fs_root (str): The qualifier to be used for kerchunk references containing relative paths.

skip_variables (Iterable[str]): Variables in the file that will be ignored when creating the ManifestStore.

store_registry (ObjectStoreRegistry): A user defined ObjectStoreRegistry to be used for reading data for kerchunk references contain paths to multiple locations.

Instantiate a parser with parser-specific parameters that can be used in the

`__call__` method.

Parameters

----------

group

The group within the file to be used as the Zarr root group for the ManifestStore.

fs_root

The qualifier to be used for kerchunk references containing relative paths.

skip_variables

Variables in the file that will be ignored when creating the ManifestStore.

store_registry

A user defined ObjectStoreRegistry to be used for reading data for kerchunk

references contain paths to multiple locations.

chuckwondo · 2025-06-12T14:22:05Z

virtualizarr/parsers/kerchunk_json.py

+        file_url
+            The URI or path to the input file (e.g., "s3://bucket/kerchunk.json").
+        object_store
+            An obstore ObjectStore instance for accessing the file specified in the `file_url` parameter.


Suggested change

An obstore ObjectStore instance for accessing the file specified in the `file_url` parameter.

An obstore ObjectStore instance for accessing the file specified in the

`file_url` parameter.

chuckwondo · 2025-06-12T14:22:51Z

virtualizarr/parsers/kerchunk_parquet.py

+        reader_options: dict | None = None,
+    ):
+        """
+        Instantiate a parser with parser-specific parameters that can be used in the __call__ method.


Suggested change

Instantiate a parser with parser-specific parameters that can be used in the __call__ method.

Instantiate a parser with parser-specific parameters that can be used in the

`__call__` method.

chuckwondo · 2025-06-12T14:23:08Z

virtualizarr/parsers/kerchunk_parquet.py

+        file_url
+            The URI or path to the input parquet directory (e.g., "s3://bucket/file.parq").
+        object_store
+            An obstore ObjectStore instance for accessing the file specified in the `file_url` parameter.


Suggested change

An obstore ObjectStore instance for accessing the file specified in the `file_url` parameter.

An obstore ObjectStore instance for accessing the file specified in the

`file_url` parameter.

chuckwondo · 2025-06-12T14:23:48Z

virtualizarr/parsers/kerchunk_parquet.py

+    reader_options
+        dict containing kwargs to pass to file opener, by default {}
+    fs
+        The fsspec filesystem object, created in __post_init__


Suggested change

The fsspec filesystem object, created in __post_init__

The fsspec filesystem object, created in the `__post_init__` method.

chuckwondo · 2025-06-12T14:24:42Z

virtualizarr/parsers/netcdf3.py

+        reader_options: dict | None = None,
+    ):
+        """
+        Instantiate a parser with parser-specific parameters that can be used in the `__call__` method.


Suggested change

Instantiate a parser with parser-specific parameters that can be used in the `__call__` method.

Instantiate a parser with parser-specific parameters that can be used in the

`__call__` method.

chuckwondo · 2025-06-12T14:24:57Z

virtualizarr/parsers/netcdf3.py

+        file_url
+            The URI or path to the input file (e.g., "s3://bucket/file.nc").
+        object_store
+            An obstore ObjectStore instance for accessing the file specified in the `file_url` parameter.


Suggested change

An obstore ObjectStore instance for accessing the file specified in the `file_url` parameter.

An obstore ObjectStore instance for accessing the file specified in the

`file_url` parameter.

chuckwondo · 2025-06-12T14:25:20Z

virtualizarr/parsers/tiff.py

+        remote_options: dict | None = None,
+    ):
+        """
+        Instantiate a parser with parser-specific parameters that can be used in the `__call__` method.


Suggested change

Instantiate a parser with parser-specific parameters that can be used in the `__call__` method.

Instantiate a parser with parser-specific parameters that can be used in the

`__call__` method.

chuckwondo · 2025-06-12T14:25:36Z

virtualizarr/parsers/tiff.py

+        file_url
+            The URI or path to the input file (e.g., "s3://bucket/file.tiff").
+        object_store
+            An obstore ObjectStore instance for accessing the file specified in the `file_url` parameter.


Suggested change

An obstore ObjectStore instance for accessing the file specified in the `file_url` parameter.

An obstore ObjectStore instance for accessing the file specified in the

`file_url` parameter.

chuckwondo · 2025-06-12T14:26:06Z

virtualizarr/parsers/zarr.py

+        file_url
+            The URI or path to the input Zarr store (e.g., "s3://bucket/store.zarr").
+        object_store
+            An obstore ObjectStore instance for accessing the directory specified in the `file_url` parameter.


Suggested change

An obstore ObjectStore instance for accessing the directory specified in the `file_url` parameter.

An obstore ObjectStore instance for accessing the directory specified in the

`file_url` parameter.

TomNicholas and others added 30 commits April 11, 2025 11:34

sketch kerchunk-based constructor method on ManifestStore

5bba81d

sketch how FITS reader would use this constructor

0508fa1

tidy up imports

ecb538c

make new constructor a public method

5fb5c25

refactor kerchunk module to create a ManifestStore instead of using x…

cc7822e

…array

show how this changes the fits reader

dece713

switch all kerchunk readers to use ManifestStore

fda8ce3

fix double import

c7aa584

change test to use ManifestStore

4ca1027

Merge branch 'develop' into refactor-kerchunk-readers-to-use-Manifest…

6e7b2b2

…Store

remove apparently uneeded list_dir function

232f8b2

remove ManifestStore constructor

9526913

improve docstring

38085ab

use function instead of method

53b466c

[pre-commit.ci] auto fixes from pre-commit.com hooks

8e67aa5

for more information, see https://pre-commit.ci

Merge branch 'develop' into Refactor-kerchunk-readers-to-use-Manifest…

16a3510

…Store

Merge branch 'refactor-kerchunk-readers-to-use-ManifestStore' of http…

a19ca31

…s://github.com/TomNicholas/VirtualiZarr into refactor-kerchunk-readers-to-use-ManifestStore

Initial steps for functional backends.

2337329

Use updated backend protocol.

a5e009c

Revert utils ReadableFile change.

5bd8d02

Use ObstoreReader util in hdf backend.

f70240c

Merge branch '547' into refactor/backends

ef8d29b

Intermediate updates for refactored kerchunk backend.

929dcd6

DMRPP reader updated to return ManifestStore.

bd707d5

Update open_virtual_mfdataset to use ManifestStore open_virtual_mfdat…

854e784

…aset.

Update Zarr backend to return ManifestStore.

30c2550

Typing updates for zarr backend obstore compatability.

809ebfb

Update test_xarray with new open_virtual_dataset signature.

d9d2500

Use ManifestStore verification tests for HDF backend.

530778e

Update HDF backend integration tests for open_virtual_dataset signature.

b121941

maxrjones temporarily deployed to test-release June 10, 2025 18:42 — with GitHub Actions Inactive

More context managers

497918e

maxrjones temporarily deployed to test-release June 10, 2025 18:55 — with GitHub Actions Inactive

maxrjones added 2 commits June 10, 2025 14:58

Move imports

9294606

Move import

f2d30d4

maxrjones temporarily deployed to test-release June 10, 2025 19:03 — with GitHub Actions Inactive

chuckwondo added 2 commits June 11, 2025 08:36

Address failing tests in test-py311 env

fba6315

Merge develop and resolve conflicts

922395e

chuckwondo temporarily deployed to test-release June 11, 2025 12:44 — with GitHub Actions Inactive

[pre-commit.ci] auto fixes from pre-commit.com hooks

1e34558

for more information, see https://pre-commit.ci

pre-commit-ci bot temporarily deployed to test-release June 11, 2025 12:45 Inactive

Merge branch 'develop' into refactor/backends

08e6d0b

maxrjones temporarily deployed to test-release June 11, 2025 14:21 — with GitHub Actions Inactive

Merge branch 'develop' into refactor/backends

fb9e036

maxrjones temporarily deployed to test-release June 11, 2025 20:08 — with GitHub Actions Inactive

Fix typing errors and notes

c071650

chuckwondo temporarily deployed to test-release June 11, 2025 23:54 — with GitHub Actions Inactive

Fix numpydoc-validation error

22c2438

chuckwondo deployed to test-release June 12, 2025 00:14 — with GitHub Actions View deployment

chuckwondo reviewed Jun 12, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Refactor codebase to support a new simplified Parser->ManifestStore model. #601

Refactor codebase to support a new simplified Parser->ManifestStore model. #601

sharkinsspatial commented May 20, 2025 •

edited

Loading

Uh oh!

chuckwondo Jun 12, 2025

Uh oh!

chuckwondo Jun 12, 2025

Uh oh!

chuckwondo Jun 12, 2025

Uh oh!

chuckwondo Jun 12, 2025

Uh oh!

chuckwondo Jun 12, 2025

Uh oh!

chuckwondo Jun 12, 2025

Uh oh!

chuckwondo Jun 12, 2025

Uh oh!

chuckwondo Jun 12, 2025

Uh oh!

chuckwondo Jun 12, 2025

Uh oh!

chuckwondo Jun 12, 2025

Uh oh!

chuckwondo Jun 12, 2025

Uh oh!

Uh oh!

		Instantiate a parser with parser-specific parameters that can be used in the __call__ method.
		Parameters

	An obstore ObjectStore instance for accessing the file specified in the `file_url` parameter.
	An obstore ObjectStore instance for accessing the file specified in the
	`file_url` parameter.

	Instantiate a parser with parser-specific parameters that can be used in the __call__ method.
	Instantiate a parser with parser-specific parameters that can be used in the
	`__call__` method.

	The fsspec filesystem object, created in __post_init__
	The fsspec filesystem object, created in the `__post_init__` method.

Refactor codebase to support a new simplified Parser->ManifestStore model. #601

Are you sure you want to change the base?

Refactor codebase to support a new simplified Parser->ManifestStore model. #601

Conversation

sharkinsspatial commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sharkinsspatial commented May 20, 2025 •

edited

Loading