Merge main

Author: paraseba

docs/docs/icechunk-python/configuration.md

@@ -69,11 +69,11 @@ config.storage = icechunk.StorageSettings(
 
 ### [`virtual_chunk_containers`](./reference.md#icechunk.RepositoryConfig.virtual_chunk_containers)
 
-Icechunk allows repos to contain [virtual chunks](./virtual.md). To allow for referencing these virtual chunks, you can configure the `virtual_chunk_containers` parameter to specify the storage locations and configurations for any virtual chunks. Each virtual chunk container is specified by a [`VirtualChunkContainer`](./reference.md#icechunk.VirtualChunkContainer) object which contains a name, a url prefix, and a storage configuration. When a container is added to the settings, any virtual chunks with a url that starts with the configured prefix will use the storage configuration for that matching container.
+Icechunk allows repos to contain [virtual chunks](./virtual.md). To allow for referencing these virtual chunks, you must configure the `virtual_chunk_containers` parameter to specify the storage locations and configurations for any virtual chunks. Each virtual chunk container is specified by a [`VirtualChunkContainer`](./reference.md#icechunk.VirtualChunkContainer) object which contains a url prefix, and a storage configuration. When a container is added to the settings, any virtual chunks with a url that starts with the configured prefix will use the storage configuration for that matching container.
 
 !!! note
 
-    Currently only `s3` compatible storage and `local_filesystem` storage are supported for virtual chunk containers. Other storage backends such as `gcs`, `azure`, and `https` are on the roadmap.
+    Currently only `s3` compatible storage, `gcs`, `local_filesystem` and `http[s]` storages are supported for virtual chunk containers. Other storage backends such as `azure` are on the roadmap.
 
 #### Example
 
@@ -82,7 +82,6 @@ For example, if we wanted to configure an icechunk repo to be able to contain vi
 ```python
 config.virtual_chunk_containers = [
     icechunk.VirtualChunkContainer(
-        name="my-s3-bucket",
         url_prefix="s3://my-s3-bucket/",
         storage=icechunk.StorageSettings(
             storage=icechunk.s3_storage(bucket="my-s3-bucket", region="us-east-1"),
@@ -96,7 +95,6 @@ If we also wanted to configure the repo to be able to contain virtual chunks fro
 ```python
 config.set_virtual_chunk_container(
     icechunk.VirtualChunkContainer(
-        name="my-other-s3-bucket",
         url_prefix="s3://my-other-s3-bucket/",
         storage=icechunk.StorageSettings(
             storage=icechunk.s3_storage(bucket="my-other-s3-bucket", region="us-west-2"),
@@ -105,11 +103,11 @@ config.set_virtual_chunk_container(
 )
 ```
 
-Now at read time, if icechunk encounters a virtual chunk url that starts with `s3://my-other-s3-bucket/`, it will use the storage configuration for the `my-other-s3-bucket` container.
+Now at read time, if Icechunk encounters a virtual chunk url that starts with `s3://my-other-s3-bucket/`, it will use the storage configuration for the `my-other-s3-bucket` container.
 
 !!! note
 
-    While virtual chunk containers specify the storage configuration for any virtual chunks, they do not contain any authentication information. The credentials must also be specified when opening the repository using the [`virtual_chunk_credentials`](./reference.md#icechunk.Repository.open) parameter. See the [Virtual Chunk Credentials](#virtual-chunk-credentials) section for more information.
+    While virtual chunk containers specify the storage configuration for any virtual chunks, they do not contain any authentication information. The credentials must also be specified when opening the repository using the [`authorize_virtual_chunk_access`](./reference.md#icechunk.Repository.open) parameter. This parameter also serves as a way for the user to authorize the access to the virtual chunk containers, containers that are not explicitly allowed with `authorize_virtual_chunk_access` won't be able to fetch their chunks. See the [Virtual Chunk Credentials](#virtual-chunk-credentials) section for more information.
 
 ### [`manifest`](./reference.md#icechunk.RepositoryConfig.manifest)
 
@@ -269,21 +267,22 @@ The next time this repo is opened, the persisted config will be loaded by defaul
 
 ## Virtual Chunk Credentials
 
-When using virtual chunk containers, the credentials for the storage backend must also be specified. This is done using the [`virtual_chunk_credentials`](./reference.md#icechunk.Repository.open) parameter when creating or opening the repo. Credentials are specified as a dictionary of container names mapping to credential objects. A helper function, [`containers_credentials`](./reference.md#icechunk.containers_credentials), is provided to make it easier to specify credentials for multiple containers.
+When using virtual chunk containers, the containers must be authorized by the repo user, and the credentials for the storage backend must be specified. This is done using the [`authorize_virtual_chunk_access`](./reference.md#icechunk.Repository.open) parameter when creating or opening the repo. Credentials are specified as a dictionary of container url prefixes mapping to credential objects or `None`. A `None` credential will fetch credentials from the process environment or it will use anonymous credentials if the container allows it. A helper function, [`containers_credentials`](./reference.md#icechunk.containers_credentials), is provided to make it easier to specify credentials for multiple containers.
 
 ### Example
 
 Expanding on the example from the [Virtual Chunk Containers](#virtual-chunk-containers) section, we can configure the repo to use the credentials for the `my-s3-bucket` and `my-other-s3-bucket` containers.
 
 ```python
 credentials = icechunk.containers_credentials(
-    my_s3_bucket=icechunk.s3_credentials(bucket="my-s3-bucket", region="us-east-1"),
-    my_other_s3_bucket=icechunk.s3_credentials(bucket="my-other-s3-bucket", region="us-west-2"),
+    { "s3://my_s3_bucket": icechunk.s3_credentials(bucket="my-s3-bucket", region="us-east-1"),
+      "s3://my_other_s3_bucket": icechunk.s3_credentials(bucket="my-other-s3-bucket", region="us-west-2"),
+    }
 )
 
 repo = icechunk.Repository.open(
     storage=storage,
     config=config,
-    virtual_chunk_credentials=credentials,
+    authorize_virtual_chunk_access=credentials,
 )
 ```

docs/docs/icechunk-python/virtual.md

@@ -92,12 +92,12 @@ storage = icechunk.local_filesystem_storage(
 )
 
 config = icechunk.RepositoryConfig.default()
-config.set_virtual_chunk_container(icechunk.VirtualChunkContainer("s3", "s3://", icechunk.s3_store(region="us-east-1")))
-credentials = icechunk.containers_credentials(s3=icechunk.s3_credentials(anonymous=True))
+config.set_virtual_chunk_container(icechunk.VirtualChunkContainer("s3://mybucket/my/data/", icechunk.s3_store(region="us-east-1")))
+credentials = icechunk.containers_credentials({"s3://mybucket/my/data/": icechunk.s3_credentials(anonymous=True)})
 repo = icechunk.Repository.create(storage, config, credentials)
 ```
 
-With the repo created, lets write our virtual dataset to Icechunk with VirtualiZarr!
+With the repo created, and the virtual chunk container added, lets write our virtual dataset to Icechunk with VirtualiZarr!
 
 ```python
 session = repo.writable_session("main")
@@ -150,13 +150,17 @@ ds.sst.isel(time=26, zlev=0).plot(x='lon', y='lat', vmin=0)
 
 ![oisst](../assets/datasets/oisst.png)
 
+!!! note
+
+    Users of the repo will need to enable the virtual chunk container by passing the `credentials` argument to `Repository.open`. This way, the repo user, flags the container as authorized. `credentials` argument must be a dict using url prefixes as keys and optional credentials as values. If the container requires no credentials, `None` can be used as the value in the map. Failing to authorize a container, will generate an error when a chunk is fetched from it.
+
 ## Virtual Reference API
 
 While `VirtualiZarr` is the easiest way to create virtual datasets with Icechunk, the Store API that it uses to create the datasets in Icechunk is public. `IcechunkStore` contains a [`set_virtual_ref`](./reference.md#icechunk.IcechunkStore.set_virtual_ref) method that specifies a virtual ref for a specified chunk.
 
 ### Virtual Reference Storage Support
 
-Currently, Icechunk supports two types of storage for virtual references:
+Currently, Icechunk supports four types of storage for virtual references:
 
 #### S3 Compatible
 
@@ -167,13 +171,48 @@ References to files accessible via S3 compatible storage.
 Here is how we can set the chunk at key `c/0` to point to a file on an s3 bucket,`mybucket`, with the prefix `my/data/file.nc`:
 
 ```python
-store.set_virtual_ref('c/0', 's3://mybucket/my/data/file.nc', offset=1000, length=200)
+config = icechunk.RepositoryConfig.default()
+config.set_virtual_chunk_container(icechunk.VirtualChunkContainer("s3://mybucket/my/data", icechunk.s3_store(region="us-east-1")))
+repo = icechunk.Repository.create(storage, config)
+session = repo.writable_session("main")
+session.store.set_virtual_ref('c/0', 's3://mybucket/my/data/file.nc', offset=1000, length=200)
 ```
 
 ##### Configuration
 
 S3 virtual references require configuring credential for the store to be able to access the specified s3 bucket. See [the configuration docs](./configuration.md#virtual-reference-storage-config) for instructions.
 
+#### GCS
+
+References to files accessible on Google Cloud Storage
+
+##### Example
+
+Here is how we can set the chunk at key `c/0` to point to a file on an s3 bucket,`mybucket`, with the prefix `my/data/file.nc`:
+
+```python
+config = icechunk.RepositoryConfig.default()
+config.set_virtual_chunk_container(icechunk.VirtualChunkContainer("gcs://mybucket/my/data", icechunk.gcs_store(options={})))
+repo = icechunk.Repository.create(storage, config)
+session = repo.writable_session("main")
+session.store.set_virtual_ref('c/0', 'gcs://mybucket/my/data/file.nc', offset=1000, length=200)
+```
+
+#### HTTP
+
+References to files accessible via http(s) protocol
+
+##### Example
+
+Here is how we can set the chunk at key `c/0` to point to a file on `myserver`, with the prefix `my/data/file.nc`:
+
+```python
+config = icechunk.RepositoryConfig.default()
+config.set_virtual_chunk_container(icechunk.VirtualChunkContainer("https://myserver/my/data", icechunk.http_store(options={})))
+repo = icechunk.Repository.create(storage, config)
+session = repo.writable_session("main")
+session.store.set_virtual_ref('c/0', 'https://myserver/my/data/file.nc', offset=1000, length=200)
+```
 
 #### Local Filesystem
 
@@ -184,7 +223,11 @@ References to files accessible via local filesystem. This requires any file path
 Here is how we can set the chunk at key `c/0` to point to a file on my local filesystem located at `/path/to/my/file.nc`:
 
 ```python
-store.set_virtual_ref('c/0', 'file:///path/to/my/file.nc', offset=20, length=100)
+config = icechunk.RepositoryConfig.default()
+config.set_virtual_chunk_container(icechunk.VirtualChunkContainer("s3://mybucket/my/data", icechunk.local_filesystem_store("/path/to/my")))
+repo = icechunk.Repository.create(storage, config)
+session = repo.writable_session("main")
+session.store.set_virtual_ref('c/0', 'file:///path/to/my/file.nc', offset=20, length=100)
 ```
 
 No extra configuration is necessary for local filesystem references.

icechunk-python/python/icechunk/__init__.py

@@ -78,6 +78,7 @@
     http_store,
     in_memory_storage,
     local_filesystem_storage,
+    local_filesystem_store,
     r2_storage,
     s3_storage,
     s3_store,
@@ -151,6 +152,7 @@
     "in_memory_storage",
     "initialize_logs",
     "local_filesystem_storage",
+    "local_filesystem_store",
     "print_debug_info",
     "r2_storage",
     "s3_anonymous_credentials",

icechunk-python/python/icechunk/_icechunk_python.pyi

@@ -1237,23 +1237,23 @@ class PyRepository:
         storage: Storage,
         *,
         config: RepositoryConfig | None = None,
-        virtual_chunk_credentials: dict[str, AnyCredential] | None = None,
+        authorize_virtual_chunk_access: dict[str, AnyCredential | None] | None = None,
     ) -> PyRepository: ...
     @classmethod
     def open(
         cls,
         storage: Storage,
         *,
         config: RepositoryConfig | None = None,
-        virtual_chunk_credentials: dict[str, AnyCredential] | None = None,
+        authorize_virtual_chunk_access: dict[str, AnyCredential | None] | None = None,
     ) -> PyRepository: ...
     @classmethod
     def open_or_create(
         cls,
         storage: Storage,
         *,
         config: RepositoryConfig | None = None,
-        virtual_chunk_credentials: dict[str, AnyCredential] | None = None,
+        authorize_virtual_chunk_access: dict[str, AnyCredential | None] | None = None,
     ) -> PyRepository: ...
     @staticmethod
     def exists(storage: Storage) -> bool: ...

icechunk-python/python/icechunk/credentials.py

@@ -1,6 +1,7 @@
 import pickle
 from collections.abc import Callable, Mapping
 from datetime import datetime
+from typing import cast
 
 from icechunk._icechunk_python import (
     AzureCredentials,
@@ -341,14 +342,14 @@ def azure_credentials(
 
 
 def containers_credentials(
-    m: Mapping[str, AnyS3Credential] = {}, **kwargs: AnyS3Credential
-) -> dict[str, Credentials.S3]:
+    m: Mapping[str, AnyS3Credential | AnyGcsCredential | AnyAzureCredential | None],
+) -> dict[str, AnyCredential | None]:
     """Build a map of credentials for virtual chunk containers.
 
     Parameters
     ----------
-    m: Mapping[str, AnyS3Credential]
-        A mapping from container name to credentials.
+    m: Mapping[str, AnyS3Credential | AnyGcsCredential | AnyAzureCredential ]
+        A mapping from container url prefixes to credentials.
 
     Examples
     --------
@@ -365,24 +366,36 @@ def containers_credentials(
         s3_compatible=True,
         force_path_style=True,
     )
-    container = ic.VirtualChunkContainer("s3", "s3://", virtual_store_config)
+    container = ic.VirtualChunkContainer("s3://somebucket", virtual_store_config)
     config.set_virtual_chunk_container(container)
     credentials = ic.containers_credentials(
-        s3=ic.s3_credentials(access_key_id="ACCESS_KEY", secret_access_key="SECRET")
+        {"s3://somebucket": ic.s3_credentials(access_key_id="ACCESS_KEY", secret_access_key="SECRET"}
     )
 
     repo = ic.Repository.create(
         storage=ic.local_filesystem_storage(store_path),
         config=config,
-        virtual_chunk_credentials=credentials,
+        authorize_virtual_chunk_access=credentials,
     )
     ```
 
     """
-    res = {}
-    for name, cred in {**m, **kwargs}.items():
-        if isinstance(cred, AnyS3Credential):
+    res: dict[str, AnyCredential | None] = {}
+    for name, cred in m.items():
+        if cred is None:
+            res[name] = None
+        elif isinstance(cred, AnyS3Credential):
             res[name] = Credentials.S3(cred)
+        elif (
+            isinstance(cred, GcsCredentials.FromEnv)
+            or isinstance(cred, GcsCredentials.Static)
+            or isinstance(cred, GcsCredentials.Refreshable)
+        ):
+            res[name] = Credentials.Gcs(cast(GcsCredentials, cred))
+        elif isinstance(cred, AzureCredentials.FromEnv) or isinstance(
+            cred, AzureCredentials.Static
+        ):
+            res[name] = Credentials.Azure(cast(AzureCredentials, cred))
         else:
             raise ValueError(f"Unknown credential type {type(cred)}")
     return res

icechunk-python/python/icechunk/repository.py

@@ -27,7 +27,7 @@ def create(
         cls,
         storage: Storage,
         config: RepositoryConfig | None = None,
-        virtual_chunk_credentials: dict[str, AnyCredential] | None = None,
+        authorize_virtual_chunk_access: dict[str, AnyCredential | None] | None = None,
     ) -> Self:
         """
         Create a new Icechunk repository.
@@ -43,8 +43,13 @@ def create(
             The storage configuration for the repository.
         config : RepositoryConfig, optional
             The repository configuration. If not provided, a default configuration will be used.
-        virtual_chunk_credentials : dict[str, AnyCredential], optional
-            Credentials for virtual chunks.
+        authorize_virtual_chunk_access : dict[str, AnyCredential | None], optional
+            Authorize Icechunk to access virtual chunks in these containers. A mapping
+            from container url_prefix to the credentials to use to access chunks in
+            that container. If credential is `None`, they will be fetched from the
+            environment, or anonymous credentials will be used if the container allows it.
+            As a security measure, Icechunk will block access to virtual chunks if the
+            container is not authorized using this argument.
 
         Returns
         -------
@@ -55,7 +60,7 @@ def create(
             PyRepository.create(
                 storage,
                 config=config,
-                virtual_chunk_credentials=virtual_chunk_credentials,
+                authorize_virtual_chunk_access=authorize_virtual_chunk_access,
             )
         )
 
@@ -64,7 +69,7 @@ def open(
         cls,
         storage: Storage,
         config: RepositoryConfig | None = None,
-        virtual_chunk_credentials: dict[str, AnyCredential] | None = None,
+        authorize_virtual_chunk_access: dict[str, AnyCredential | None] | None = None,
     ) -> Self:
         """
         Open an existing Icechunk repository.
@@ -82,8 +87,13 @@ def open(
         config : RepositoryConfig, optional
             The repository settings. If not provided, a default configuration will be
             loaded from the repository.
-        virtual_chunk_credentials : dict[str, AnyCredential], optional
-            Credentials for virtual chunks.
+        authorize_virtual_chunk_access : dict[str, AnyCredential | None], optional
+            Authorize Icechunk to access virtual chunks in these containers. A mapping
+            from container url_prefix to the credentials to use to access chunks in
+            that container. If credential is `None`, they will be fetched from the
+            environment, or anonymous credentials will be used if the container allows it.
+            As a security measure, Icechunk will block access to virtual chunks if the
+            container is not authorized using this argument.
 
         Returns
         -------
@@ -94,7 +104,7 @@ def open(
             PyRepository.open(
                 storage,
                 config=config,
-                virtual_chunk_credentials=virtual_chunk_credentials,
+                authorize_virtual_chunk_access=authorize_virtual_chunk_access,
             )
         )
 
@@ -103,7 +113,7 @@ def open_or_create(
         cls,
         storage: Storage,
         config: RepositoryConfig | None = None,
-        virtual_chunk_credentials: dict[str, AnyCredential] | None = None,
+        authorize_virtual_chunk_access: dict[str, AnyCredential | None] | None = None,
     ) -> Self:
         """
         Open an existing Icechunk repository or create a new one if it does not exist.
@@ -122,8 +132,13 @@ def open_or_create(
         config : RepositoryConfig, optional
             The repository settings. If not provided, a default configuration will be
             loaded from the repository.
-        virtual_chunk_credentials : dict[str, AnyCredential], optional
-            Credentials for virtual chunks.
+        authorize_virtual_chunk_access : dict[str, AnyCredential | None], optional
+            Authorize Icechunk to access virtual chunks in these containers. A mapping
+            from container url_prefix to the credentials to use to access chunks in
+            that container. If credential is `None`, they will be fetched from the
+            environment, or anonymous credentials will be used if the container allows it.
+            As a security measure, Icechunk will block access to virtual chunks if the
+            container is not authorized using this argument.
 
         Returns
         -------
@@ -134,7 +149,7 @@ def open_or_create(
             PyRepository.open_or_create(
                 storage,
                 config=config,
-                virtual_chunk_credentials=virtual_chunk_credentials,
+                authorize_virtual_chunk_access=authorize_virtual_chunk_access,
             )
         )