Skip to content

config homogenization #323

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Mar 6, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions docs/api/store/aws.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
# AWS S3

::: obstore.store.S3Store
::: obstore.store.S3ConfigInput
options:
show_if_no_docstring: true
::: obstore.store.S3Config
options:
show_if_no_docstring: true
Expand Down
3 changes: 0 additions & 3 deletions docs/api/store/azure.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
# Microsoft Azure

::: obstore.store.AzureStore
::: obstore.store.AzureConfigInput
options:
show_if_no_docstring: true
::: obstore.store.AzureConfig
options:
show_if_no_docstring: true
Expand Down
3 changes: 0 additions & 3 deletions docs/api/store/gcs.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
# Google Cloud Storage

::: obstore.store.GCSStore
::: obstore.store.GCSConfigInput
options:
show_if_no_docstring: true
::: obstore.store.GCSConfig
options:
show_if_no_docstring: true
Expand Down
Binary file removed docs/assets/fsspec-type-hinting.jpg
Binary file not shown.
Binary file added docs/assets/fsspec-type-hinting.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions docs/authentication.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Note that many authentication variants are already supported natively.

- Basic authentication, where an access key ID, secret access key, and optionally token are passed in via environment variables or configuration parameters.
- [WebIdentity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html). This requires the `AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_ARN` environment variables to be set. Additionally, `AWS_ROLE_SESSION_NAME` can be set to specify a session name.
- [Container credentials](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html). Ensure you pass [`aws_container_credentials_relative_uri`][obstore.store.S3ConfigInput.aws_container_credentials_relative_uri] to the `S3Store`.
- [Container credentials](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html). Ensure you pass [`container_credentials_relative_uri`][obstore.store.S3Config.container_credentials_relative_uri] to the `S3Store`.
- [Instance credentials](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html).

(A transcription of [this underlying code](https://github.com/apache/arrow-rs/blob/a00f9f43a0530b9255e4f9940e43121deedb0cc7/object_store/src/aws/builder.rs#L900-L970)).
Expand All @@ -44,7 +44,7 @@ Note that many authentication variants are already supported natively.
- Workload identity OAuth2, using a `client_id`, `tenant_id`, and `federated_token_file` passed in by the user
- OAuth2, using a `client_id`, `client_secret`, and `tenant_id` passed in by the user
- A SAS key passed in by the user.
- Azure CLI. (If you want to ensure the IMDS authentication is used below, pass [`use_azure_cli=False`][obstore.store.AzureConfigInput.use_azure_cli] to `AzureStore`.)
- Azure CLI. (If you want to ensure the IMDS authentication is used below, pass [`use_azure_cli=False`][obstore.store.AzureConfig.use_azure_cli] to `AzureStore`.)
- IMDS Managed Identity Provider.

(A transcription of [this underlying code](https://github.com/apache/arrow-rs/blob/a00f9f43a0530b9255e4f9940e43121deedb0cc7/object_store/src/azure/builder.rs#L942-L1019)).
Expand Down
10 changes: 5 additions & 5 deletions docs/fsspec.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,18 +114,18 @@ import fsspec
from obstore.fsspec import register

if TYPE_CHECKING:
from obstore.store import S3ConfigInput
from obstore.store import S3Config

register("s3")

config: S3ConfigInput = {"region": "us-west-2", "skip_signature": True}
config: S3Config = {"region": "us-west-2", "skip_signature": True}
fs = fsspec.filesystem("s3", config=config)
```

Then your type checker will validate that the `config` dictionary is compatible with [`S3ConfigInput`][obstore.store.S3ConfigInput]. VSCode also provides auto suggestions for parameters:
Then your type checker will validate that the `config` dictionary is compatible with [`S3Config`][obstore.store.S3Config]. VSCode also provides auto suggestions for parameters:

![](./assets/fsspec-type-hinting.jpg)
![](./assets/fsspec-type-hinting.png)

!!! note

`S3ConfigInput` is a "type-only" construct, and so it needs to be imported from within an `if TYPE_CHECKING` block. Additionally, `from __future__ import annotations` must be at the top of the file.
`S3Config` is a "type-only" construct, and so it needs to be imported from within an `if TYPE_CHECKING` block. Additionally, `from __future__ import annotations` must be at the top of the file.
6 changes: 3 additions & 3 deletions obstore/python/obstore/auth/boto3.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

import botocore.session

from obstore.store import S3ConfigInput, S3Credential
from obstore.store import S3Config, S3Credential


class PolicyDescriptorTypeTypeDef(TypedDict): # noqa: D101
Expand Down Expand Up @@ -56,7 +56,7 @@ class Boto3CredentialProvider:
""" # noqa: E501

credentials: botocore.credentials.Credentials
config: S3ConfigInput
config: S3Config
ttl: timedelta

def __init__(
Expand Down Expand Up @@ -114,7 +114,7 @@ class StsCredentialProvider:
store.
""" # noqa: E501

config: S3ConfigInput
config: S3Config
session: boto3.session.Session

def __init__(
Expand Down
6 changes: 3 additions & 3 deletions obstore/python/obstore/auth/earthdata.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
from typing import TYPE_CHECKING

if TYPE_CHECKING:
from obstore.store import S3ConfigInput, S3Credential
from obstore.store import S3Config, S3Credential

CREDENTIALS_API = "https://archive.podaac.earthdata.nasa.gov/s3credentials"

Expand Down Expand Up @@ -42,7 +42,7 @@ class NasaEarthdataCredentialProvider:
[NASA Earthdata]: https://www.earthdata.nasa.gov/
""" # noqa: E501

config: S3ConfigInput
config: S3Config

def __init__(
self,
Expand Down Expand Up @@ -118,7 +118,7 @@ class NasaEarthdataAsyncCredentialProvider:
[NASA Earthdata]: https://www.earthdata.nasa.gov/
""" # noqa: E501

config: S3ConfigInput
config: S3Config

def __init__(
self,
Expand Down
27 changes: 8 additions & 19 deletions obstore/python/obstore/fsspec.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,14 +53,11 @@
from obstore import Attributes, Bytes, ReadableFile, WritableFile
from obstore.store import (
AzureConfig,
AzureConfigInput,
ClientConfig,
GCSConfig,
GCSConfigInput,
ObjectStore,
RetryConfig,
S3Config,
S3ConfigInput,
)

__all__ = [
Expand Down Expand Up @@ -119,56 +116,48 @@ def __init__(
self,
protocol: Literal["s3", "s3a"],
*args: Any,
config: S3Config | S3ConfigInput | None = None,
config: S3Config | None = None,
client_options: ClientConfig | None = None,
retry_config: RetryConfig | None = None,
asynchronous: bool = False,
max_cache_size: int = 10,
loop: Any = None,
batch_size: int | None = None,
**kwargs: Unpack[S3ConfigInput],
**kwargs: Unpack[S3Config],
) -> None: ...
@overload
def __init__(
self,
protocol: Literal["gs"],
*args: Any,
config: GCSConfig | GCSConfigInput | None = None,
config: GCSConfig | None = None,
client_options: ClientConfig | None = None,
retry_config: RetryConfig | None = None,
asynchronous: bool = False,
max_cache_size: int = 10,
loop: Any = None,
batch_size: int | None = None,
**kwargs: Unpack[GCSConfigInput],
**kwargs: Unpack[GCSConfig],
) -> None: ...
@overload
def __init__(
self,
protocol: Literal["az", "adl", "azure", "abfs", "abfss"],
*args: Any,
config: AzureConfig | AzureConfigInput | None = None,
config: AzureConfig | None = None,
client_options: ClientConfig | None = None,
retry_config: RetryConfig | None = None,
asynchronous: bool = False,
max_cache_size: int = 10,
loop: Any = None,
batch_size: int | None = None,
**kwargs: Unpack[AzureConfigInput],
**kwargs: Unpack[AzureConfig],
) -> None: ...
def __init__( # noqa: PLR0913
self,
protocol: SUPPORTED_PROTOCOLS_T | str | None = None,
*args: Any,
config: (
S3Config
| S3ConfigInput
| GCSConfig
| GCSConfigInput
| AzureConfig
| AzureConfigInput
| None
) = None,
config: (S3Config | GCSConfig | AzureConfig | None) = None,
client_options: ClientConfig | None = None,
retry_config: RetryConfig | None = None,
asynchronous: bool = False,
Expand All @@ -184,7 +173,7 @@ def __init__( # noqa: PLR0913
"gcs", or "abfs". If `None`, the default class-level protocol
is used. Default to None.
config: Configuration for the cloud storage provider, which can be one of
S3Config, S3ConfigInput, GCSConfig, GCSConfigInput, AzureConfig,
S3Config, GCSConfig, AzureConfig,
or AzureConfigInput. Any of these values will be applied after checking
for environment variables. If `None`, no cloud storage configuration is
applied beyond what is found in environment variables.
Expand Down
17 changes: 7 additions & 10 deletions obstore/python/obstore/store/__init__.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,18 @@ from pathlib import Path
from typing import Any, TypeAlias, Unpack, overload

from ._aws import S3Config as S3Config
from ._aws import S3ConfigInput as S3ConfigInput
from ._aws import S3Credential as S3Credential
from ._aws import S3CredentialProvider as S3CredentialProvider
from ._aws import S3Store as S3Store
from ._azure import AzureAccessKey as AzureAccessKey
from ._azure import AzureBearerToken as AzureBearerToken
from ._azure import AzureConfig as AzureConfig
from ._azure import AzureConfigInput as AzureConfigInput
from ._azure import AzureCredential as AzureCredential
from ._azure import AzureCredentialProvider as AzureCredentialProvider
from ._azure import AzureSASToken as AzureSASToken
from ._azure import AzureStore as AzureStore
from ._client import ClientConfig as ClientConfig
from ._gcs import GCSConfig as GCSConfig
from ._gcs import GCSConfigInput as GCSConfigInput
from ._gcs import GCSCredential as GCSCredential
from ._gcs import GCSCredentialProvider as GCSCredentialProvider
from ._gcs import GCSStore as GCSStore
Expand All @@ -30,31 +27,31 @@ from ._retry import RetryConfig as RetryConfig
def from_url(
url: str,
*,
config: S3Config | S3ConfigInput | None = None,
config: S3Config | None = None,
client_options: ClientConfig | None = None,
retry_config: RetryConfig | None = None,
credential_provider: S3CredentialProvider | None = None,
**kwargs: Unpack[S3ConfigInput],
**kwargs: Unpack[S3Config],
) -> ObjectStore: ...
@overload
def from_url(
url: str,
*,
config: GCSConfig | GCSConfigInput | None = None,
config: GCSConfig | None = None,
client_options: ClientConfig | None = None,
retry_config: RetryConfig | None = None,
credential_provider: GCSCredentialProvider | None = None,
**kwargs: Unpack[GCSConfigInput],
**kwargs: Unpack[GCSConfig],
) -> ObjectStore: ...
@overload
def from_url(
url: str,
*,
config: AzureConfig | AzureConfigInput | None = None,
config: AzureConfig | None = None,
client_options: ClientConfig | None = None,
retry_config: RetryConfig | None = None,
credential_provider: AzureCredentialProvider | None = None,
**kwargs: Unpack[AzureConfigInput],
**kwargs: Unpack[AzureConfig],
) -> ObjectStore: ...
@overload
def from_url(
Expand All @@ -69,7 +66,7 @@ def from_url(
def from_url(
url: str,
*,
config: S3ConfigInput | GCSConfigInput | AzureConfigInput | None = None,
config: S3Config | GCSConfig | AzureConfig | None = None,
client_options: ClientConfig | None = None,
retry_config: RetryConfig | None = None,
credential_provider: Callable | None = None,
Expand Down
Loading
Loading