Description
Kedro datasets reach s3:// paths through fsspec/s3fs, and credentials entries already accept client_kwargs.endpoint_url, so Kedro works against Amazon S3 and other S3-compatible object stores today. Two small gaps follow:
-
The S3 client does not identify Kedro. The s3fs/boto3 client Kedro constructs sends only the default agent, so Kedro traffic is not distinguishable to operators or storage providers. Adding a kedro/<version> suffix (the common library/version convention) helps with debugging and support. It is appended to the existing agent, never a replacement, and transmits nothing beyond what s3fs already sends.
-
endpoint_url is documented only inside the MinIO example. docs/catalog-data/data_catalog_yaml_examples.md shows it in one provider-specific section, so a reader configuring any other S3-compatible store does not learn that the same key is the general mechanism.
Context
Would the maintainers be open to either or both? Per the PR template's requirement to open an issue and discuss the approach first, I am asking here before requesting review. I have a draft reference implementation in #5710 so the diff is concrete, and I am happy to close it, reshape it, or drop either half based on your preference.
I searched existing and past issues for user_agent_extra, s3fs endpoint_url, and S3-compatible storage and found nothing proposing this, so this is not a duplicate.
Description
Kedro datasets reach
s3://paths throughfsspec/s3fs, and credentials entries already acceptclient_kwargs.endpoint_url, so Kedro works against Amazon S3 and other S3-compatible object stores today. Two small gaps follow:The S3 client does not identify Kedro. The s3fs/boto3 client Kedro constructs sends only the default agent, so Kedro traffic is not distinguishable to operators or storage providers. Adding a
kedro/<version>suffix (the commonlibrary/versionconvention) helps with debugging and support. It is appended to the existing agent, never a replacement, and transmits nothing beyond what s3fs already sends.endpoint_urlis documented only inside the MinIO example.docs/catalog-data/data_catalog_yaml_examples.mdshows it in one provider-specific section, so a reader configuring any other S3-compatible store does not learn that the same key is the general mechanism.Context
Would the maintainers be open to either or both? Per the PR template's requirement to open an issue and discuss the approach first, I am asking here before requesting review. I have a draft reference implementation in #5710 so the diff is concrete, and I am happy to close it, reshape it, or drop either half based on your preference.
I searched existing and past issues for
user_agent_extra, s3fsendpoint_url, and S3-compatible storage and found nothing proposing this, so this is not a duplicate.