Skip to content

Identify Kedro in the S3 client user agent, and document endpoint_url generally #5711

Description

@goanpeca

Description

Kedro datasets reach s3:// paths through fsspec/s3fs, and credentials entries already accept client_kwargs.endpoint_url, so Kedro works against Amazon S3 and other S3-compatible object stores today. Two small gaps follow:

  1. The S3 client does not identify Kedro. The s3fs/boto3 client Kedro constructs sends only the default agent, so Kedro traffic is not distinguishable to operators or storage providers. Adding a kedro/<version> suffix (the common library/version convention) helps with debugging and support. It is appended to the existing agent, never a replacement, and transmits nothing beyond what s3fs already sends.

  2. endpoint_url is documented only inside the MinIO example. docs/catalog-data/data_catalog_yaml_examples.md shows it in one provider-specific section, so a reader configuring any other S3-compatible store does not learn that the same key is the general mechanism.

Context

Would the maintainers be open to either or both? Per the PR template's requirement to open an issue and discuss the approach first, I am asking here before requesting review. I have a draft reference implementation in #5710 so the diff is concrete, and I am happy to close it, reshape it, or drop either half based on your preference.

I searched existing and past issues for user_agent_extra, s3fs endpoint_url, and S3-compatible storage and found nothing proposing this, so this is not a duplicate.

Metadata

Metadata

Assignees

No one assigned

    Labels

    CommunityIssue/PR opened by the open-source community

    Type

    No type

    Projects

    Status
    Wizard inbox

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions