[FEATURE] Add a great-expectations user-agent suffix to S3 clients

[goanpeca]

Why

Great Expectations already lets a datasource pass boto3_options (including endpoint_url and a custom botocore.config.Config) to the underlying boto3 S3 client, so it works with Amazon S3 and any S3-compatible object store. Two gaps today: the S3 client goes out with only the default boto3/botocore user agent, so operators and providers cannot see which requests originate from Great Expectations; and the docs document endpoint_url but do not make clear that setting it is how you connect to a non-AWS S3-compatible store.

This change adds a great-expectations/<version> suffix to the S3 client user agent and clarifies that docs option. It follows the common library/version convention that libraries use to attribute their S3 traffic, and helps with usage attribution, support, and debugging on Amazon S3 as well as on any S3-compatible object store (for example Backblaze B2, Cloudflare R2, or MinIO). The change is intentionally small and additive: it does not alter datasource configuration, credentials handling, or request behavior.

What

• Added get_s3_boto3_options(boto3_options) to great_expectations/compatibility/aws.py. It returns the caller's options with a user_agent_extra of great-expectations/<distribution version>. If the caller already supplied a Config with a user_agent_extra, the suffix is appended rather than replacing it (via Config.merge), so a user's own agent string and any endpoint_url are preserved.
• Falls back to a "dev" version when the installed distribution metadata is unavailable, and is a no-op that returns the options unchanged when botocore's Config is not importable, so environments without the AWS extras are unaffected.
• Routed the three existing S3 client construction sites through the helper: PandasS3Datasource, SparkS3Datasource, and PandasExecutionEngine._instantiate_s3_client. No call signatures or public APIs change.
• Docs: clarified the endpoint_url option on the S3 Data Source page (docs/docusaurus/docs/core/connect_to_data/filesystem_data/_create_a_data_source/_s3/_s3.md) to note that it connects Great Expectations to an S3-compatible object store (Amazon S3, or a compatible provider such as Backblaze B2, Cloudflare R2, or MinIO), and to leave it unset for Amazon S3.

Scope

This is a minimal, additive change: the AWS compatibility module, the three call sites that build a boto3 S3 client, and a one-line docs clarification. There is no change to configuration schema, type stubs, credentials handling, or the requests that Great Expectations issues. The framing is deliberately generic: Amazon S3 is named first and S3-compatible providers appear only as alphabetical examples; the user-agent suffix is appended to, never a replacement for, a user-provided agent string.

Test plan

Locally, against the unit marker:

• get_s3_boto3_options({}) returns options whose config.user_agent_extra equals great-expectations/<version>.
• When a Config(user_agent_extra="my-app/1.0") is passed in, the result is my-app/1.0 great-expectations/<version> (append, not overwrite), and an accompanying endpoint_url in the options is preserved.
• The datasource/engine call sites still build a client when boto3 and botocore are available, matching the existing mock_boto3_options unit fixtures in tests/datasource/fluent/test_pandas_s3_datasource.py.

I have not committed a new test in this branch yet. If the reviewers would like the behavior locked in, I am happy to add a small @pytest.mark.unit test for get_s3_boto3_options in tests/ (append vs set, and endpoint_url preservation) before merge.

---

• Description of PR changes above includes a link to an existing GitHub issue - no existing issue; this is a small, additive attribution and docs change. Happy to open a tracking issue or discuss in Slack first if the team prefers.
• PR title is prefixed with one of: [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], [CONTRIB], [MINORBUMP]
• Code is linted - run invoke lint (uses ruff format + ruff check)
• Appropriate tests and docs have been updated - docs updated (the S3 Data Source endpoint_url clarification); no unit test committed yet, see the Test plan above. I can add a @pytest.mark.unit test before merge if desired.

[netlify]

👷 Deploy request for niobium-lead-7998 pending review.

Visit the deploys page to approve it

Name	Link
🔨 Latest commit	6ead077

[Copilot]

Pull request overview

This PR adds a Great Expectations–specific great-expectations/<version> suffix to the boto3 S3 client user agent to make S3 traffic attributable (and easier to support/debug) across AWS S3 and S3-compatible object stores, while preserving any caller-provided boto3 options and agent strings.

Changes:

• Added get_s3_boto3_options(...) in great_expectations/compatibility/aws.py to inject/append user_agent_extra using botocore Config.
• Routed S3 client construction in the Pandas and Spark S3 datasources through the new helper.
• Routed PandasExecutionEngine S3 client instantiation through the new helper.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
great_expectations/compatibility/aws.py	Adds helper that appends great-expectations/<version> to botocore Config.user_agent_extra for S3 clients.
great_expectations/execution_engine/pandas_execution_engine.py	Uses the helper when instantiating the execution engine’s internal S3 client.
great_expectations/datasource/fluent/pandas_s3_datasource.py	Uses the helper when constructing the datasource S3 client from boto3_options.
great_expectations/datasource/fluent/spark_s3_datasource.py	Uses the helper when constructing the datasource S3 client from boto3_options.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

[joshua-stauffer]

Thanks for the PR, @goanpeca -- the static-analysis check is fixed on develop, so merging in the latest should resolve it. You'll also need to sign the CLA, if you haven't already. Once CI is green we'll get this merged.

[github-actions]

All committers have signed the CLA. ✅

[goanpeca]

@cla-bot check

[goanpeca]

Thanks, @joshua-stauffer

I’ve rebased the branch onto the latest develop and confirmed the static-analysis checks and full local unit suite pass. The CLA check is also passing now. It looks like the remaining CI jobs are currently blocked by the fork-permission workflow, so please let me know if there’s anything else needed from my side.

[joshua-stauffer]

Thanks @goanpeca, I left a couple comments, but the core logic here is solid.

[joshua-stauffer]

I'd like to see this promoted to a public method (no leading _) and tested. On my machine this currently returns a minor version off, presumably due to editable install. Better would be to use the canonical __version__ constant. Since that's a top level export of GX, we should use a local import within the function to avoid circular references.__version__ also has the benefit of being always available, so no need for a fallback.

[joshua-stauffer]

this isn't idempotent - calling twice in a row would result in the suffix appended twice. Not currently a bug, but future surface area for one.