Skip to content

[CONTRIB] Add ExpectColumnValuesToNotBeOutliers across Pandas, SQL, and Spark - #12011

Open
chavalasantosh wants to merge 2 commits into
fivetran:developfrom
chavalasantosh:feature/expect-column-values-to-not-be-outliers
Open

[CONTRIB] Add ExpectColumnValuesToNotBeOutliers across Pandas, SQL, and Spark#12011
chavalasantosh wants to merge 2 commits into
fivetran:developfrom
chavalasantosh:feature/expect-column-values-to-not-be-outliers

Conversation

@chavalasantosh

Copy link
Copy Markdown

Summary

Closes #12003.

Promotes ExpectColumnValuesToNotBeOutliers from the experimental contrib implementation into supported core, with consistent IQR and standard-deviation outlier detection across Pandas, SQL, and Spark.

Implementation

  • Adds ExpectColumnValuesToNotBeOutliers as a supported ColumnMapExpectation.
  • Implements aggregate-backed center and spread metrics for:
    • Pandas using median/IQR and mean/sample standard deviation.
    • SQL using dialect-aware continuous percentiles and portable sample standard deviation statistics.
    • Spark using percentile_approx, mean, and stddev_samp.
  • Excludes null values consistently from aggregate statistics and row-level evaluation.
  • Preserves the specified inclusive outlier boundary: values where distance equals the threshold are outliers.
  • Raises NotImplementedError for unsupported methods.
  • Adds production metadata, Gallery-compatible documentation, a prescriptive renderer, package exports, and generated schema.
  • Adds cross-engine integration coverage for both methods, clean data, injected outliers, null handling, and threshold equality.
  • Adds the full requested datasource matrix for maintainer CI.
  • Adds no new top-level dependency.

Local validation

  • Pandas and SQLite expectation tests: 15 passed
  • Spark expectation tests: 6 passed
  • Generated schema synchronization: passed
  • Pytest marker coverage: passed
  • Existing Z-score expectation regression: passed
  • invoke lint: passed
  • invoke fmt --check: passed
  • Targeted mypy validation for the new production modules: passed

PostgreSQL and MySQL containers were unavailable in the local environment. Credentialed BigQuery, Snowflake, Databricks, Redshift, and SQL Server validation is delegated to maintainer CI as described in #12003.

  • Description of PR changes above includes a link to an existing GitHub issue
  • PR title is prefixed with one of: [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], [CONTRIB], [MINORBUMP]
  • Code is linted - run invoke lint (uses ruff format + ruff check)
  • Appropriate tests and docs have been updated
  • For any behavioral change to a data source, validation mechanic, or Expectation, at least one integration test exists in tests/integration/data_sources_and_expectations (see AGENTS.md#integration-test-requirement)
  • CI is green, including linting, mypy type-checking, and tests - this is required for merge
  • If this PR proposes adopting or recommending a particular third-party library or service, any affiliation with it (employment, financial interest, maintainership) is disclosed above for the reviewer's context

For more information about contributing, visit our community resources.

After you submit your PR, keep the page open and monitor the statuses of the various checks made by our continuous integration process at the bottom of the page. Please fix any issues that come up and reach out on Slack if you need help. Thanks for contributing!

@netlify

netlify Bot commented Jul 29, 2026

Copy link
Copy Markdown

👷 Deploy request for niobium-lead-7998 pending review.

Visit the deploys page to approve it

Name Link
🔨 Latest commit 712e3c7

@github-actions github-actions Bot added the cla-not-signed https://github.com/fivetran/great_expectations/blob/develop/CLA.md label Jul 29, 2026
@github-actions

github-actions Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

All committers have signed the CLA. ✅

@chavalasantosh chavalasantosh changed the title Add cross-engine column outlier expectation [CONTRIB] Add ExpectColumnValuesToNotBeOutliers across Pandas, SQL, and Spark Jul 29, 2026
@chavalasantosh

Copy link
Copy Markdown
Author

Hi maintainers — the required CI appears to be blocked by the fork-permission workflow rather than by an implementation or test failure.

The Notify on Lack of Permission step returned HTTP 400 with missing_args from the Slack webhook. This caused the static-analysis and unit-test jobs to be skipped, after which ci-required failed.

Could someone please approve the fork workflows and retry the failed jobs when convenient? Thank you!

@chavalasantosh

Copy link
Copy Markdown
Author

@cla-bot check

@github-actions github-actions Bot added cla-signed and removed cla-not-signed https://github.com/fivetran/great_expectations/blob/develop/CLA.md labels Jul 29, 2026
@joshua-stauffer

Copy link
Copy Markdown
Collaborator

hey @chavalasantosh, thanks for the PR! A maintainer needs to manually approve CI to run on fork PRs - I've kicked it off now, and will review this week.

@chavalasantosh chavalasantosh left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix null filtering for outlier map metric

@chavalasantosh

Copy link
Copy Markdown
Author

@cla-bot check

@joshua-stauffer

Copy link
Copy Markdown
Collaborator

hey @chavalasantosh, looks like unit tests are failing on your branch. I'll review once CI is passing. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Promote ExpectColumnValuesToNotBeOutliers to supported core with SQL + Spark

2 participants