Use case
ExpectColumnValuesToBeOfType behaves as a row-level check on Pandas object columns
but as a schema-level check everywhere else (see #11076). The row-level check generates one result type, the schema-level check another. We're splitting that overloaded behavior out into two separate Expectations; ExpectColumnValuesToBeOfType will continue checking each row, and will eventually be restricted to Pandas backends, while ExpectColumnTypeToBe will explicitly validate at the schema level for all backends.
Proposed capability
A new BatchExpectation (not ColumnMapExpectation) that asserts a column's declared
type:
- Supports SQLAlchemy dialects, Spark, and Pandas dtypes via the existing
table.column_types metric.
- Result is
{"observed_value": "<type name>"} — no row-level fields, no mostly.
- Type-matching semantics can reuse the existing
_validate_sqlalchemy /
_validate_spark / _validate_pandas aggregate logic in
expect_column_values_to_be_of_type.py.
Is this a net-new capability or an enhancement?
New Expectation
Alternatives considered
A number of users have proposed adding additional metrics to ExpectColumnValuesToBeOfType so that we can return the ColumnMapExpectation result format. Those queries are potentially expensive, and won't return any new information for the backends currently following the BatchExpectation path, since the column type is enforced at schema level.
Additional context
No response
Use case
ExpectColumnValuesToBeOfTypebehaves as a row-level check on Pandasobjectcolumnsbut as a schema-level check everywhere else (see #11076). The row-level check generates one result type, the schema-level check another. We're splitting that overloaded behavior out into two separate Expectations;
ExpectColumnValuesToBeOfTypewill continue checking each row, and will eventually be restricted to Pandas backends, whileExpectColumnTypeToBewill explicitly validate at the schema level for all backends.Proposed capability
A new
BatchExpectation(notColumnMapExpectation) that asserts a column's declaredtype:
table.column_typesmetric.{"observed_value": "<type name>"}— no row-level fields, nomostly._validate_sqlalchemy/_validate_spark/_validate_pandasaggregate logic inexpect_column_values_to_be_of_type.py.Is this a net-new capability or an enhancement?
New Expectation
Alternatives considered
A number of users have proposed adding additional metrics to
ExpectColumnValuesToBeOfTypeso that we can return the ColumnMapExpectation result format. Those queries are potentially expensive, and won't return any new information for the backends currently following theBatchExpectationpath, since the column type is enforced at schema level.Additional context
No response