[Spark] Advertise SUPPORT_COLUMN_DEFAULT_VALUE catalog capability#6948
Open
TimothyW553 wants to merge 10 commits into
Open
[Spark] Advertise SUPPORT_COLUMN_DEFAULT_VALUE catalog capability#6948TimothyW553 wants to merge 10 commits into
TimothyW553 wants to merge 10 commits into
Conversation
TimothyW553
force-pushed
the
timothy/delta-column-default-capability
branch
from
e657fca to
b3eb469
Compare
DeltaCatalog now overrides capabilities() to advertise TableCatalogCapability.SUPPORT_COLUMN_DEFAULT_VALUE on top of the delegate's capabilities. UCSingleCatalog forwards its delegate's capabilities, and its proxy reports none, so DeltaCatalog is what surfaces the capability to Spark. Without it, Spark's analyzer (validateCatalogForDefaultValue) rejects column DEFAULT clauses on tables created through Unity Catalog. Bumps the pinned Unity Catalog SHA used by the UC integration tests to 7ad03012 (unitycatalog#1589) so the end-to-end path is exercised. Adds an end-to-end test creating a Delta table with DEFAULT values across string, int, bigint, boolean, double, decimal, date, array, map, and struct columns.
TimothyW553
force-pushed
the
timothy/delta-column-default-capability
branch
from
b3eb469 to
ac00116
Compare
TimothyW553
commented
Jun 5, 2026
| # `unityCatalogVersion` is obtained by running this script with `--print-version`, so these two | ||
| # values are the single source of truth. | ||
| UC_PIN_SHA=e1f6e52acc39b925fd6a42180d400ca4e0a3895f | ||
| UC_PIN_SHA=7ad030127959453b9010285f4b814dd78b947070 |
Collaborator
Author
There was a problem hiding this comment.
this is temporary and may be reverted if the new SHA on master is higher.
…/delta-column-default-capability
openinx
reviewed
Jun 5, 2026
Collaborator
openinx
left a comment
openinx
left a comment
There was a problem hiding this comment.
LGTM, thanks @TimothyW553 for the work.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
DeltaCatalognow overridescapabilities()to advertiseTableCatalogCapability.SUPPORT_COLUMN_DEFAULT_VALUEon top of whatever the delegate catalog already reports.Spark's analyzer gates column
DEFAULTclauses on the catalog capability (ResolveDefaultColumns.validateCatalogForDefaultValue). When a Delta table is created through a non-session V2 catalog (for example Unity Catalog'sUCSingleCatalog, which delegates toDeltaCatalog), the missing capability made Spark rejectDEFAULTclauses even though Delta already supports theallowColumnDefaultstable feature. The session-catalog path is unaffected because Spark'sV2SessionCatalogalready reports this capability.How was this patch tested?
New tests in
DeltaTableCreationSuite:override adds
SUPPORT_COLUMN_DEFAULT_VALUE(this fails without the change).DEFAULTvalues acrossstring, int, bigint, boolean, double, decimal, and date columns, inserts a
row omitting those columns, and verifies the defaults are materialized.
build/sbt 'spark/testOnly org.apache.spark.sql.delta.DeltaTableCreationSuite -- -z "Default column values:"'-> 10 tests succeeded, 0 failed.Does this PR introduce any user-facing changes?
Yes. Column
DEFAULTclauses are now accepted on Delta tables created through anon-session Delta catalog (e.g. Unity Catalog), consistent with the existing
session-catalog behavior.