Repository navigation
[SPARK-58111][SQL] Scan and write schema narrowing for column-level UPDATE in DSv2 - #55518
anuragmantri wants to merge 16 commits into
Conversation
fb14c34 to
ae635f4
Compare
| val required = | ||
| AttributeSet(dataAttrs) ++ AttributeSet(Seq(cond)) ++ AttributeSet(rowIdAttrs) | ||
| val narrowOutput = relation.output.filter(required.contains) | ||
| relation.copy(table = table, output = dedupAttrs(narrowOutput ++ rowIdAttrs ++ metadataAttrs)) |
There was a problem hiding this comment.
Can an attribute in required be missing from relation.output?
rowIdAttrs seems to be added 2 times.
If we already have a dedupAttrs() then probably doesn't make sense build AttributeSets.
There was a problem hiding this comment.
Can an attribute in required be missing from relation.output?
No. dataAttrs come from the connector's requiredDataAttributes() which are resolved against relation (via V2ExpressionUtils.resolveRefs), so they're guaranteed to be present. The condition's referenced columns are also table columns from the user's WHERE clause. rowIdAttrs and metadataAttrs can be absent from relation.output (they're resolved separately), but they're not part of the filter. They're appended unconditionally afterward via dedupAttrs(narrowOutput ++ rowIdAttrs ++ metadataAttrs)
rowIdAttrs seems to be added 2 times. If we already have dedupAttrs() then probably doesn't make sense to build AttributeSets.
Agreed. I fixed it.
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Could you resolve the conflicts, @anuragmantri ?
ae635f4 to
a99bb2d
Compare
Thanks. I rebased and fixed the conflicts. |
| return new NamedReference[0]; | ||
| } | ||
|
|
||
|
|
There was a problem hiding this comment.
nit. Remove redundant empty line.
| * including the columns being updated. If {@link #requiredDataAttributes()} returns an empty | ||
| * array, Spark sends only the non-identity assigned columns (heuristic path). | ||
| * | ||
| * @since 4.2.0 |
| * <p> | ||
| * When empty (the default), Spark falls back to sending only the non-identity assigned columns. | ||
| * | ||
| * @since 4.2.0 |
There was a problem hiding this comment.
The APIs are now @since 4.4.0, since 4.3.0 has been released.
| val table = buildOperationTable(tbl, UPDATE, CaseInsensitiveStringMap.empty()) | ||
| val updatedCols = assignments.collect { | ||
| case Assignment(key: AttributeReference, value) | ||
| if !isIdentityAssignment(key, value) => |
There was a problem hiding this comment.
One liner doesn't violate the line-length rule, does it?
- case Assignment(key: AttributeReference, value)
- if !isIdentityAssignment(key, value) =>
+ case Assignment(key: AttributeReference, value) if !isIdentityAssignment(key, value) =>| // | ||
| // When dataAttrs is non-empty, the relation output is narrowed to include only columns | ||
| // required for a column-update write. When dataAttrs is empty, the full relation.output is | ||
| // preserved. |
There was a problem hiding this comment.
For function description, please follow the community style like the other code path.
/**
* ...
*/
| // When the connector supports column updates and declares required data attributes, | ||
| // the read relation is narrowed at analysis time so that | ||
| // GroupBasedRowLevelOperationScanPlanning uses only the needed columns for the scan. | ||
| // Otherwise the full relation output is used. |
There was a problem hiding this comment.
For function description, please follow the community style like the other code path.
/**
* ...
*/
| WriteDelta(writeRelation, cond, rowDeltaPlan, relation, projections, groupFilterCond) | ||
| } | ||
|
|
||
| // Builds the row delta projection for the column update path. |
There was a problem hiding this comment.
For function description, please follow the community style like the other code path.
/**
* ...
*/
| dataAttrsResolved(inRowAttrs) | ||
| } | ||
|
|
||
| // Validates the narrow-write-schema row projection output. |
There was a problem hiding this comment.
For function description, please follow the community style like the other code path.
/**
* ...
*/
| table.skipSchemaResolution || areCompatible(inRowAttrs, outRowAttrs) | ||
| table.skipSchemaResolution || | ||
| areCompatible(inRowAttrs, outRowAttrs) || | ||
| dataAttrsResolved(inRowAttrs) |
There was a problem hiding this comment.
nit. Please minimize the change of existing code as much as possible like the following.
table.skipSchemaResolution || areCompatible(inRowAttrs, outRowAttrs) ||
dataAttrsResolved(inRowAttrs)
| * is ignored and the full table row is sent (the default behavior). | ||
| * <p> | ||
| * When non-empty, the returned columns become the write schema in declared order. | ||
| * The connector must declare all columns it wants to receive, including the columns being |
There was a problem hiding this comment.
This is very strong assumption, but it seems that this PR didn't have a protection. May I ask if we have some kind of assertion or a test coverage for this?
There was a problem hiding this comment.
Each column the connector returns passes through V2ExpressionUtils.resolveRefs which throws AnalysisException if the column is non existent.
I added a test test("column-update: requiredDataAttributes throws AnalysisException for invalid column")
| // | ||
| // ColumnPruning observes exactly these references and narrows the physical scan accordingly. | ||
| // Connectors that need additional columns in the scan (e.g., partition columns for | ||
| // distribution) should declare them in requiredDataAttributes(). |
There was a problem hiding this comment.
IIUC, for the correctness, we need to throw AnalysisException if requiredDataAttributes is invalid.
There was a problem hiding this comment.
Each column the connector returns passes through V2ExpressionUtils.resolveRefs which throws AnalysisException if the column is non existent.
I added a test test("column-update: requiredDataAttributes throws AnalysisException for invalid column")
| // Connectors that need additional columns in the scan (e.g., partition columns for | ||
| // distribution) should declare them in requiredDataAttributes(). | ||
| // | ||
| // Note: AlignUpdateAssignments guarantees all assignment keys are top-level |
There was a problem hiding this comment.
Do we have a test coverage for this, AlignUpdateAssignments contract?
There was a problem hiding this comment.
I added a new test test("column-update: nested struct field update narrows to the root struct column") that updates an inner field in a struct, the AlignUpdateAssignment returns only the root key.
| * whether pk is already in the updated columns list and, if not, add it to | ||
| * requiredDataAttributes(). | ||
| * | ||
| * @since 4.2.0 |
| // build a plan to replace read groups in the table | ||
| val writeRelation = relation.copy(table = operationTable) | ||
| val projections = buildReplaceDataProjections(query, relation.output, metadataAttrs) | ||
| val query = updatedAndRemainingRowsPlan |
There was a problem hiding this comment.
This looks like duplications: Let's use one variable instead of mixing two variables, updatedAndRemainingRowsPlan and query.
There was a problem hiding this comment.
Done, used a single variable
| // GroupBasedRowLevelOperationScanPlanning needs explicit column declarations to narrow. | ||
| val rowAttrs: Seq[Attribute] = if (isNarrow) connectorDataAttrs else relation.output | ||
|
|
||
| (readRelation, rowAttrs) |
There was a problem hiding this comment.
Please return metadataAttrs too to avoid the following recomputation in the caller-side.
val metadataAttrs = resolveRequiredMetadataAttrs(relation, operationTable.operation)
There was a problem hiding this comment.
I changed this to return metadataAttrs too.
| // | ||
| // Works for both the full-scan and narrow-scan CoW paths. In the narrow case, | ||
| // readRelation.output is already restricted by buildCoWReadSetup, so projecting | ||
| // all plan.output gives the correct narrow write schema. |
There was a problem hiding this comment.
Use function description style.
| * | ||
| * @since 4.2.0 | ||
| */ | ||
| default boolean supportsColumnUpdates() { |
There was a problem hiding this comment.
Given the scope of this PR, shall we mention that DELETE and MERGE ignores this method?
There was a problem hiding this comment.
Done. The SupportsColumnUpdates Javadoc says Spark currently narrows only UPDATE, and RowLevelOperationInfo.updatedColumns() says other commands report an empty array.
| * | ||
| * @since 4.2.0 | ||
| */ | ||
| default NamedReference[] requiredDataAttributes() { |
There was a problem hiding this comment.
Given the scope of this PR, shall we mention that DELETE and MERGE ignores this method?
There was a problem hiding this comment.
Even though the scope of this PR is UPDATE only, we'd like this API to work for MERGE as well (DELETE doesn't benefit since it doesn't write data columns). I'm still assessing what it takes and will add a section in the SPIP on how it could be implemented.
Happy to add a "currently only consulted for UPDATE" note in the Javadoc for now and remove it when MERGE support lands.
|
I finished the first round review, @anuragmantri . |
There was a problem hiding this comment.
Thanks for the review @dongjoon-hyun. I addressed your comments and cleaned up some AI generated comments which were redundant.
| return new NamedReference[0]; | ||
| } | ||
|
|
||
|
|
| * including the columns being updated. If {@link #requiredDataAttributes()} returns an empty | ||
| * array, Spark sends only the non-identity assigned columns (heuristic path). | ||
| * | ||
| * @since 4.2.0 |
| * is ignored and the full table row is sent (the default behavior). | ||
| * <p> | ||
| * When non-empty, the returned columns become the write schema in declared order. | ||
| * The connector must declare all columns it wants to receive, including the columns being |
There was a problem hiding this comment.
Each column the connector returns passes through V2ExpressionUtils.resolveRefs which throws AnalysisException if the column is non existent.
I added a test test("column-update: requiredDataAttributes throws AnalysisException for invalid column")
| * whether pk is already in the updated columns list and, if not, add it to | ||
| * requiredDataAttributes(). | ||
| * | ||
| * @since 4.2.0 |
| // | ||
| // When dataAttrs is non-empty, the relation output is narrowed to include only columns | ||
| // required for a column-update write. When dataAttrs is empty, the full relation.output is | ||
| // preserved. |
| // Connectors that need additional columns in the scan (e.g., partition columns for | ||
| // distribution) should declare them in requiredDataAttributes(). | ||
| // | ||
| // Note: AlignUpdateAssignments guarantees all assignment keys are top-level |
There was a problem hiding this comment.
I added a new test test("column-update: nested struct field update narrows to the root struct column") that updates an inner field in a struct, the AlignUpdateAssignment returns only the root key.
| // | ||
| // ColumnPruning observes exactly these references and narrows the physical scan accordingly. | ||
| // Connectors that need additional columns in the scan (e.g., partition columns for | ||
| // distribution) should declare them in requiredDataAttributes(). |
There was a problem hiding this comment.
Each column the connector returns passes through V2ExpressionUtils.resolveRefs which throws AnalysisException if the column is non existent.
I added a test test("column-update: requiredDataAttributes throws AnalysisException for invalid column")
| dataAttrsResolved(inRowAttrs) | ||
| } | ||
|
|
||
| // Validates the narrow-write-schema row projection output. |
| table.skipSchemaResolution || areCompatible(inRowAttrs, outRowAttrs) | ||
| table.skipSchemaResolution || | ||
| areCompatible(inRowAttrs, outRowAttrs) || | ||
| dataAttrsResolved(inRowAttrs) |
| * | ||
| * @since 4.2.0 | ||
| */ | ||
| default NamedReference[] requiredDataAttributes() { |
There was a problem hiding this comment.
Even though the scope of this PR is UPDATE only, we'd like this API to work for MERGE as well (DELETE doesn't benefit since it doesn't write data columns). I'm still assessing what it takes and will add a section in the SPIP on how it could be implemented.
Happy to add a "currently only consulted for UPDATE" note in the Javadoc for now and remove it when MERGE support lands.
| .getOrElse { | ||
| throw new AnalysisException( | ||
| errorClass = "_LEGACY_ERROR_TEMP_3075", | ||
| messageParameters = Map( | ||
| "tableAttr" -> tableAttr.toString, | ||
| "scanAttrs" -> scanAttrs.mkString(","))) | ||
| } | ||
| } |
There was a problem hiding this comment.
I believe this is safe because condition-referenced columns are guaranteed to be in the scan. Please correct me if I'm wrong.
There was a problem hiding this comment.
No. Unfortunately, this PR should not remove this because the existing sanity check is used for other code path in the existing test cases. Please recover it.
I guess you may achieve your goal via the following. Please review and revise the following example for your purpose.
private def buildTableToScanAttrMap(
tableAttrs: Seq[Attribute],
scanAttrs: Seq[Attribute],
requiredAttrs: AttributeSet): AttributeMap[Attribute] = {
// Table attrs may be legitimately absent from a column-update narrowed scan, so map only
// those that have a matching scan attribute. Attrs referenced by the condition must always
// be present (computeNarrowReadAttrs keeps them in the scan); failing to map one would
// leave a dangling reference in the group filter, so keep the strict check for them.
val attrMapping = tableAttrs.flatMap { tableAttr =>
val matched = scanAttrs.find(scanAttr => conf.resolver(scanAttr.name, tableAttr.name))
if (matched.isEmpty && requiredAttrs.contains(tableAttr)) {
throw new AnalysisException(
errorClass = "_LEGACY_ERROR_TEMP_3075",
messageParameters = Map(
"tableAttr" -> tableAttr.toString,
"scanAttrs" -> scanAttrs.mkString(",")))
}
matched.map(scanAttr => tableAttr -> scanAttr)
}
AttributeMap(attrMapping)
}There was a problem hiding this comment.
Makes sense. Similar to other changes for column updates paths, I created a conditional method buildNarrowTableToScanAttrMap() which is called only during column updates and throws when any of the condition references are missing. My rationale is that the runtime filtering applies to only the filters so it is sufficient remap the filters only. Let me know if this understanding is incorrect.
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Thank you for updating, @anuragmantri .
BTW, I cannot find the vote for the mentioned SPIP. Does pass the vote officially, @anuragmantri ? For SPIP, we need an official vote result to move forward including merging something, don't we? (cc @huaxingao as the Shepherd of SPARK-56599 JIRA issue)
What changes were proposed in this pull request?
For SPIP: SPARK-56599
cc @aokolnychyi too because RowLevelOperation.java has been never changed since being added 4 years ago via the following.
|
Thanks for the review @dongjoon-hyun. For the SPIP, we are waiting for a few more maintainers to also review the design as well as the PR before going for a vote. |
e806004 to
4060cbf
Compare
There was a problem hiding this comment.
Re-checked from scratch through 72ec505. Findings 21 and 22 are resolved. 21: the exprId map-back is in at both remaining resolution sites, with two new case-sensitivity tests. 22: validateNoRowIdReassignment now takes connectorDataAttrs and honours the second remedy its error message advertises, with a test for it.
scanOnlyDataAttributes() and validatePartitionAttrsDeclared both came out this round (r3785330664, r3786136982). That brings finding 17 back, and the failure site I measured this time is worse than the one I described at round 5. I also ran a table shape none of the six earlier rounds tried, a table carrying a CHECK constraint, and it does not survive narrowing. Both are below with the arms I ran. The two column-update suites are 43/43 green on this head.
Two threads are still waiting on a reply from you: the columnUpdateSchema() rename question at r3785101803, and "I am worried about these constant if statements" at r3787781513.
Blocking
- 27. A table CHECK constraint breaks column-level UPDATE (new):
ResolveTableConstraintsresolves the constraint's columns againstquery.output, which this narrowing shrinks, so the UPDATE either fails withUNRESOLVED_COLUMNon a column the statement never mentions or spins out the Resolution batch withMax iterations (100) reached. Both wide paths handle the same table and constraint fine, so this is a narrowing regression. inline:RewriteUpdateTable.scala:438 - 17. An undeclared partition source column aborts the UPDATE (regressed): with the guard gone, a partitioned table whose scan reports
KeyGroupedPartitioningover a column the connector did not declare now dies inV2ScanPartitioningAndOrderingwith_LEGACY_ERROR_TEMP_1137. That rule is SPJ reporting, so it should degrade rather than throw, and the javadoc paragraph that replaced the guard reads advisory when the declaration is in fact mandatory. inline:SupportsColumnUpdates.java:49 - 28. The description documents a removed API (new): the "Public API additions" and "user-facing change" sections both describe
scanOnlyDataAttributes(), which this head deleted, and the write-side bullet ("scanOnlyDataAttributes()columns stay in the scan for planning but are excluded from the write payload") now says the opposite of the new javadoc, which states such columns do appear in the written row. dongjoon's note that thewriteUpdatedefaults throw rather than "delegate towrite(...)" (pullrequestreview-4890264624) is still unfixed. The five new error conditions are not mentioned at all.
Non-blocking
- 29.
writeUpdate(metadata, record)should default towriteUpdate(record)(new): which overload Spark calls depends on whether the operation declares required metadata attributes, so a connector that implements only the single-argument form still hitsDATA_SOURCE_WRITE_UPDATE_NOT_IMPLEMENTED. The siblingwrite(metadata, record)already delegates towrite(record). inline:DataWriter.java:104 - 20. Undocumented analysis-time contracts (round 6): r3741376797. Partly addressed. The
updatedColumns()coverage rule and the nested root-column rule are onrequiredDataAttributes()now. Two enforced rules are still unstated: an empty array is rejected, and every row-ID column must be declared whenrepresentUpdateAsDeleteAndInsert()is true. - 23. A nested declaration escapes as
ClassCastException(round 6): r3741376788. Re-measured on this head with"pk,dep,s.c1", stillclass Alias cannot be cast to class AttributeReference. A guard onfieldNames.length != 1closes it. - 24. Duplicate declarations are not rejected (round 6): r3741376790. Re-measured,
"pk,pk,salary,dep"still reaches the connector asupdateSchema = struct<pk,pk,salary,dep>with the UPDATE succeeding. - 25. The column-update suites bypass the existing UPDATE battery (round 6): r3741376793. Unchanged.
- 26. The fixture drops
info.options(round 6): r3741376794. Unchanged,InMemoryRowLevelOperationTable.scala:308still passesCaseInsensitiveStringMap.empty().
Minor
- 30. Scan-only wording left behind (new): three comments and two test names still describe the removed
scanOnlyDataAttributes()category. inline:DeltaBasedColumnUpdateTableSuite.scala:627
| isIdentityAssignment(a.key.asInstanceOf[Attribute], a.value)) | ||
| .flatMap(_.value.references.toSeq) | ||
| .toSeq | ||
| val extraRefs = (cond.references.toSeq ++ nonIdentityRhsRefs) |
There was a problem hiding this comment.
Finding 27. A table carrying a CHECK constraint cannot be updated through this path at all.
ResolveTableConstraints wraps the rewritten query in Filter(CheckInvariant(...), query) and resolves the constraint's column references against query.output (ResolveTableConstraints.scala:60-61). RowLevelOperationTable.constraints() delegates to the real table, so the rule fires for the operation table too. Narrowing takes those columns out of both the scan and the write query, and nothing here accounts for it.
Measured on 72ec505. Same table in every arm, pk INT NOT NULL, id INT, dep STRING, extra INT partitioned by identity(dep), same constraint ALTER TABLE ... ADD CONSTRAINT positive_extra CHECK (extra > 0), same statement UPDATE t SET id = -1 WHERE pk = 1:
| connector | result |
|---|---|
wide supports-deltas (MoR) |
works |
| wide default fixture (CoW) | works |
column-update (MoR) |
[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter with name `extra` cannot be resolved. Did you mean one of the following? [`dep`, `id`, `pk`, `index`, `_partition`] |
column-update-cow (CoW) |
same |
So it is a narrowing regression, not pre-existing. The error names a column the statement never mentions and says nothing about requiredDataAttributes().
The second arm is worse. When the constraint's column is in the narrow scan, because the condition or an assignment RHS references it, the delta path still drops it from the write query in the Project below (:320-323), and the Resolution batch never converges:
| connector | statement | result |
|---|---|---|
column-update (MoR) |
SET id = -1 WHERE extra > 3 |
RuntimeException: Max iterations (100) reached for batch Resolution |
column-update (MoR) |
SET id = extra + 1 WHERE pk = 1 |
same |
column-update-cow (CoW) |
SET id = -1 WHERE extra > 3 |
works |
The CoW row is the control for that pair: buildNarrowReplaceDataUpdateProjection maps all of plan.output, so the column survives into query.output there, and only the delta builder drops it.
I applied the matching fix locally and it holds. Carrying the remaining narrow scan columns through the Project:
val connectorIds = connectorDataAttrs.map(_.exprId).toSet
val carryAlong = plan.output.filterNot { a =>
MetadataAttribute.isValid(a.metadata) || rowIdAttrSet.contains(a) ||
assignedKeyIds.contains(a.exprId) || connectorIds.contains(a.exprId)
}
Project(
Seq(operationType) ++ assignedValues ++ connectorPassThroughValues ++
metadataValues ++ rowIdValues ++ originalRowIdValues ++ carryAlong,
plan)turns both looping arms green and keeps the payload narrow: updateSchema stays struct<pk, dep, id> and extra does not leak into it, because updateRowProjection picks connectorDataAttrs out of the Project by name. DeltaBasedColumnUpdateTableSuite + GroupBasedColumnUpdateTableSuite stay 43/43. And the constraint is genuinely enforced again afterwards: with CHECK (id > 0) instead, SET id = -1 WHERE extra > 3 gives CHECK_CONSTRAINT_VIOLATION.
That still leaves the first arm, where the constraint column is not in the narrow scan at all. Reading it is the only way Spark can re-validate the constraint on the new row, so computeNarrowReadAttrs would have to include the Check constraints' references as well. The alternative is to refuse to narrow a table that carries Check constraints, with a clear error. I would widen rather than refuse, and note in the javadoc that a table with CHECK constraints narrows less.
There was a problem hiding this comment.
Good catch. For constraints I did both read side and the write side changes to include the constraint columns. They are discarded by the update projects before the write so it should still pass narrow writes to the connector.
| * For updates on nested fields such as {@code SET t.s.c1 = -1} the connector should declare the | ||
| * root struct column {@code s} rather than any nested field. | ||
| * <p> | ||
| * This also covers columns needed only for planning, e.g. resolving the table's partitioning |
There was a problem hiding this comment.
Finding 17 (regressed). Dropping validatePartitionAttrsDeclared is a fair policy call, and I agree Spark should not decide what a connector wants. But the thing that breaks when a partition source column is missing is not the connector's write requirement. It is Spark's own SPJ reporting rule, and it throws instead of degrading.
This paragraph reads as advisory, "also covers columns needed only for planning". For any partitioned table whose scan reports KeyGroupedPartitioning it is mandatory. Measured on 72ec505, requiredDataAttributes() = [pk, id], table pk INT NOT NULL, id INT, dep STRING partitioned by identity(dep), UPDATE t SET id = -1 WHERE pk = 1:
org.apache.spark.sql.AnalysisException: Unable to resolve dep given [pk,id,_partition,index].
at QueryCompilationErrors$.cannotResolveAttributeError(QueryCompilationErrors.scala:1988)
at V2ExpressionUtils$.resolveRef(V2ExpressionUtils.scala:56)
at V2ExpressionUtils$.toCatalystTransformOpt(V2ExpressionUtils.scala:147)
at V2ExpressionUtils$.toCatalystOpt(V2ExpressionUtils.scala:128)
at V2ScanPartitioningAndOrdering$$anonfun$partitioning$2 ... (V2ScanPartitioningAndOrdering.scala:51)
The same connector against an unpartitioned table passes, so the partitioning is the trigger.
This also corrects what I wrote at round 5 (r3735139324), where I said the read side would "silently report no partitioning". It does not. V2ScanPartitioningAndOrdering.partitioning resolves the keys against scanRelation.relation (ExtractV2ScanInfo, DataSourceV2Relation.scala:457-461), and that relation is exactly what this PR narrows. toCatalystTransformOpt's IdentityTransform case then calls the throwing resolveRef. Both the Opt naming and the comment two lines below the call site ("Keep the partitioning when at least one of its keys is still in the scan output") say the rule means to tolerate a pruned key. It never gets the chance. The write-side site I measured at round 4 (r3735141175) is still reachable too, for a connector that clusters on a data column.
Either fix closes the read side:
- resolve the keys with a non-throwing lookup and drop the partitioning when a key is absent. This asks nothing of connectors and is what the rule already intends;
- or have
computeNarrowReadAttrsadd the table's partition-transform references to the scan. This is not restrictive the way the removed guard was: the payload is projected down toconnectorDataAttrsby name, so a partition column sitting in the scan never reaches the writer. I confirmed that projection behaviour while measuring finding 27, where a carried-along column stayed out ofupdateSchema().
Whichever you take, this sentence should say the declaration is required rather than that it "covers" the case, and there should be a test for a column-update UPDATE on a partitioned table whose partition column is not declared. Every fixture in the suite declares dep, which is why 43 tests pass over this.
There was a problem hiding this comment.
Thanks for testing this. I like option 1 but I think that change belongs to it is own PR as it touches code unrelated to this PR. If you agree, I will create a separate JIRA for this. I would also defer adding the test that skips adding the partition column since it will fail now.
There was a problem hiding this comment.
SPARK-59721 (#58979) is merged, so I added the deferred test: column-update: partition source column not in requiredDataAttributes declares [pk, id] on a table partitioned by dep. It passes, and fails with Unable to resolve dep given [pk,id,_partition,index] if the #58979 change to V2ScanPartitioningAndOrdering is reverted. The description links the JIRA.
| throw new SparkUnsupportedOperationException( | ||
| "DATA_SOURCE_WRITE_UPDATE_NOT_IMPLEMENTED", | ||
| Map.of("class", getClass().getName())); | ||
| } |
There was a problem hiding this comment.
Finding 29. A connector that implements only writeUpdate(record) still fails here.
Which overload Spark calls depends on whether the operation declares required metadata attributes (ReplaceDataExec.writingTask, WriteToDataSourceV2Exec.scala:385-397), which is not visible from the interface. The single-argument form's doc says it is "Equivalent to writeUpdate(Object, Object) for writers that do not require metadata", so implementing that one alone reads as sufficient, and it is not.
The sibling pair a few lines up already solves this: write(T metadata, T record) defaults to write(record) (:83-85). Mirroring it keeps the two channels consistent:
| throw new SparkUnsupportedOperationException( | |
| "DATA_SOURCE_WRITE_UPDATE_NOT_IMPLEMENTED", | |
| Map.of("class", getClass().getName())); | |
| } | |
| writeUpdate(record); | |
| } |
A connector that overrides neither form still gets DATA_SOURCE_WRITE_UPDATE_NOT_IMPLEMENTED, from the single-argument default. The trade-off is the one write already makes: a writer that needs the metadata must override the two-argument form.
There was a problem hiding this comment.
I took the suggestion and added java docs to reflect this.
| |""".stripMargin) | ||
|
|
||
| // requiredDataAttributes = [pk, id]; cond refs [pk] only; RHS is a literal. | ||
| // `dep` is scan-only declared (needed for scan-side partitioning resolution) so it must |
There was a problem hiding this comment.
Finding 30. Wording left over from the removed scanOnlyDataAttributes(). There is no scan-only category any more: dep is in requiredDataAttributes() like every other declared column.
Here, and again at :654-657 ("dep is scan-only declared (needed for scan-side partitioning resolution) so it also appears; extra is neither declared, scan-only, nor referenced"). Also InMemoryRowLevelOperationTable.scala:303, which calls dep "the write-side clustering key, see clusterColumnRef" while clusterColumnRef is PARTITION_COLUMN_REF (_partition), a metadata column, not dep.
Two test names in this file no longer match what they assert, same cause. "rowSchema contains only the single assigned column" (:42) asserts [pk, dep, id], and "rowSchema is empty for a full identity update" (:82) asserts [pk, dep]. "rowSchema" is also the pre-redesign name for what is now updateSchema().
There was a problem hiding this comment.
Done. I fixed all the mentioned places.
|
Thanks for the new round @peter-toth. I addressed them in 9932f88 except |
There was a problem hiding this comment.
Re-checked from scratch through 9932f88 — findings 20, 23, 24, 25, 26, 28, 29 and 30 resolved, nothing regressed. Finding 25's fix is the one that pays off most: with both suites on the existing UPDATE bases the column-update baseline goes from 43 tests to 155 green.
Finding 27 is fixed for every shape I measured at round 7, and carryAlongValues is the right half of it. One shape it does not cover is a CHECK constraint on a nested field. Check#predicate().references() hands back a two-part FieldReference, resolveRef turns that into an Alias, and the asInstanceOf[AttributeReference] behind it is erased, so it escapes as a ClassCastException a few frames later. That is finding 31, measured on both narrow paths with the wide path as the control.
On finding 17 — a separate JIRA and PR is the right split, and the javadoc now reads mandatory rather than advisory, which was half of what I raised. I have moved it to non-blocking on that basis. Two asks: link the follow-up JIRA in the description, and let the deferred test land with that PR rather than being dropped.
Two threads are still waiting on a reply from you: the columnUpdateSchema() rename question at r3785101803, and "I am worried about these constant if statements" at r3787781513.
Blocking
- 31. A nested-field CHECK constraint is a
ClassCastException(new):resolveCheckConstraintAttrsresolves the constraint's references at full depth, soCHECK (s.c1 > 0)yields anAliaswhere anAttributeReferenceis required. Both narrow paths die; the wide delta path handles the same table, constraint and statement fine. Resolving each reference's root column closes it. inline:RewriteUpdateTable.scala:473
Non-blocking
- 17. An undeclared partition source column aborts the UPDATE (round 7): re-measured on this head,
AnalysisException: Unable to resolve dep given [pk,id,_partition,index]out ofV2ScanPartitioningAndOrdering. Moved down from blocking: the declaration is now documented as mandatory and the statement fails rather than corrupting anything, so the non-throwing key resolution can travel separately. r4053024878
Minor
- 32. The duplicate error names a column the connector never declared (new): with case-insensitive analysis the message carries the lower-cased name, so
["PK", "Pk"]reports[pk]. The sibling nested-attribute check reportsdescribe(), i.e. the declared spelling. inline:RewriteRowLevelCommand.scala:146
| val refs = checks.flatMap(_.predicate().references()).toSeq | ||
| V2ExpressionUtils.resolveRefs[AttributeReference](refs, relation) |
There was a problem hiding this comment.
Finding 31. Check#predicate().references() returns references at full depth, so a constraint on a nested field gives a two-part FieldReference. V2ExpressionUtils.resolveRef puts that through LogicalPlan.resolve, which returns an Alias(GetStructField(...)) for a nested path, and its asInstanceOf[T] is erased — so the Alias rides along inside a Seq[AttributeReference] and blows up at the first frame that needs the real type.
Measured on 9932f88, delta MoR (column-update) and CoW (column-update-cow), table pk INT NOT NULL, id INT, dep STRING, s STRUCT<c1: INT, c2: INT> partitioned by dep:
ALTER TABLE t ADD CONSTRAINT positive_c1 CHECK (s.c1 > 0);
UPDATE t SET id = -1 WHERE pk = 1;java.lang.ClassCastException: class org.apache.spark.sql.catalyst.expressions.Alias cannot be cast to
class org.apache.spark.sql.catalyst.expressions.AttributeReference
at RewriteRowLevelCommand.dedupAttrs(RewriteRowLevelCommand.scala:107)
at RewriteUpdateTable$.computeNarrowReadAttrs(RewriteUpdateTable.scala:454)
Control: the same table, constraint and statement on the wide supports-deltas connector passes, so this is specific to narrowing. A CHECK on a top-level undeclared column is fine on all three of delta MoR, CoW and the split path — I ran that too.
Narrowing is at root-column granularity anyway, so truncating each reference to its root column is enough:
| val refs = checks.flatMap(_.predicate().references()).toSeq | |
| V2ExpressionUtils.resolveRefs[AttributeReference](refs, relation) | |
| val refs = checks.flatMap(_.predicate().references()).toSeq | |
| .map(ref => FieldReference(Seq(ref.fieldNames.head))) | |
| V2ExpressionUtils.resolveRefs[AttributeReference](refs, relation) |
Needs import org.apache.spark.sql.connector.expressions.FieldReference. With that applied both probes pass and the two suites stay at 155. Worth a nested-field-constraint test on both narrow paths.
There was a problem hiding this comment.
With the read relation no longer narrowed, CHECK constraints resolve against the full write query, so a nested-field CHECK such as CHECK (s.c1 > 0) no longer needs special handling. Covered by CHECK constraint on a nested field narrows correctly in both suites.
| val duplicates = normalizedNames.groupBy(identity).collect { | ||
| case (_, occurrences) if occurrences.length > 1 => occurrences.head | ||
| }.toSeq |
There was a problem hiding this comment.
Finding 32. occurrences.head is the normalized name, not the declared one, so with spark.sql.caseSensitive off a declaration of ["PK", "Pk"] reports [pk] — a spelling the connector never wrote. The nested-attribute check nine lines up reports _.describe(), so the two messages disagree about which spelling the connector author gets to see.
Keeping the first declared spelling:
| val duplicates = normalizedNames.groupBy(identity).collect { | |
| case (_, occurrences) if occurrences.length > 1 => occurrences.head | |
| }.toSeq | |
| val duplicates = refs.zip(normalizedNames).groupBy(_._2).collect { | |
| case (_, occurrences) if occurrences.length > 1 => occurrences.head._1.describe() | |
| }.toSeq |
There was a problem hiding this comment.
The duplicate error reports the spelling the connector declared, as the nested check does. ["PK", "Pk"] reports [PK] (column-update: case-only duplicate requiredDataAttributes throws AnalysisException).
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
I reviewed 9932f88 from scratch, skipping everything already raised in the earlier rounds (nested CHECK CCE, undeclared partition/clustering columns, @since/MiMa version, the if structure, the group-filter attr map). The inline comments below are the new items: four narrowing correctness issues, one API-contract issue that is the residue of the writeUpdate delegation change, one doc/message mismatch, and a few test-infra gaps. All were verified by reading the code; I did not run them.
| .collect { case a: AttributeReference => a } | ||
| .filter(relationSet.contains) | ||
| val checkConstraintAttrs = resolveCheckConstraintAttrs(relation) | ||
| dedupAttrs(connectorDataAttrs ++ extraRefs ++ checkConstraintAttrs) |
There was a problem hiding this comment.
The exprId map-back added for the connector declaration (resolveRequiredDataAttrs) is missing for the other sources that feed the narrow relation output: cond.references, the assignment RHS references, resolveCheckConstraintAttrs, and rowIdAttrs. Those instances come out of AttributeSeq.resolve as a.withName(<requested spelling>), and since dedupAttrs keeps the first instance per exprId, a user-spelled attribute becomes the scan's column name whenever the column is not also declared.
With the default spark.sql.caseSensitive=false, column-update table pk INT NOT NULL, salary INT, bonus INT, dep STRING (declared [pk, dep, salary]):
UPDATE t SET salary = salary + BONUS WHERE pk = 1narrow output = [pk#1, dep#4, salary#2, BONUS#3, ...] -> pruneColumns asks the connector for BONUS -> InMemoryScanBuilder drops it (case-sensitive name match) -> the scan relation has no BONUS#3 -> INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND at execution. A connector that echoes the table spelling instead hits NoSuchElementException in PushDownUtils.toOutputAttrs. WHERE EXTRA > 3 and CHECK (EXTRA > 0) take the same path. The wide path is immune because it always starts from relation.output.
Simplest fix is to build the narrow output from the relation's own attributes so both spelling and column order are preserved:
val narrowSet = AttributeSet(connectorDataAttrs ++ extraRefs ++ checkConstraintAttrs)
relation.output.filter(narrowSet.contains)(and the same relation.output.filter(...) for rowIdAttrs in buildNarrowRelationWithAttrs). Worth a case test with an undeclared, differently-cased column in the RHS and in the WHERE clause.
There was a problem hiding this comment.
This no longer applies: RewriteUpdateTable now keeps the existing read relation and narrows only the write relation, so no attribute in the connector's or the user's spelling can reach a scan. The declared attributes are still mapped back to the relation's own attributes by expr ID. I kept the regression tests for your example on both paths, and they now also check that the scan uses the table's spelling (undeclared, differently-cased columns in the RHS and WHERE clause are read with the table's own spelling).
| } | ||
| } | ||
|
|
||
| val rowIdValues = plan.output.filter(rowIdAttrSet.contains) |
There was a problem hiding this comment.
When a row-ID column is reassigned, this projection emits pk twice with different exprIds: assignedValues contributes Alias(pk + 10, "pk") (fresh exprId) and rowIdValues contributes the original pk#1. The write projections survive only because findColOrdinal takes the first name match, but name-based resolution over query.output does not:
ResolveTableConstraintswraps the query inFilter(CheckInvariant(UnresolvedAttribute("pk") > 0), query)for a table withCHECK (pk > 0)->AttributeSeq.resolvesees two candidates ->AMBIGUOUS_REFERENCE.- A connector whose
Writeclusters or orders bypkfails the same way inV2WritesviaDistributionAndOrderingUtils.prepareQuery.
Repro on the non-split column-update fixture: ALTER TABLE t ADD CONSTRAINT positive_pk CHECK (pk > 0) then UPDATE t SET pk = pk + 10, salary = -1 WHERE dep = 'hr'. The existing test pk already in updatedColumns is not duplicated only inspects updateSchema() (already de-duplicated by name lookup), so it cannot catch this. The wide buildWriteDeltaUpdateProjection replaces the value in place and never has the duplicate.
Building the output in a single pass over plan.output (the way buildNarrowReplaceDataUpdateProjection does) -- assigned attrs replaced in place, metadata preserved/nullified, everything else passed through -- followed by originalRowIdValues removes the duplicate and the first-match ordering dependency at the same time.
There was a problem hiding this comment.
The rewrite now uses the existing projection builders, which project the original row ID under a separate name, so pk appears once. The row-ID reassignment tests with a CHECK constraint on pk still pass, for example column-update: CHECK constraint on a reassigned row-ID column is enforced.
| dataAttrs: Seq[AttributeReference], | ||
| metadataAttrs: Seq[AttributeReference], | ||
| rowIdAttrs: Seq[AttributeReference] = Nil): DataSourceV2Relation = { | ||
| val attrs = dedupAttrs(dataAttrs ++ rowIdAttrs ++ metadataAttrs) |
There was a problem hiding this comment.
This is the first time a leaf DataSourceV2Relation.output is a strict subset of the table's columns, and PushDownUtils.pruneColumns still assumes the old invariant in two places:
toOutputAttrsmaps every field inscan.readSchema()back torelation.outputby exact name (nameToAttr(a.name)). A connector that reports more columns than requested -- whichSupportsPushDownRequiredColumnsexplicitly allows ("it's also OK to do the pruning partially") and which our ownSIMULATE_PARTIAL_COLUMN_PRUNINGfixture models -- now throwsNoSuchElementException: key not found: salaryfrom the optimizer. Same call inGroupBasedRowLevelOperationScanPlanning.- The
case _ => scanBuilder.build() -> relation.outputfallback for aScanBuilderwithoutSupportsPushDownRequiredColumnspairs full-width reader rows with the narrowoutput, soBatchScanExecbinds the wrong ordinals and misaligned values flow intowriteUpdate/updatewith no error.
Nothing validates or documents that a SupportsColumnUpdates scan builder must implement SupportsPushDownRequiredColumns and must not over-report. A sub-suite with SIMULATE_PARTIAL_COLUMN_PRUNING -> "true" in extraTableProps would reproduce (1) immediately. I would either make toOutputAttrs tolerate extra fields (they are unreferenced above the scan) or reject/document both shapes.
There was a problem hiding this comment.
This no longer applies: the read relation keeps every table column, so PushDownUtils.pruneColumns and toOutputAttrs see the same relation as for any other row-level operation. A scan builder without SupportsPushDownRequiredColumns reads every column and the write stays correct; the Javadoc now says the builder should implement it. The partial-pruning case is covered by scan that returns unrequested columns still writes narrow rows in both suites.
| throw QueryCompilationErrors.duplicateRequiredDataAttributeError( | ||
| operation.getClass.getName, duplicates) | ||
| } | ||
| val resolved = V2ExpressionUtils.resolveRefs[AttributeReference]( |
There was a problem hiding this comment.
LogicalPlan.resolve falls back to metadataOutput, so a metadata column such as _partition is accepted here as a data attribute (the map-back byExprId.getOrElse(a.exprId, a) passes it through untouched). The javadoc actively steers connectors this way ("Partition columns ... must be declared here too"), and in our fixture the partition column is exactly a metadata column.
On the split path this corrupts the row silently: buildNarrowRelationWithAttrs leaves the metadata attr in the middle of the data attrs (declared order, dedupAttrs first-wins), buildNarrowDeletesAndInserts partitions output into rowAttrs ++ metadataAttrs for the DELETE/REINSERT projections, but the Expand output attrs keep the interleaved order, so projection slot i and output attribute i disagree. With column-update-split-req-attrs = "pk,_partition,dep" and UPDATE t SET dep = 'x' WHERE pk = 1, dep's value lands in the _partition slot and vice versa (both STRING, so nothing fails); with a type mismatch it reads garbage. The plan stays resolved because areCompatible compares names/types only. The non-split and CoW paths are self-consistent (name/exprId based) but still put the metadata column into updateSchema().
Suggest rejecting MetadataAttribute.isValid(a.metadata) attrs here with a dedicated error (or resolving against LocalRelation(relation.output) so the fallback never applies), and clarifying in the javadoc that metadata partition columns travel through requiredMetadataAttributes().
There was a problem hiding this comment.
A metadata column in requiredDataAttributes() is now rejected with COLUMN_UPDATE_METADATA_REQUIRED_DATA_ATTRIBUTE, and the Javadoc says to return metadata columns from requiredMetadataAttributes() instead. Covered on the in-place and split delta paths (metadata column in requiredDataAttributes throws AnalysisException).
| * | ||
| * @since 4.3.0 | ||
| */ | ||
| default void writeUpdate(T record) throws IOException { |
There was a problem hiding this comment.
After the delegation change, the javadoc on both overloads says overriding either one is enough ("Implementations must override this method, or writeUpdate(Object, Object)", and both @throws clauses say "overrides neither"). That is not what the dispatch does: ReplaceDataExec.writingTask picks DataWithProjectionWritingSparkTask whenever requiredMetadataAttributes() is empty (the default), and that task calls the 1-arg overload only. So a group-based connector that overrides just writeUpdate(metadata, record) -- the overload SupportsColumnUpdates' javadoc links to -- gets DATA_SOURCE_WRITE_UPDATE_NOT_IMPLEMENTED ("does not implement writeUpdate", which is false) on every UPDATE. The reverse shape works, so the contract is asymmetric in exactly the direction the docs invite.
The PR description already states the stricter rule ("must override writeUpdate(record)"). I would make the javadoc and the error text say that, and drop the "or" clause. Also, no test goes through the 1-arg dispatch at all: PartitionBasedColumnUpdateOperation always declares [_partition, index] and ignores no-metadata, so every CoW column-update test uses DataAndMetadataWritingSparkTask. A no-metadata CoW fixture variant would cover it.
There was a problem hiding this comment.
The overloads are now writeColumnUpdate(record) and writeColumnUpdate(metadata, record). The Javadoc says Spark calls the metadata overload when the operation declares requiredMetadataAttributes() and the record-only overload otherwise. The record-only default throws and the metadata overload delegates to it, so a writer must override the overload Spark calls. Tests cover both dispatches, a writer that overrides only writeColumnUpdate(record), and a writer that overrides neither.
| val expanded = new BufferedRows(buf.key, schema) | ||
| buf.log.foreach { logRow => | ||
| val opName = logRow.getUTF8String(0) | ||
| if (opName == reinsertOpName) { |
There was a problem hiding this comment.
This doCommit only materializes Reinsert entries; Insert (and Update) entries are dropped, while the parent DeltaBasedColumnUpdateOperation.doCommit handles both. The operation is returned for every command, so a MERGE ... WHEN NOT MATCHED THEN INSERT on a column-update-split table would commit nothing for the inserted rows and no test would notice: neither column-update suite runs MERGE INTO, and the only DELETE test (delta) asserts just updatedColumns.isEmpty; there is no CoW DELETE test at all. Since the mix-in's documented contract is precisely that DELETE and MERGE fall back to full-width rows, that fallback deserves a test on both connector types. Mirroring the parent's insertOpName branch here is a one-liner.
There was a problem hiding this comment.
The split fixture's commit now handles Insert as well as Reinsert, and MERGE runs on the split path too (column-update split: MERGE falls back to full-width rows).
| var lastWriteInfo: LogicalWriteInfo = _ | ||
| // used in column-update tests to verify that Spark passed the correct updated column list | ||
| // to the connector via RowLevelOperationInfo.updatedColumns() | ||
| var lastUpdatedColumns: Array[NamedReference] = Array.empty |
There was a problem hiding this comment.
Two bookkeeping gaps for this new field:
copy()below copiesreplacedPartitions,lastWriteInfoandlastWriteLogbut notlastUpdatedColumns, soInMemoryRowLevelOperationTableCatalog.loadTablesnapshots always report an empty array. The suites happen to readliveTable, so nothing fails today, but the next test that follows thelastWriteInfo/loadTablepattern will.TxnTable.newRowLevelOperationBuilder(txns.scala:146) writes it to the live delegate immediately, whereas the sibling fields are staged and copied incommit()-- which already copieslastUpdatedColumnstoo, so the immediate write is redundant. After an UPDATE that fails analysis,table.lastUpdatedColumnsreflects the failed statement whilelastWriteInfo/lastWriteLogreflect the previous commit.
Add copied.lastUpdatedColumns = lastUpdatedColumns and delete the immediate delegate write in txns.scala.
There was a problem hiding this comment.
copy() now carries lastUpdatedColumns, and the transactional table sets it on commit instead of when the operation is built.
| * ColumnPruning may further tighten in ways unrelated to the narrowing contract). | ||
| */ | ||
| protected def checkLastScanExcludes(excludedNames: String*): Unit = { | ||
| val schema = Option(table.lastScanSchema).getOrElse(StructType(Nil)) |
There was a problem hiding this comment.
With lastScanSchema == null (the state before {} resets to) this substitutes an empty schema, so the negative assertion passes vacuously. GroupBasedColumnUpdateTableSuite's "scan excludes columns outside required + cond" relies solely on this helper, so removing recordLastScanSchema from PartitionBasedColumnUpdateOperation's scan builder would leave it green. assert(table.lastScanSchema != null, ...) instead of getOrElse(StructType(Nil)) closes it.
Related: the group-path "runtime group filtering data correctness" test only calls checkAnswer, so it cannot tell whether a group-filter subquery was injected or which partitions were rewritten. DeltaBasedUpdateTableSuite.checkUpdateRuntimeGroupFiltering shows the shape (executeAndCheckScans(..., groupFilterScanSchema = Some(...)) + checkReplacedPartitions(Seq("hr"))); the narrow variant should assert the same.
There was a problem hiding this comment.
checkLastScanExcludes and checkLastScanIncludes now fail when no scan schema was recorded. The group-filtering test checks both scan schemas and the replaced partitions.
| * The write schema must exactly match the columns declared via | ||
| * `SupportsColumnUpdates.requiredDataAttributes()` (same columns, same order). | ||
| */ | ||
| private def dataAttrsResolved(inRowAttrs: Seq[Attribute]): Boolean = { |
There was a problem hiding this comment.
nit. This is byte-identical to ReplaceData.dataAttrsResolved, and the inUpdateAttrs / insertResolved / updateResolved blocks above differ from ReplaceData's only in outRowAttrs. Everything they touch (operation, projectedDataAttrs, areCompatible) already lives on RowLevelWrite, next to projectedMetadataAttrs, so both could move there (with an abstract updateRowProjection accessor, since the two projections types differ). The operation.isInstanceOf[SupportsColumnUpdates] guard is also redundant: projectedDataAttrs returns Nil otherwise and areCompatible fails on the size mismatch.
There was a problem hiding this comment.
This code is gone. ReplaceData and WriteDelta validate against the narrowed write relation with the existing rowAttrsResolved. The only addition is RowLevelWrite.references, defined once in the trait.
| props | ||
| } | ||
|
|
||
| private def createAndInitTableReplaceData(schemaString: String, jsonData: String): Unit = { |
There was a problem hiding this comment.
nit. This helper is functionally identical to the inherited createAndInitTable (same identity(dep) partitioning, and column-update-cow is exactly this suite's extraTableProps), and DeltaBasedColumnUpdateTableSuite.createAndInitTableFromInfo is the same story because DeltaBasedColumnUpdateOperationFromInfo is an empty subclass of DeltaBasedColumnUpdateOperation (the column-update-from-info flag and class can go). The other six createAndInitTableXxx copies differ only in one props.put; a props parameter on the base helper (replacing, not merging, extraTableProps, since column-update is matched first in newRowLevelOperationBuilder) would absorb all of them.
While here: the ten hand-written table.lastUpdatedColumns.map(_.describe()).toSet == ... assertions in the two suites bypass the checkLastUpdatedColumns helper this PR adds, three of the Delta suite's updatedColumns tests duplicate the parent-suite tests added by the same PR (same DDL, SQL and expectation), writeUpdateLogEntry/writeUpdateWithMetadataLogEntry in RowLevelOperationSuiteBase have no callers, and the three ~17-line "overlay narrow row on base row" blocks in InMemoryRowLevelOperationTable could share one helper.
There was a problem hiding this comment.
Removed the duplicate helpers and the FromInfo fixture. The suites use one createAndInitTable(schema, data, tableProps) overload.
|
Summarizing the blocking items from my inline review. All four are correctness issues specific to the narrow path; the same statements work on the wide path.
The remaining inline comments (the |
|
Gentle ping, @anuragmantri . |
…erence unresolvable columns ### What changes were proposed in this pull request? When a V2 scan reports a `KeyGroupedPartitioning` or an `ordering` that references a column the relation can't resolve, the query used to fail with `Unable to resolve X given [...]`. This PR makes `V2ScanPartitioningAndOrdering` check the columns referenced by the reported partition keys and ordering (recursively, including nested transform arguments) against the relation before converting them. If any can't be resolved, including a missing nested field, an ambiguous name, or an empty name, the rule logs a warning naming the relation, the scan class, and those columns, and ignores the report. The partitioning falls back to `UnknownPartitioning`, and the ordering is dropped. This PR adds a `resolveRefOpt`, and makes `resolveRef` a thin wrapper around it. `resolveRefOpt` still throws for a missing nested field or an ambiguous name, so the rule treats those errors as unresolvable and includes the error condition in the warning. The `toCatalyst*` methods are unchanged. Keys whose columns all resolve take the existing conversion path unchanged, so an unsupported expression type (e.g. `id + 1`) or a nested transform whose function can't be loaded still fails the query as before. Because the whole report is checked before any key is converted, a report that also references an unresolvable column falls back with a warning instead, e.g. `[id + 1, identity(missing)]`. The Javadoc of `KeyGroupedPartitioning#keys()` and `SupportsReportOrdering#outputOrdering()` now says that Spark resolves the column references against the table columns, including columns pruned from the scan output, and ignores the whole report if any of them can't be resolved. The docs of `spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled` now mention that an ignored ordering can be replaced by one derived from the partition keys. ### Why are the changes needed? Found while reviewing SPARK-58111 ([PR #55518](#55518)). This is a general, pre-existing bug in how `V2ScanPartitioningAndOrdering` handles reported partitioning and ordering, not something introduced by #55518. ### Does this PR introduce _any_ user-facing change? Yes. A query against a data source whose scan reports a `KeyGroupedPartitioning` with a key that can't be resolved previously failed outright. It now succeeds, with partitioning degraded to `UnknownPartitioning` instead, and Spark logs a warning naming the relation, the scan class, and the columns that can't be resolved. The same applies to an ambiguous or empty column name. A report that combines an unsupported expression with an unresolvable column, e.g. `missing + 1` or `[id + 1, identity(missing)]`, used to fail with `_LEGACY_ERROR_TEMP_3054` and now falls back with a warning. An unsupported expression over resolvable columns still fails as before. If the scan keeps its partitioning but its reported ordering is ignored, Spark derives the ordering from the partition keys when `spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled` is on (the default). ### How was this patch tested? New tests in `V2ExpressionUtilsSuite`: - `resolveRefOpt` returns `None` for an unresolvable reference. - `resolveRefOpt` throws `FIELD_NOT_FOUND` for a missing nested field and `AMBIGUOUS_REFERENCE` for an ambiguous reference. New tests in `KeyGroupedPartitioningSuite`: - A join against a table whose scan reports unresolvable partition keys succeeds, plans a shuffle instead of a storage-partitioned join, and logs a warning naming the relation, the scan class, and only the unresolvable columns. The cases are: - a `bucket` key, a nested transform key, and a multi-key partitioning with one bad key; - duplicate references, and two distinct unresolvable columns; - a missing nested field; - an unsupported expression next to or over an unresolvable column (`[id + 1, identity(missing)]`, `missing + 1`); - a connector-defined transform that hides the reference from `children()`. Resolvable keys (`id`, case-insensitive `ID`, and a connector-defined transform with an extra reference only in `children()`) keep the partitioning and plan a storage-partitioned join without a shuffle. - A reported ordering that can't be resolved is ignored as a whole with a warning, and the scan still derives `id ASC` from its kept partitioning. The cases are a column that doesn't exist, one hidden from a connector-defined sort order's `children()`, an empty name, and a two-key ordering with one unresolvable key (`[id, missing]`). Resolvable orderings (`id`, case-insensitive `ID`, the metadata column `index`, and a connector-defined sort order with an extra reference only in `children()`) are kept without a warning. - A reported partition key of an unsupported expression type (`id + 1`), or a nested transform whose function can't be loaded (`f(g(id))`), still fails the query with `_LEGACY_ERROR_TEMP_3054`. New test in `DataSourceV2Suite`: - An unresolvable ordering reported through connector-defined `SortOrder`, `Transform`, and `NamedReference` classes (`OrderAndPartitionAwareDataSource` and its Java counterpart) is ignored with a warning, and the scan relation keeps no ordering. Also ran `V2ExpressionUtilsSuite`, `KeyGroupedPartitioningSuite`, `WriteDistributionAndOrderingSuite`, `DataSourceV2Suite`, `MergeSubplansSuite` (comment update only), `SQLConfSuite`, and `ProjectedOrderingAndPartitioningSuite` to confirm no regression in the partitioning/SPJ, ordering, and write paths this change touches. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Sonnet 5, Opus 5.5) Verified manually by me. Closes #58979 from anuragmantri/v2expr-opt-resolve-degrade. Authored-by: Anurag Mantripragada <amantripragada@apple.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
…erence unresolvable columns ### What changes were proposed in this pull request? When a V2 scan reports a `KeyGroupedPartitioning` or an `ordering` that references a column the relation can't resolve, the query used to fail with `Unable to resolve X given [...]`. This PR makes `V2ScanPartitioningAndOrdering` check the columns referenced by the reported partition keys and ordering (recursively, including nested transform arguments) against the relation before converting them. If any can't be resolved, including a missing nested field, an ambiguous name, or an empty name, the rule logs a warning naming the relation, the scan class, and those columns, and ignores the report. The partitioning falls back to `UnknownPartitioning`, and the ordering is dropped. This PR adds a `resolveRefOpt`, and makes `resolveRef` a thin wrapper around it. `resolveRefOpt` still throws for a missing nested field or an ambiguous name, so the rule treats those errors as unresolvable and includes the error condition in the warning. The `toCatalyst*` methods are unchanged. Keys whose columns all resolve take the existing conversion path unchanged, so an unsupported expression type (e.g. `id + 1`) or a nested transform whose function can't be loaded still fails the query as before. Because the whole report is checked before any key is converted, a report that also references an unresolvable column falls back with a warning instead, e.g. `[id + 1, identity(missing)]`. The Javadoc of `KeyGroupedPartitioning#keys()` and `SupportsReportOrdering#outputOrdering()` now says that Spark resolves the column references against the table columns, including columns pruned from the scan output, and ignores the whole report if any of them can't be resolved. The docs of `spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled` now mention that an ignored ordering can be replaced by one derived from the partition keys. ### Why are the changes needed? Found while reviewing SPARK-58111 ([PR #55518](#55518)). This is a general, pre-existing bug in how `V2ScanPartitioningAndOrdering` handles reported partitioning and ordering, not something introduced by #55518. ### Does this PR introduce _any_ user-facing change? Yes. A query against a data source whose scan reports a `KeyGroupedPartitioning` with a key that can't be resolved previously failed outright. It now succeeds, with partitioning degraded to `UnknownPartitioning` instead, and Spark logs a warning naming the relation, the scan class, and the columns that can't be resolved. The same applies to an ambiguous or empty column name. A report that combines an unsupported expression with an unresolvable column, e.g. `missing + 1` or `[id + 1, identity(missing)]`, used to fail with `_LEGACY_ERROR_TEMP_3054` and now falls back with a warning. An unsupported expression over resolvable columns still fails as before. If the scan keeps its partitioning but its reported ordering is ignored, Spark derives the ordering from the partition keys when `spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled` is on (the default). ### How was this patch tested? New tests in `V2ExpressionUtilsSuite`: - `resolveRefOpt` returns `None` for an unresolvable reference. - `resolveRefOpt` throws `FIELD_NOT_FOUND` for a missing nested field and `AMBIGUOUS_REFERENCE` for an ambiguous reference. New tests in `KeyGroupedPartitioningSuite`: - A join against a table whose scan reports unresolvable partition keys succeeds, plans a shuffle instead of a storage-partitioned join, and logs a warning naming the relation, the scan class, and only the unresolvable columns. The cases are: - a `bucket` key, a nested transform key, and a multi-key partitioning with one bad key; - duplicate references, and two distinct unresolvable columns; - a missing nested field; - an unsupported expression next to or over an unresolvable column (`[id + 1, identity(missing)]`, `missing + 1`); - a connector-defined transform that hides the reference from `children()`. Resolvable keys (`id`, case-insensitive `ID`, and a connector-defined transform with an extra reference only in `children()`) keep the partitioning and plan a storage-partitioned join without a shuffle. - A reported ordering that can't be resolved is ignored as a whole with a warning, and the scan still derives `id ASC` from its kept partitioning. The cases are a column that doesn't exist, one hidden from a connector-defined sort order's `children()`, an empty name, and a two-key ordering with one unresolvable key (`[id, missing]`). Resolvable orderings (`id`, case-insensitive `ID`, the metadata column `index`, and a connector-defined sort order with an extra reference only in `children()`) are kept without a warning. - A reported partition key of an unsupported expression type (`id + 1`), or a nested transform whose function can't be loaded (`f(g(id))`), still fails the query with `_LEGACY_ERROR_TEMP_3054`. New test in `DataSourceV2Suite`: - An unresolvable ordering reported through connector-defined `SortOrder`, `Transform`, and `NamedReference` classes (`OrderAndPartitionAwareDataSource` and its Java counterpart) is ignored with a warning, and the scan relation keeps no ordering. Also ran `V2ExpressionUtilsSuite`, `KeyGroupedPartitioningSuite`, `WriteDistributionAndOrderingSuite`, `DataSourceV2Suite`, `MergeSubplansSuite` (comment update only), `SQLConfSuite`, and `ProjectedOrderingAndPartitioningSuite` to confirm no regression in the partitioning/SPJ, ordering, and write paths this change touches. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Sonnet 5, Opus 5.5) Verified manually by me. Closes #58979 from anuragmantri/v2expr-opt-resolve-degrade. Authored-by: Anurag Mantripragada <amantripragada@apple.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 9de605f) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
|
I merged the following. Do you think you can rebase and update this PR, @anuragmantri ? |
|
Gentle ping, @anuragmantri . Please rebase your PR in order to keep healthy in sync with the Apache Spark |
9932f88 to
c871adc
Compare
- Rename the five SupportsColumnUpdates error conditions under a common COLUMN_UPDATE_ prefix so they sort and are discoverable together, instead of three unrelated prefixes (EMPTY_REQUIRED_DATA_ATTRIBUTES, REQUIRED_DATA_ATTRIBUTES_*, SPLIT_UPDATE_*). - Add test coverage for DataWriter#writeUpdate's default implementation, which throws DATA_SOURCE_WRITE_UPDATE_NOT_IMPLEMENTED when a connector mixes in SupportsColumnUpdates without overriding it. - Fix a handful of doc/comment nits in RewriteUpdateTable.scala (missing space, redundant blank line, missing punctuation) and use an import for UpdateSummary instead of the fully-qualified name.
Finding 21: connector attribute declarations resolve case-insensitively but resolveRefs keeps the declared spelling, and dedupAttrs keys on exprId, so a mis-cased declaration (e.g. PK instead of pk) silently replaced the table's real attribute name in the narrow scan/write schema, breaking column pruning with INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND. Map each resolved attribute back to the relation's own (by exprId) at all three resolution sites: resolveRequiredDataAttrs, resolveScanOnlyDataAttrs, and RowLevelWrite.projectedDataAttrs. Finding 22: SPLIT_UPDATE_ROW_ID_REASSIGNMENT's message advertised two remedies -- avoid reassigning row ID columns, or declare every table column in requiredDataAttributes() -- but validateNoRowIdReassignment only implemented the first. Skip the check when the declaration covers every column in the relation, since the REINSERT payload is then the full row with the new row-ID value and the DELETE half still carries the original row-ID via newLazyRowIdProjection, making reassignment safe. Written test-first: both fixtures/tests were confirmed to fail against the unmodified code before implementing each fix.
…comments Rework column-level UPDATE narrowing so that RewriteUpdateTable keeps master's read relation and plan builders and narrows only the write relation to requiredDataAttributes(). The rewrite orders the write query so the columns the write reads come first, and the ColumnPruning case for RowLevelWrite prunes only the columns after the last one the write reads. - Rename DataWriter.writeUpdate to writeColumnUpdate and LogicalWriteInfo.updateSchema to columnUpdateSchema, and the related error to DATA_SOURCE_WRITE_COLUMN_UPDATE_NOT_IMPLEMENTED. - Reject a write whose distribution or ordering references a column the column update does not read (COLUMN_UPDATE_UNDECLARED_WRITE_REQUIREMENT_COLUMNS), and a declared column that does not exist (COLUMN_UPDATE_UNKNOWN_REQUIRED_DATA_ATTRIBUTE). - Plan copy-on-write scans from the columns the write query reads for writes that deliver narrow rows; other writes read every column as before. - Build the runtime group filter attribute map from the original table. - Mark the new APIs @SInCE 4.4.0 and move the MiMa exclude to 4.4. - Fix Javadoc and error messages, and add tests for the rows and metadata the connector receives, cross-column assignments, case-sensitive analysis and MERGE updatedColumns().
c871adc to
2463813
Compare
anuragmantri
left a comment
anuragmantri
left a comment
There was a problem hiding this comment.
Thanks for the patience, @dongjoon-hyun, @peter-toth and @aokolnychyi. I rebased on master and also made significant design changes to simplify the several branches in the previous revision.
The main change is that RewriteUpdateTable keeps the existing read relation and plan builders and narrows only the write relation. The narrow read relation and the code built around it are gone, which also removes the cause of several findings from the last round. Here is the summary of other changes I made in this revision.
- Write side: the write relation is narrowed to
requiredDataAttributes(). The rewrite orders the write query so the columns the write reads come first, and a newColumnPruningcase forRowLevelWriteprunes the rest. - Read side: copy-on-write scans read only the columns the write query uses, and only for writes that deliver narrow rows. Other writes, including DELETE and MERGE, are planned as before.
- API: renamed to
writeColumnUpdateandcolumnUpdateSchema(), as @aokolnychyi suggested, and marked@since 4.4.0. - Tests: new tests check the exact rows and metadata the connector receives, and every error condition is checked with
checkError.
Since the design changed, would you take another look when you have a chance?
| val refs = checks.flatMap(_.predicate().references()).toSeq | ||
| V2ExpressionUtils.resolveRefs[AttributeReference](refs, relation) |
There was a problem hiding this comment.
With the read relation no longer narrowed, CHECK constraints resolve against the full write query, so a nested-field CHECK such as CHECK (s.c1 > 0) no longer needs special handling. Covered by CHECK constraint on a nested field narrows correctly in both suites.
| val duplicates = normalizedNames.groupBy(identity).collect { | ||
| case (_, occurrences) if occurrences.length > 1 => occurrences.head | ||
| }.toSeq |
There was a problem hiding this comment.
The duplicate error reports the spelling the connector declared, as the nested check does. ["PK", "Pk"] reports [PK] (column-update: case-only duplicate requiredDataAttributes throws AnalysisException).
| .collect { case a: AttributeReference => a } | ||
| .filter(relationSet.contains) | ||
| val checkConstraintAttrs = resolveCheckConstraintAttrs(relation) | ||
| dedupAttrs(connectorDataAttrs ++ extraRefs ++ checkConstraintAttrs) |
There was a problem hiding this comment.
This no longer applies: RewriteUpdateTable now keeps the existing read relation and narrows only the write relation, so no attribute in the connector's or the user's spelling can reach a scan. The declared attributes are still mapped back to the relation's own attributes by expr ID. I kept the regression tests for your example on both paths, and they now also check that the scan uses the table's spelling (undeclared, differently-cased columns in the RHS and WHERE clause are read with the table's own spelling).
| } | ||
| } | ||
|
|
||
| val rowIdValues = plan.output.filter(rowIdAttrSet.contains) |
There was a problem hiding this comment.
The rewrite now uses the existing projection builders, which project the original row ID under a separate name, so pk appears once. The row-ID reassignment tests with a CHECK constraint on pk still pass, for example column-update: CHECK constraint on a reassigned row-ID column is enforced.
| dataAttrs: Seq[AttributeReference], | ||
| metadataAttrs: Seq[AttributeReference], | ||
| rowIdAttrs: Seq[AttributeReference] = Nil): DataSourceV2Relation = { | ||
| val attrs = dedupAttrs(dataAttrs ++ rowIdAttrs ++ metadataAttrs) |
There was a problem hiding this comment.
This no longer applies: the read relation keeps every table column, so PushDownUtils.pruneColumns and toOutputAttrs see the same relation as for any other row-level operation. A scan builder without SupportsPushDownRequiredColumns reads every column and the write stays correct; the Javadoc now says the builder should implement it. The partial-pruning case is covered by scan that returns unrequested columns still writes narrow rows in both suites.
| * For updates on nested fields such as {@code SET t.s.c1 = -1} the connector should declare the | ||
| * root struct column {@code s} rather than any nested field. | ||
| * <p> | ||
| * This also covers columns needed only for planning, e.g. resolving the table's partitioning |
There was a problem hiding this comment.
SPARK-59721 (#58979) is merged, so I added the deferred test: column-update: partition source column not in requiredDataAttributes declares [pk, id] on a table partitioned by dep. It passes, and fails with Unable to resolve dep given [pk,id,_partition,index] if the #58979 change to V2ScanPartitioningAndOrdering is reverted. The description links the JIRA.
| * | ||
| * @since 4.3.0 | ||
| */ | ||
| default void writeUpdate(T metadata, T record) throws IOException { |
There was a problem hiding this comment.
Renamed to writeColumnUpdate(record) and writeColumnUpdate(metadata, record). The Javadoc says it receives both updated and copied rows of a group-based operation.
| * | ||
| * @since 4.3.0 | ||
| */ | ||
| default Optional<StructType> updateSchema() { |
There was a problem hiding this comment.
Renamed to columnUpdateSchema(), together with writeColumnUpdate.
| val matchedRowsPlan = Filter(cond, readRelation) | ||
| val rowDeltaPlan = if (operation.representUpdateAsDeleteAndInsert) { | ||
| buildDeletesAndInserts(matchedRowsPlan, assignments, rowIdAttrs) | ||
| val rowDeltaPlan = if (supportsColumnUpdate) { |
There was a problem hiding this comment.
Reworked. The column-update decisions are now in two places: resolveWriteAttrs in RewriteUpdateTable, a single three-arm match that returns the write relation's columns, and V2Writes.hasNarrowRows, which the write builders, the distribution check, copy-on-write scan planning and the exec routing use. The plan builders are the existing ones and have no column-update branches.
| val updatedAndRemainingRowsPlan = Union(updatedRowsPlan, remainingRowsPlan) | ||
|
|
||
| // build a plan to replace read groups in the table | ||
| val writeRelation = relation.copy(table = operationTable) |
There was a problem hiding this comment.
You were right. The PR now narrows only the write relation here, with the existing read relation and plan builders, and validates against relation.output as before. To keep the API MERGE-proof, the narrow layout is reported as columnUpdateSchema() and schema() covers only newly inserted rows, so MERGE can narrow its write relation later without API changes.
This branch has not been deployed
5 participants
What changes were proposed in this pull request?
For SPIP: SPARK-56599
This PR adds an opt-in DSv2 mix-in for connectors to receive narrow rows on column-level UPDATE, so connectors can avoid reading and writing full-table rows when only a subset of columns is being updated.
Public API additions (since 4.4.0):
SupportsColumnUpdates(@Experimental): a new mix-in onRowLevelOperation. Connectors implementrequiredDataAttributes(): NamedReference[]to declare every data column they need in the rows they receive: the columns Spark reports as updated, plus anything else needed for row lookup, routing, or planning (e.g. partition source columns and columns used by the write's required distribution and ordering). The declared columns must be top-level data columns that exist in the table, non-empty, without duplicates, and must cover every updated column. Violations are rejected at analysis time with dedicated error conditions.RowLevelOperationInfo.updatedColumns(): NamedReference[](@Experimental): the columns the UPDATE assigns a new value, at root-column granularity for nested field updates. Identity assignments such asSET a = aare excluded. Spark populates it before the connector's operation builder runs, so the connector can sizerequiredDataAttributes()accordingly. Other commands report an empty array.LogicalWriteInfo.columnUpdateSchema(): Optional<StructType>(@Evolving): the schema of updated, copied, and reinserted rows in a column-level update. When it is present,schema()covers only newly inserted rows, so it is empty for UPDATE.DataWriter.writeColumnUpdate(record)andwriteColumnUpdate(metadata, record)(@Evolving): the write channel for updated and copied rows of a group-based operation whencolumnUpdateSchema()is present. Spark calls the metadata overload when the operation declaresrequiredMetadataAttributes(), and the record-only overload otherwise. The record-only default throwsDATA_SOURCE_WRITE_COLUMN_UPDATE_NOT_IMPLEMENTED, and the metadata overload delegates to it. Delta-based operations receive narrow rows through the existingDeltaWriter.updateandDeltaWriter.reinsert.When an UPDATE runs against an operation that mixes in
SupportsColumnUpdates:RewriteUpdateTablekeeps the existing read relation and plan builders and narrows only the write relation torequiredDataAttributes(), so the rows the connector receives contain exactly the declared columns, in declared order. A connector that doesn't want to persist a declared column (e.g. a partition source column) must project it away before writing.ColumnPruningcase forRowLevelWriteprunes the columns after the last one the write reads. Unread columns stay in the analyzed query until then becauseResolveTableConstraintsresolves CHECK constraints by name against it. Copy-on-write scans are planned from the columns the write query reads (GroupBasedRowLevelOperationScanPlanning), and delta-based scans are pruned by the existing pushdown. Writes that do not deliver narrow rows, including DELETE and MERGE, are planned as before.COLUMN_UPDATE_UNDECLARED_WRITE_REQUIREMENT_COLUMNS, whether or not column pruning ran. Required metadata columns and, for delta-based operations, row ID columns are allowed. For operations that represent UPDATE as delete and insert, every row ID column must reach the reinserted row, and row ID columns cannot be reassigned unless every column is declared.RowLevelOperationRuntimeGroupFilteringbuilds its attribute map from the original table, as the write table may be narrower than the columns the condition references.MERGE and DELETE are not narrowed yet. The API is keyed on whether
columnUpdateSchema()is present rather than on the command, so MERGE can be narrowed later without API changes.Why are the changes needed?
Schema narrowing lets connectors request only the updated columns, enabling efficient column-level updates of wide tables.
Does this PR introduce any user-facing change?
Yes, new public DSv2 connector APIs:
RowLevelOperationmix-inSupportsColumnUpdates(requiredDataAttributes())RowLevelOperationInfo.updatedColumns()LogicalWriteInfo.columnUpdateSchema()DataWriter.writeColumnUpdate(...)COLUMN_UPDATE_EMPTY_REQUIRED_DATA_ATTRIBUTESCOLUMN_UPDATE_NESTED_REQUIRED_DATA_ATTRIBUTECOLUMN_UPDATE_DUPLICATE_REQUIRED_DATA_ATTRIBUTECOLUMN_UPDATE_METADATA_REQUIRED_DATA_ATTRIBUTECOLUMN_UPDATE_UNKNOWN_REQUIRED_DATA_ATTRIBUTECOLUMN_UPDATE_REQUIRED_DATA_ATTRIBUTES_MISSING_UPDATED_COLUMNSCOLUMN_UPDATE_SPLIT_ROW_ID_NOT_DECLAREDCOLUMN_UPDATE_SPLIT_ROW_ID_REASSIGNMENTCOLUMN_UPDATE_UNDECLARED_WRITE_REQUIREMENT_COLUMNSDATA_SOURCE_WRITE_COLUMN_UPDATE_NOT_IMPLEMENTED, raised by a data writer that does not overridewriteColumnUpdate(record)Connectors that do not mix in
SupportsColumnUpdatessee no change.RowLevelOperationInfo.updatedColumns()is a new abstract method on an interface that Spark implements and passes to connectors.How was this patch tested?
New suites:
DeltaBasedColumnUpdateTableSuite(extendsDeltaBasedUpdateTableSuiteBase) andGroupBasedColumnUpdateTableSuite(extendsUpdateTableSuiteBase), so the existing UPDATE tests also run against narrow writes. They also cover the rows and metadata the connector receives, every error condition above, scan schemas, CHECK constraints, subquery conditions, runtime group filtering, case-sensitive analysis, and DELETE and MERGE falling back to full-width rows.RowLevelWriteColumnPruningSuite: plan tests for theColumnPruningcase.Updated:
DeltaBasedUpdateTableSuiteBase(updatedColumns()tests),RowLevelOperationSuiteBase, and the in-memory test connectors (InMemoryRowLevelOperationTable,InMemoryBaseTable,txns.scala).Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5.5). I reviewed the generated code and tests.