[Kernel] Introduce IcebergCompatV3 #4595

KaiqiJinWow · 2025-05-21T16:57:25Z

Which Delta project/connector is this regarding?

Description

Introduce IcebergCompatV3 feature, which requires row tracking and column mapping to be enabled. We block the disabling or enabling IcebergCompatV3 on existing tables.

This PR is a stacked PR and based on #4731 and #4732

How was this patch tested?

Unit tests and e2e tests.

Does this PR introduce any user-facing changes?

.../java/io/delta/kernel/internal/icebergcompat/IcebergCompatV3MetadataValidatorAndUpdater.java

lzlfred · 2025-06-09T23:53:11Z

...in/java/io/delta/kernel/internal/icebergcompat/IcebergCompatMetadataValidatorAndUpdater.java

+        || dataType instanceof ArrayType
+        || dataType instanceof MapType
+        || dataType instanceof StructType
+        || dataType instanceof VariantType;


remove this

lzlfred · 2025-06-09T23:54:46Z

.../java/io/delta/kernel/internal/icebergcompat/IcebergCompatV3MetadataValidatorAndUpdater.java

+
+  @Override
+  List<TableFeature> requiredDependencyTableFeatures() {
+    return Stream.of(ICEBERG_COMPAT_V3_W_FEATURE, COLUMN_MAPPING_RW_FEATURE).collect(toList());


add row tracking here ?

lzlfred · 2025-06-10T00:02:21Z

.../java/io/delta/kernel/internal/icebergcompat/IcebergCompatV3MetadataValidatorAndUpdater.java

+                field -> {
+                  DataType dataType = field.getDataType();
+                  // IcebergCompatV3 supports variants and all the IcebergCompatV2 supported types
+                  return !isSupportedDataTypesForV2(dataType) && !(dataType instanceof VariantType);


make the new v3 types in a separate list to facilitate adding more later.

kernel/kernel-api/src/main/java/io/delta/kernel/Transaction.java

vkorukanti · 2025-06-10T18:21:17Z

kernel/kernel-api/src/main/java/io/delta/kernel/internal/TableConfig.java

+   * Compatibility V3 protocol.
+   *
+   * @see <a
+   *     href="https://github.com/delta-io/delta/blob/master/PROTOCOL.md#delta-iceberg-compatibility-v3">


seems like this link doesn't exists?

vkorukanti · 2025-06-10T18:22:47Z

...in/java/io/delta/kernel/internal/icebergcompat/IcebergCompatMetadataValidatorAndUpdater.java

+  }
+
+  protected static boolean isAdditionalSupportedDataTypesForV3(DataType dataType) {
+    return dataType instanceof VariantType;


also check isSupportedDataTypesForV2 and the new type supported?

.../java/io/delta/kernel/internal/icebergcompat/IcebergCompatV3MetadataValidatorAndUpdater.java

vkorukanti · 2025-06-10T18:29:01Z

.../java/io/delta/kernel/internal/icebergcompat/IcebergCompatV3MetadataValidatorAndUpdater.java

+  private static final IcebergCompatV3MetadataValidatorAndUpdater INSTANCE =
+      new IcebergCompatV3MetadataValidatorAndUpdater();
+
+  private static final IcebergCompatRequiredTablePropertyEnforcer ICEBERG_COMPAT_V3_CM_REQUIREMENT =


create a follow up task to define this check in the base class and reuse it. You can post that as the first PR precuros to this PR.

vkorukanti · 2025-06-10T18:29:45Z

.../java/io/delta/kernel/internal/icebergcompat/IcebergCompatV3MetadataValidatorAndUpdater.java

+              if (Boolean.parseBoolean(
+                  inputContext.newMetadata.getConfiguration().getOrDefault(prop, "false"))) {
+                throw DeltaErrors.icebergCompatIncompatibleVersionEnabled(
+                    INSTANCE.compatFeatureName(), prop);
+              }


can you check if you use the utility in TableConfig.PROP.fromMetadata here?

.../java/io/delta/kernel/internal/icebergcompat/IcebergCompatV3MetadataValidatorAndUpdater.java

vkorukanti · 2025-06-10T18:31:32Z

.../java/io/delta/kernel/internal/icebergcompat/IcebergCompatV3MetadataValidatorAndUpdater.java

+        }
+      };
+
+  private static final IcebergCompatCheck ICEBERG_COMPAT_V3_CHECK_HAS_ALLOWED_PARTITION_TYPES =


same comment here. create common check in the base class.

vkorukanti · 2025-06-10T18:31:50Z

.../java/io/delta/kernel/internal/icebergcompat/IcebergCompatV3MetadataValidatorAndUpdater.java

+        // this to allow checking the partition columns aren't changed
+      };
+
+  private static final IcebergCompatCheck ICEBERG_COMPAT_V3_CHECK_HAS_SUPPORTED_TYPE_WIDENING =


.../java/io/delta/kernel/internal/icebergcompat/IcebergCompatV3MetadataValidatorAndUpdater.java

KaiqiJinWow · 2025-06-12T23:40:17Z

...el/kernel-defaults/src/test/scala/io/delta/kernel/defaults/DeltaIcebergCompatBaseSuite.scala

+ * Base suite containing common test cases for Delta Iceberg compatibility features.
+ * This includes tests that apply to both V2 and V3 compatibility modes.
+ */
+trait DeltaIcebergCompatBaseSuite extends DeltaTableWriteSuiteBase with ColumnMappingSuiteBase {


Note to Reviewers: I move some common test cases from DeltaIcebergCompatV2Suite so that these test cases could be shared with both DeltaIcebergCompatV2Suite and DeltaIcebergCompatV3Suite. Since IcebergCompatV3 does not support being disabled or enabled for existing tables, so the existing tables related suites are kept in DeltaIcebergCompatV2Suite

.../java/io/delta/kernel/internal/icebergcompat/IcebergCompatV3MetadataValidatorAndUpdater.java

...el/kernel-defaults/src/test/scala/io/delta/kernel/defaults/DeltaIcebergCompatBaseSuite.scala

huan233usc · 2025-06-13T18:00:34Z

kernel/kernel-defaults/src/test/scala/io/delta/kernel/defaults/DeltaIcebergCompatV3Suite.scala

+    }
+  }
+
+  test("subsequent writes to icebergCompatV3 enabled tables doesn't update metadata") {


nit: this test case seems a bit over kill to me, what's the case for other iceberg compat?

We also have the same test case in IcebergCoompatV2, https://github.com/delta-io/delta/blob/master/kernel/kernel-defaults/src/test/scala/io/delta/kernel/defaults/DeltaIcebergCompatV2Suite.scala#L192-L224

kernel/kernel-api/src/main/java/io/delta/kernel/Transaction.java

vkorukanti · 2025-06-13T23:42:06Z

kernel/kernel-api/src/main/java/io/delta/kernel/internal/util/ColumnMapping.java

+        || Boolean.parseBoolean(
+            metadata
+                .getConfiguration()
+                .getOrDefault(TableConfig.ICEBERG_COMPAT_V3_ENABLED.getKey(), "false"))) {


please create a utility method in IcebergCompatMetadataValidatorAndUpdader and move this code there. Also use TableConfig.ICEBERG_COMPAT_V3_ENABLED.fromMetadata()

vkorukanti · 2025-06-13T23:52:38Z

.../delta/kernel/internal/icebergcompat/IcebergCompatMetadataValidatorAndUpdaterSuiteBase.scala

      s"icebergCompat$icebergCompatVersion does not support type widening present in table"))
  }

-  Seq("id", "name").foreach { existingCMMode =>


shouldn't this test be also added to IcebergWriterCompatV1MetadataValidatorAndUpdaterSuite?

or is it wrong in the first place where it is ran as part of the WriterCompatV2 suite, but it is always using icebergCompatV2?

Qn: does icebergCompatV3 can work with either modes or just id?

This test case previously existed only in the icebergCompatV2 suite. Since icebergCompatV2 supports both column mapping modes, whereas icebergWriteCompatV2 supports only the id mode, the test—which only verifies whether the column mapping modes can be enabled with icebergCompatV2—belongs specifically in the icebergCompatV2 suite. Therefore, it’s appropriate to keep it there rather than moving it to the shared base suite.

vkorukanti · 2025-06-14T00:02:57Z

.../io/delta/kernel/internal/icebergcompat/IcebergCompatV3MetadataValidatorAndUpdateSuite.scala

+    }
+  }
+
+  Seq("id", "name").foreach { existingCMMode =>


why can't move this to the base class?

Same reason #4595 (comment)

vkorukanti · 2025-06-14T00:04:03Z

.../io/delta/kernel/internal/icebergcompat/IcebergCompatV3MetadataValidatorAndUpdateSuite.scala

+            " not compatible with icebergCompatV3 requirements"))
+      }
+    }
+  }


what about the test where row tracking is enabled (if not enabled)?

also only icebergCompatV3 is enabled and not others compact versions.

are we getting the coverage for type widening?

also deletion vectors? do we need to test anything there?

type widening and deletion vectors(we mark isDeletionVectorsSupported as true here) are in base class

added case to verify row tracking is auto enabled and only icebergCompatV3 is enabled with not others compact versions.

vkorukanti

Looks great!

one test on variant support (not supported yet)
replace table case

kernel/kernel-api/src/test/scala/io/delta/kernel/TransactionSuite.scala

vkorukanti · 2025-06-16T22:09:56Z

kernel/kernel-api/src/main/java/io/delta/kernel/Transaction.java

          if (isIcebergCompatV2Enabled) {
            IcebergCompatV2MetadataValidatorAndUpdater.validateDataFileStatus(dataFileStatus);
+          } else if (isIcebergCompatV3Enabled) {
+            IcebergCompatV3MetadataValidatorAndUpdater.validateDataFileStatus(dataFileStatus);
          }


you can use isIcebergCompatEnabled and just call IcebergCompatMetadataValidatorAndUpdater.validateDataFileStatus

I would push back this as IcebergCompatMetadataValidatorAndUpdater.validateDataFileStatus also need a String compatFeatureName to better expose the error message, so it would not reduce the code imo.

.../io/delta/kernel/internal/icebergcompat/IcebergCompatV3MetadataValidatorAndUpdateSuite.scala

kernel/kernel-defaults/src/test/scala/io/delta/kernel/defaults/DeltaIcebergCompatV3Suite.scala

lzlfred

lgtm

KaiqiJinWow added 3 commits May 20, 2025 16:07

init

1f6001f

update

3283154

update

074b131

lzlfred reviewed May 22, 2025

View reviewed changes

KaiqiJinWow changed the title ~~[WIP][Kernel] IcebergCompatV3~~ [WIP][Kernel] Introduce IcebergCompatV3 Jun 9, 2025

Merge branch 'master' into stack/add_iceberg_compatV3

053c23c

murali-db mentioned this pull request Jun 9, 2025

Introduce WriterCompatV3 #4730

Open

5 tasks

KaiqiJinWow force-pushed the stack/add_iceberg_compatV3 branch from 018b936 to efa3249 Compare June 9, 2025 22:39

update

7e10891

KaiqiJinWow force-pushed the stack/add_iceberg_compatV3 branch from efa3249 to 7e10891 Compare June 9, 2025 22:39

KaiqiJinWow changed the title ~~[WIP][Kernel] Introduce IcebergCompatV3~~ 【Kernel] Introduce IcebergCompatV3 Jun 9, 2025

KaiqiJinWow changed the title ~~【Kernel] Introduce IcebergCompatV3~~ [Kernel] Introduce IcebergCompatV3 Jun 9, 2025

KaiqiJinWow requested a review from lzlfred June 9, 2025 22:56

move the data allow list

a343b1b

lzlfred reviewed Jun 9, 2025

View reviewed changes

lzlfred reviewed Jun 10, 2025

View reviewed changes

KaiqiJinWow added 2 commits June 10, 2025 10:19

add etsts

0233915

update

afd5b86

KaiqiJinWow self-assigned this Jun 10, 2025

KaiqiJinWow requested review from lzlfred, raveeram and vkorukanti June 10, 2025 17:33

raveeram-db requested review from allisonport-db, huan233usc and scottsand-db and removed request for lzlfred June 10, 2025 17:50

KaiqiJinWow commented Jun 10, 2025

View reviewed changes

kernel/kernel-api/src/main/java/io/delta/kernel/Transaction.java Show resolved Hide resolved

vkorukanti requested changes Jun 10, 2025

View reviewed changes

Merge branch 'master' into stack/add_iceberg_compatV3

0a01a32

KaiqiJinWow added 5 commits June 12, 2025 12:51

refactor test

a681913

trigger CIn

158d04b

move test

27e79fa

add e2e tests

4688720

small refactor

0514737

KaiqiJinWow commented Jun 12, 2025

View reviewed changes

KaiqiJinWow requested a review from vkorukanti June 12, 2025 23:40

vkorukanti reviewed Jun 13, 2025

View reviewed changes

.../java/io/delta/kernel/internal/icebergcompat/IcebergCompatV3MetadataValidatorAndUpdater.java Outdated Show resolved Hide resolved

address comment

36d5736

KaiqiJinWow requested a review from vkorukanti June 13, 2025 16:47

huan233usc reviewed Jun 13, 2025

View reviewed changes

...el/kernel-defaults/src/test/scala/io/delta/kernel/defaults/DeltaIcebergCompatBaseSuite.scala Outdated Show resolved Hide resolved

huan233usc reviewed Jun 13, 2025

View reviewed changes

...el/kernel-defaults/src/test/scala/io/delta/kernel/defaults/DeltaIcebergCompatBaseSuite.scala Outdated Show resolved Hide resolved

huan233usc reviewed Jun 13, 2025

View reviewed changes

address comment

b6bc89c

KaiqiJinWow requested a review from huan233usc June 13, 2025 20:34

huan233usc approved these changes Jun 13, 2025

View reviewed changes

vkorukanti requested changes Jun 14, 2025

View reviewed changes

address comment

84299dd

KaiqiJinWow requested a review from vkorukanti June 16, 2025 17:59

Merge branch 'master' into stack/add_iceberg_compatV3

a80d79a

KaiqiJinWow force-pushed the stack/add_iceberg_compatV3 branch from 1886ff0 to a80d79a Compare June 16, 2025 18:06

remove redudent func

f03342c

vkorukanti requested changes Jun 16, 2025

View reviewed changes

KaiqiJinWow added 2 commits June 16, 2025 16:42

add replace table test cases

2a93f08

add more tests

6966bfe

KaiqiJinWow requested a review from vkorukanti June 17, 2025 16:23

lzlfred approved these changes Jun 17, 2025

View reviewed changes

vkorukanti approved these changes Jun 17, 2025

View reviewed changes

vkorukanti merged commit 7e5b26f into delta-io:master Jun 17, 2025
21 checks passed

KaiqiJinWow mentioned this pull request Jul 3, 2025

[kernel] Implement Row Tracking support for IcebergCompatV3 #4879

Open

Uh oh!

[Kernel] Introduce IcebergCompatV3 #4595

[Kernel] Introduce IcebergCompatV3 #4595

Uh oh!

Conversation

KaiqiJinWow commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which Delta project/connector is this regarding?

Description

How was this patch tested?

Does this PR introduce any user-facing changes?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vkorukanti Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vkorukanti left a comment

Choose a reason for hiding this comment

Uh oh!

KaiqiJinWow commented May 21, 2025 •

edited

Loading

vkorukanti Jun 14, 2025 •

edited

Loading