Delta Kernel Draft PR #729

vaibhavk1992 · 2025-06-30T10:42:16Z

Important Read

Please ensure the GitHub issue is mentioned at the beginning of the PR

What is the purpose of the pull request

(For example: This pull request implements the sync for delta format.)

Brief change log

(for example:)

Fixed JSON parsing error when persisting state
Added unit tests for schema evolution

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end.
Added TestConversionController to verify the change.
Manually verified the change by running a job locally.

rahil-c · 2025-07-01T04:16:34Z

pom.xml

        <module>xtable-aws</module>
        <module>xtable-hive-metastore</module>
-        <module>xtable-service</module>
+<!--        <module>xtable-service</module>-->


This should be added back, any reason why you had to comment this out?

yes, xtable-service was giving build failures. As xtable-service is independent of the delta kernel changes. Can we just review the delta kernel changes as of now? I just want to get my changes validated once and anyhow the final version of delta kernel changes would have xtable-service module too.

@vaibhavk1992 I think if you rebase with latest main branch you shouldn't see those failures.

rahil-c · 2025-07-01T06:39:07Z

xtable-core/pom.xml

+        <dependency>
+            <groupId>io.delta</groupId>
+            <artifactId>delta-kernel-api</artifactId>
+            <version>4.0.0</version>


Can you add a properly in the root pom called <delta.kernel.version>4.0.0</delta.kernel.version>, instead of using the hardcoded value?

Also curious how you ended up choosing delta kernel version, is there some specific version that needs to align with delta lake version we have in the repo?

rahil-c · 2025-07-01T06:40:42Z

xtable-core/src/main/java/org/apache/xtable/delta/DeltaKernelConversionSourceProvider.java

+public class DeltaKernelConversionSourceProvider extends ConversionSourceProvider<Long> {
+  @Override
+  public DeltaKernelConversionSource getConversionSourceInstance(SourceTable sourceTable) {
+    Configuration hadoopConf = new Configuration();


Is there any reason why you are creating a new hadoopConf, can you instead use the hadoopConf from the parent class similar to what DeltaConversionSourceProvider does.

rahil-c · 2025-07-01T06:42:04Z

xtable-core/src/main/java/org/apache/xtable/delta/DeltaKernelSchemaExtractor.java

+    return INSTANCE;
+  }
+
+  public InternalSchema toInternalSchema_v2(StructType structType) {


I think we can just call this toInternalSchema, since its in its own distinct class right?

rahil-c · 2025-07-01T06:50:48Z

xtable-core/src/main/java/org/apache/xtable/delta/DeltaKernelTableExtractor.java

+      // Get schema from Delta Kernel's snapshot
+      io.delta.kernel.types.StructType schema = snapshot.getSchema();
+
+      System.out.println("Kernelschema: " + schema);


[nit] well need to remove these in final version of the pr.

rahil-c · 2025-07-01T06:51:38Z

xtable-core/src/main/java/org/apache/xtable/delta/DeltaSchemaExtractor.java

- * Converts between Delta and InternalTable schemas. Some items to be aware of:
- *
- * <ul>
- *   <li>Delta schemas are represented as Spark StructTypes which do not have enums so the enum


I think we can live this file as is right?

@vaibhavk1992 let's remove the changes to this file. They don't seem necessary

rahil-c · 2025-07-01T06:53:48Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionSource.java

+import org.apache.xtable.spi.extractor.ConversionSource;
+
+@Builder
+public class DeltaKernelConversionSource implements ConversionSource<Long> {


We will need full implementation of all the interface methods, otherwise this will fail during the table format sync. Can you refer to the impl for DeltaConversionSource for these methods?

all the methods have been added, just the commit backlog one is not fully resolved.

xtable-core/src/main/java/org/apache/xtable/delta/DeltaKernelDataFileExtractor.java

vinishjail97 · 2025-06-30T15:13:35Z

pom.xml

        <module>xtable-aws</module>
        <module>xtable-hive-metastore</module>
-        <module>xtable-service</module>
+<!--        <module>xtable-service</module>-->


why comment this?

vinishjail97 · 2025-07-28T14:51:12Z

xtable-core/src/test/resources/junit-platform.properties

 # limitations under the License.
 #
-junit.jupiter.execution.parallel.enabled=true
+junit.jupiter.execution.parallel.enabled=false


I have run the tests locally by setting this to true and they pass, can we revert this config and see the GH build?

vinishjail97

This is great progress @vaibhavk1992, added some comments.

vinishjail97 · 2025-07-28T14:53:01Z

pom.xml

        <module>xtable-aws</module>
        <module>xtable-hive-metastore</module>
-        <module>xtable-service</module>
+<!--        <module>xtable-service</module>-->


@vaibhavk1992 I think if you rebase with latest main branch you shouldn't see those failures.

vinishjail97 · 2025-07-28T14:53:21Z

pom.xml

                </executions>
                <configuration>
                    <skip>${skipUTs}</skip>
-                    <redirectTestOutputToFile>true</redirectTestOutputToFile>


Why revert this?

vinishjail97 · 2025-07-28T14:54:01Z

xtable-utilities/src/test/resources/my_config.yaml

  -
-    tableBasePath:  /Desktop/opensource/iceberg/warehouse/demo/nyc/taxis
-    tableDataPath:  /Desktop/opensource/iceberg/warehouse/demo/nyc/taxis/data
+    tableBasePath:  /Users/vaibhakumar/Desktop/opensource/iceberg/warehouse/demo/nyc/taxis


This one can be reverted too?

xtable-core/src/main/java/org/apache/xtable/delta/DeltaKernelActionsConverter.java

…itions,big fixes

vaibhavk1992 · 2025-09-08T15:30:10Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelIncrementalChangesState.java

+   * @param versionToStartFrom The version to start from.
+   */
+  @Builder
+  public DeltaKernelIncrementalChangesState(


@vinishjail97 This is where we need the implementation to happen for response conversions from kernel

vaibhavk1992 · 2025-09-08T15:31:42Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionSource.java

+
+    long versionNumberAtLastSyncInstant = snapshot.getVersion();
+    System.out.println("versionNumberAtLastSyncInstant: " + versionNumberAtLastSyncInstant);
+    //    resetState(0, engine,table);


This is the place which I have commented @vinishjail97 , call is happening inside this method

the-other-tim-brown · 2025-10-05T13:45:49Z

@vaibhavk1992 can you write up a summary of next steps and blockers for this feature?

vaibhavk1992 · 2025-10-05T18:51:54Z

Below is the summary of the difference between two schemes (Delta vs Kernel) also added the what remains the difference between two.

Comparison of Schema Responses: Delta Kernel vs Delta Log

This document outlines the differences in schema responses when using Delta Kernel and Delta Log APIs to retrieve changes in a Delta table. The comparison highlights the structure and format of the responses, providing insights into how the two approaches differ.

Delta Kernel Schema Response

When using the DeltaKernelIncrementalChangesState class to retrieve changes, the response is in the form of a row of columnar batch type. Each row is represented as an object of the io.delta.kernel.data.Row interface, which provides methods to access individual fields. The response is minimalistic and focuses on the raw data representation.

Sample Output

1 row is an object ==> io.delta.kernel.internal.data.ColumnarBatchRow@20c03e47

Key Characteristics

Row Representation: Each row is an instance of ColumnarBatchRow, which provides methods to access fields like getLong, getString, etc.
Minimal Metadata: The response contains only the essential fields (e.g., version, timestamp, commitInfo).
Raw Data: The schema is not enriched with additional metadata or actions; it is a direct representation of the data in the columnar batch.

Use Case

This format is suitable for low-level data processing where the focus is on performance and accessing raw data.

Delta Log Schema Response

When using the DeltaLog.getChanges method, the response is a tuple containing the version number and a list of actions. The actions include detailed metadata about the changes, such as CommitInfo and AddFile.
Sample Output from delta table changes
(2,
Vector(
CommitInfo(None, 2025-08-15 15:00:46.05, None, None, WRITE, Map(mode -> Append, partitionBy -> []), None, None, None, Some(1), Some(Serializable), Some(true), Some(Map(numFiles -> 1, numOutputRows -> 50, numOutputBytes -> 10226)), None, None, Some(Apache-Spark/3.4.2 Delta-Lake/2.4.0), Some(cf7b1472-4c68-4f89-aa97-c8f16512ecfc)),
AddFile(part-00000-e8eeadc8-4e26-46a7-8c61-0bf60e5e7ada-c000.snappy.parquet, Map(), 10226, 1755250246045, true, {"numRecords":50,"minValues":{"id":51,"firstName":"0WI98","lastName":"08VkW","gender":"Female","birthDate":"2013-02-16T21:18:43.000+05:30","level":"ERROR","date_field":"2025-08-15","timestamp_field":"2025-08-15T15:00:45.884+05:30","double_field":0.018425752795049544,"float_field":0.109567106,"long_field":-8844008067348082419,"record_field":{"nested_int":-2060061976}},"maxValues":{"id":100,"firstName":"xZnER","lastName":"ymLQw","gender":"Male","birthDate":"2023-08-07T15:06:55.000+05:30","level":"WARN","date_field":"2025-08-15","timestamp_field":"2025-08-15T15:00:45.885+05:30","double_field":0.9914942463945434,"float_field":0.9841615,"long_field":8775924211265194460,"record_field":{"nested_int":1923869027}},"nullCount":{"id":0,"firstName":25,"lastName":23,"gender":0,"birthDate":0,"level":0,"boolean_field":28,"date_field":24,"timestamp_field":26,"double_field":25,"float_field":28,"long_field":25,"binary_field":32,"primitive_map":22,"record_map":25,"primitive_list":28,"record_list":29,"record_field":{"nested_int":28}}}, null, null)
))
Key Characteristics

Rich Metadata: The response includes detailed metadata such as CommitInfo (e.g., operation type, timestamp, and user metadata) and AddFile (e.g., file path, size, and statistics).
Structured Actions: Each action is represented as a specific object (e.g., CommitInfo, AddFile), making it easier to interpret the changes.
Verbose Output: The response is more verbose, providing a comprehensive view of the changes.

This issue is currently in blocked state. I raised it with delta team quite a few time but no response over it.
https://delta-users.slack.com/archives/C04TRPG3LHZ/p1758730559515289

the-other-tim-brown · 2025-10-06T00:37:35Z

@vaibhavk1992 I pushed some changes in the latest commit to extract the add and remove files per version.

the-other-tim-brown · 2025-10-06T15:15:02Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelSchemaExtractor.java

+import org.apache.xtable.model.schema.InternalType;
+import org.apache.xtable.schema.SchemaUtils;
+
+public class DeltaKernelSchemaExtractor {


Can we have a unit test for this class?

I have checked in the current running test cases wherever we are using getCurrentSnapshot we are calling this class and the methods in it. So the checks are already in place. @the-other-tim-brown

I'm not asking for a functional test, I am asking for unit testing. We want high coverage with the testing of these key components.

@the-other-tim-brown I have added one of the unit test testConvertFromDeltaPartitionSinglePartition. Please confirm if this is looking fine, I will add others too.

the-other-tim-brown · 2025-10-06T15:15:19Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelPartitionExtractor.java

+
+@Log4j2
+@NoArgsConstructor(access = AccessLevel.PRIVATE)
+public class DeltaKernelPartitionExtractor {


Can we have a unit test for this class?

I have checked in the current running test cases wherever we are using getCurrentSnapshot and making InternalTable we are using this class and the methods in it. So the checks are already in place. @the-other-tim-brown

the-other-tim-brown · 2025-10-06T15:15:59Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelStatsExtractor.java

+ */
+@Log4j2
+@NoArgsConstructor(access = AccessLevel.PRIVATE)
+public class DeltaKernelStatsExtractor {


Similarly here, we could use some unit testing.

I have checked in the current running test cases under test testInsertsUpsertsAndDeletes we are using getTableChangeForCommit so we are using this class and the methods in it. So the checks are already in place. @the-other-tim-brown

the-other-tim-brown · 2025-10-07T14:09:44Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelActionsConverter.java

+    ;
+    //            String tableBasePath = snapshot.dataPath().toUri().toString();


Can this be cleaned up?

the-other-tim-brown · 2025-10-07T14:10:05Z

xtable-core/src/main/java/org/apache/xtable/delta/DeltaSchemaExtractor.java

- * Converts between Delta and InternalTable schemas. Some items to be aware of:
- *
- * <ul>
- *   <li>Delta schemas are represented as Spark StructTypes which do not have enums so the enum


@vaibhavk1992 let's remove the changes to this file. They don't seem necessary

the-other-tim-brown · 2025-10-07T14:10:47Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelDataFileExtractor.java

+      this.fields = schema.getFields();
+
+      StructType fullSchema = snapshot.getSchema(); // The full table schema
+      List<String> partitionColumns = snapshot.getPartitionColumnNames(); // List<String>


Remove the comment // List<String>?

the-other-tim-brown · 2025-10-07T14:11:01Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelDataFileExtractor.java

+      StructType fullSchema = snapshot.getSchema(); // The full table schema
+      List<String> partitionColumns = snapshot.getPartitionColumnNames(); // List<String>
+
+      List<StructField> partitionFields_strfld =


Nitpick: Use camelCase for variable names

the-other-tim-brown · 2025-10-07T14:11:59Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelDataFileExtractor.java

+      this.dataFilesIterator =
+          Collections
+              .emptyIterator(); // Initialize the dataFilesIterator by iterating over the scan files


This assignment does not seem necessary since we assign at line 139

the-other-tim-brown · 2025-10-07T14:13:07Z

xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionSource.java

+      Instant deltaCommitInstant = Instant.ofEpochMilli(snapshot.getTimestamp(engine));
+      return deltaCommitInstant.equals(instant) || deltaCommitInstant.isBefore(instant);
+    } catch (Exception e) {
+      System.err.println(


Use logging instead of print lines

vaibhavk1992 · 2025-10-13T17:56:21Z

xtable-core/src/test/java/org/apache/xtable/delta/ITDeltaKernelConversionSource.java

+  }
+
+  @Test
+  void testConvertFromDeltaPartitionSinglePartition() {


@the-other-tim-brown I have added this unit test, Please confirm if this looks fine I will add the others too.

vaibhavk1992 added 6 commits May 14, 2025 10:43

single commit

dfea6a4

adding delta kernel

b75bc7c

adding the test file

16134b3

adding workable code for iteration over data

3929e95

adding Kernel 4.0 code

c6379b5

adding the working code with xtable that check getcurrenttable

6deb5f7

vaibhavk1992 marked this pull request as draft June 30, 2025 10:42

vaibhavk1992 marked this pull request as ready for review June 30, 2025 15:40

vaibhavk1992 added 2 commits June 30, 2025 21:13

Merge branch 'main' into test-4

05d9984

adding the dependecies

c7ba4b9

rahil-c reviewed Jul 1, 2025

View reviewed changes

vaibhavk1992 added 4 commits July 19, 2025 22:15

adding getcurrentsnapshot code

0ff36a5

spotless fix

18ab9d6

spotless fix 2

e906091

spotless fix 2

e00241c

the-other-tim-brown reviewed Jul 20, 2025

View reviewed changes

xtable-core/src/main/java/org/apache/xtable/delta/DeltaKernelDataFileExtractor.java Outdated Show resolved Hide resolved

xtable-core/src/main/java/org/apache/xtable/delta/DeltaKernelDataFileExtractor.java Show resolved Hide resolved

fixed partitioned test case

3fdfd31

vinishjail97 reviewed Jul 28, 2025

View reviewed changes

setting junit parallel execution to true

e0102e3

vinishjail97 reviewed Jul 28, 2025

View reviewed changes

vaibhavk1992 added 4 commits August 5, 2025 10:08

testInsertsUpsertsAndDeletes test case addition,internal datatype add…

381722a

…itions,big fixes

added the fix for table basepath listing wrong paths

809bfe8

added the fix for table basepath listing wrong paths

40172f2

adding all tests

e0b7829

vaibhavk1992 added 2 commits August 27, 2025 22:45

adding refactored code

9ac022a

spotless fix

73f33b6

vaibhavk1992 commented Sep 8, 2025

View reviewed changes

fix change extraction

bee3e8a

the-other-tim-brown reviewed Oct 6, 2025

View reviewed changes

vaibhavk1992 added 2 commits October 7, 2025 18:04

adding the commitbacklog test cases changes

e75bb55

Merge branch 'apache:main' into test-4

21044af

the-other-tim-brown reviewed Oct 7, 2025

View reviewed changes

vaibhavk1992 added 2 commits October 13, 2025 22:43

adding a test case testConvertFromDeltaPartitionFormat

e212f52

adding a test case testConvertFromDeltaPartitionFormat

988cda1

vaibhavk1992 commented Oct 13, 2025

View reviewed changes

		;
		// String tableBasePath = snapshot.dataPath().toUri().toString();

Delta Kernel Draft PR #729

Are you sure you want to change the base?

Delta Kernel Draft PR #729

Uh oh!

Conversation

vaibhavk1992 commented Jun 30, 2025

Important Read

What is the purpose of the pull request

Brief change log

Verify this pull request

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahil-c Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vinishjail97 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

the-other-tim-brown commented Oct 5, 2025

Uh oh!

vaibhavk1992 commented Oct 5, 2025

Comparison of Schema Responses: Delta Kernel vs Delta Log

Delta Kernel Schema Response

Sample Output

Key Characteristics

Use Case

Delta Log Schema Response

Uh oh!

the-other-tim-brown commented Oct 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

rahil-c Jul 1, 2025 •

edited

Loading