Adapt Delta and Iceberg to unshimmed helpers

[gerashegalov]

Related to #14834.

Description

This PR is one reviewable layer in the unshim stack introduced by #15025. It adapts Delta and Iceberg integration points to the unshimmed helper layout after the core SQL plugin helper movement and cleanup layers are in place.

Stack context

• Stack introduction: Add default common unshim packaging flow #15025
• Previous layer: Remove old Spark 4 shim sources and update tests #15038
• Next layer: Update UDF compiler for shared helper layout #15028

Testing and validation notes

• Covered by the full-stack packaging/build validation described in Add default common unshim packaging flow #15025 and the existing Delta/Iceberg test coverage for the affected integrations.
• The full split stack was verified to be tree-equivalent to the pre-split stack top.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Covered by the validation notes in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Greptile Summary

This PR adapts Delta Lake and Iceberg integration points to the unshimmed helper layout, primarily by mechanically converting case class definitions to plain class with explicit val fields across both modules. It also adds reflection-based accessors for abort, commitOperation, and taskCommitFiles to GpuSparkWriteAccess, and threads the corresponding wrapper methods through GpuSparkWrite.

• All case class → class conversions add explicit val accessors; classes that provably need to cross JVM serialization boundaries (GpuIncrementMetricMeta, DeltaParquetChunkedReader, DeltaParquetTableReader) correctly add with Serializable.
• DeleteWriteContext.close() replaces the productIterator-based close with an explicit Seq[AutoCloseable]; a manual copy() with default-parameter semantics is added to preserve existing call sites.
• New invokeMethod/findMethod helpers in GpuSparkWriteAccess follow the existing readField reflection pattern with proper exception unwrapping and class-hierarchy traversal.

Confidence Score: 4/5

The conversion is largely mechanical and correct, but GpuDeltaFileStatistics lost its implicit Serializable from the case-class conversion without getting an explicit replacement, unlike other converted classes in the same PR.

Almost all converted types are either provably local (e.g. DeltaWriteV1Config, StartTransactionArg) or explicitly gained with Serializable (e.g. GpuIncrementMetricMeta, DeltaParquetChunkedReader). GpuDeltaFileStatistics is the outlier: it carries write stats back from executors to the driver, and its WriteTaskStats supertrait does not extend Serializable, meaning the now-absent case-class auto-Serializable was load-bearing. Everything else — including the new reflection accessors in GpuSparkWriteAccess, the manual copy() on DeleteWriteContext, and the DeltaBatchExtraInfo constructor-call expansion — looks correct.

delta-lake/common/src/main/scala/com/nvidia/spark/rapids/delta/GpuDeltaTaskStatisticsTracker.scala — verify GpuDeltaFileStatistics serialization requirement and add with Serializable if confirmed.

Important Files Changed

Filename	Overview
delta-lake/common/src/main/scala/com/nvidia/spark/rapids/delta/GpuDeltaTaskStatisticsTracker.scala	Converts GpuDeltaFileStatistics from case class to plain class; loses auto-Serializable without explicitly adding with Serializable, unlike peer converted classes in this PR.
delta-lake/common/src/main/delta-33x-41x/scala/com/nvidia/spark/rapids/delta/common/GpuDeltaParquetFileFormatBase2.scala	Mechanical case-class-to-class conversion for several inner types; DeltaBatchExtraInfo.loadedDVResults and DeltaParquetHostMemoryEmptyMetaData.allPartValues lose their defaults and are now explicit at every call site. Manual consumeHeadBuffer reconstruction and withLoadedDVResults correctly pass all fields.
iceberg/common/src/main/java/org/apache/iceberg/spark/source/GpuSparkWriteAccess.java	Adds reflection-based abort, commitOperation, taskCommitFiles accessors; invokeMethod/findMethod helpers follow existing readField pattern, with proper hierarchy traversal and exception unwrapping.
iceberg/common/src/main/scala/org/apache/iceberg/spark/source/GpuSparkPositionDeltaWrite.scala	Converts DeleteWriteContext from case class to plain class; explicit copy() method correctly mirrors case-class semantics, and close() now uses an explicit Seq[AutoCloseable] instead of productIterator.
delta-lake/common/src/main/delta-33x-41x/scala/com/nvidia/spark/rapids/delta/common/DeltaProviderBase.scala	Converts GpuIncrementMetricMeta from case class to plain class; correctly adds with Serializable since ExprMeta subclasses can be serialized as part of query plans.
iceberg/common/src/main/scala/org/apache/iceberg/spark/source/GpuSparkWrite.scala	Adds abort, files, and commitOperation wrapper methods delegating to GpuSparkWriteAccess; correct delegation pattern.
delta-lake/common/src/main/databricks/scala/com/nvidia/spark/rapids/delta/DatabricksDeltaProviderBase.scala	Local-only DeltaWriteV1Config converted from case class to plain class; correct accessor semantics maintained with explicit val fields.
delta-lake/common/src/main/delta-io/scala/org/apache/spark/sql/delta/rapids/DeltaRuntimeShim.scala	Converts StartTransactionArg from case class to plain class; function-scoped argument object, no serialization required, correct.
iceberg/common/src/main/scala/com/nvidia/spark/rapids/iceberg/GpuIcebergPartitioner.scala	Converts ColumnarBatchWithPartition from case class to plain class; AutoCloseable semantics preserved.

Sequence Diagram

sequenceDiagram
    participant Executor
    participant GpuDeltaTaskStatisticsTracker
    participant GpuDeltaFileStatistics
    participant Spark Task Result
    participant Driver
    participant GpuDeltaJobStatisticsTracker

    Executor->>GpuDeltaTaskStatisticsTracker: getFinalStats()
    GpuDeltaTaskStatisticsTracker->>GpuDeltaFileStatistics: new GpuDeltaFileStatistics(results.toMap)
    GpuDeltaFileStatistics-->>Executor: stats object
    Executor->>Spark Task Result: serialize(WriteTaskStats)
    Note over GpuDeltaFileStatistics,Spark Task Result: Missing Serializable — was auto-provided by case class
    Spark Task Result->>Driver: send serialized task result
    Driver->>GpuDeltaJobStatisticsTracker: processStats(Seq[WriteTaskStats])

Loading

_{Reviews (1): Last reviewed commit: "Adapt Delta and Iceberg to unshimmed hel..." | Re-trigger Greptile}

[greptile-apps]

Missing Serializable after case-class-to-class conversion

Scala case class automatically mixes in java.io.Serializable. Converting to a plain class removes that. WriteTaskStats is a plain sealed trait (no Serializable), so GpuDeltaFileStatistics is no longer serializable. Spark collects WriteTaskStats from executors to the driver as part of the task result, so a NotSerializableException would surface during any Delta write that uses stats collection. Every other converted class in this PR that needs to cross a serialization boundary (GpuIncrementMetricMeta, DeltaParquetChunkedReader, DeltaParquetTableReader) explicitly adds with Serializable. Delta Lake's own DeltaFileStatistics remains a case class for the same reason.

Suggested change

	class GpuDeltaFileStatistics(val stats: Map[String, String]) extends WriteTaskStats
	class GpuDeltaFileStatistics(val stats: Map[String, String]) extends WriteTaskStats with Serializable