Remove old Spark 4 shim sources and update tests

[gerashegalov]

Related to #14834.

Description

This PR is one reviewable layer in the unshim stack introduced by #15025. It removes old Spark 4 shim sources and updates tests for the shared helper layout. This is the final shim-source cleanup layer before the Delta/Iceberg follow-up.

Stack context

• Stack introduction: Add default common unshim packaging flow #15025
• Previous layer: Remove old Spark 3.5 shim sources #15037
• Next layer: Adapt Delta and Iceberg to unshimmed helpers #15027

Testing and validation notes

• This PR includes test updates for the shared helper layout and is also covered by the full-stack packaging/build validation described in Add default common unshim packaging flow #15025.
• The full split stack was verified to be tree-equivalent to the pre-split stack top.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Greptile Summary

This PR removes old Spark 4 shim source files from sql-plugin/src/main/spark{400,401,402,411} that were relocated to the new sql-plugin-shims module in an earlier stack layer, and adapts the remaining shim code and tests to the revised class layout.

• Deletions: SparkShimServiceProvider, DateTimeUtilsShims, OriginContextShim, SparkSessionUtils, TrampolineConnectShims, ShuffleManagerShims, ShuffleClientShims, and FileCommitProtocolShims are all removed from sql-plugin; their replacements exist in sql-plugin-shims or are unused.
• case class → class migrations: InvokeExprMeta, GpuGroupedPythonRunnerFactory, GpuUnboundedToUnboundedAggStages, and SecondPassAggResult are converted to regular classes; call sites and tests are updated accordingly.
• Functional addition: TimeAddShims in spark400db173 changes from an empty map to a full TimestampAddInterval GPU override (DB 17.3 and Spark 4.1+), and GpuLiteralShim is added to NullIntolerantShim.scala as a shared helper for Spark-4 shim stacks.

Confidence Score: 4/5

The structural cleanup is well-executed — all deleted service providers and utility shims have verified replacements in the new sql-plugin-shims module, and the case-class-to-class migrations are consistently applied across callers and tests.

The bulk of the change is safe bookkeeping: deletions with confirmed replacements, blank-line padding, and test adaptations. The two items worth a second look are (1) GpuLiteralShim.jsonFields not covering TimestampNTZType — affects only explain plan readability, not query correctness — and (2) the new TimestampAddInterval override omitting TIMESTAMP_NTZ from its TypeSig, which may be intentional parity with the old TimeAdd shims but is worth confirming.

sql-plugin/src/main/spark400/scala/com/nvidia/spark/rapids/shims/NullIntolerantShim.scala (GpuLiteralShim JSON serialization) and sql-plugin/src/main/spark400db173/scala/com/nvidia/spark/rapids/shims/TimeAddShims.scala (TIMESTAMP_NTZ coverage).

Important Files Changed

Filename	Overview
sql-plugin/src/main/spark400db173/scala/com/nvidia/spark/rapids/shims/TimeAddShims.scala	Replaces empty Map with a full TimestampAddInterval GPU override; only TypeSig.TIMESTAMP (LTZ) is accepted — TIMESTAMP_NTZ falls back to CPU, consistent with historical TimeAdd shims.
sql-plugin/src/main/spark400/scala/com/nvidia/spark/rapids/shims/NullIntolerantShim.scala	Adds GpuLiteralShim abstract class for JSON serialization; handles TimestampType but not TimestampNTZType — NTZ literals will emit raw microsecond longs in explain output.
sql-plugin/src/main/spark411/scala/org/apache/spark/sql/rapids/execution/python/shims/GpuGroupedPythonRunnerFactory.scala	Converted from case class to class + Serializable; argNames default removed — both callers already pass it explicitly, so no compilation breakage.
sql-plugin/src/main/spark400/scala/org/apache/spark/sql/rapids/shims/InvokeExprMeta.scala	Converted from case class to plain class; both Spark400PlusCommonShims and Spark400PlusDBShims updated to use explicit lambda constructors.
tests/src/test/scala/com/nvidia/spark/rapids/GpuSortRetrySuite.scala	Positional GpuSortEachBatchIterator arguments replaced with named parameters — readability improvement, no logic change.
tests/src/test/scala/com/nvidia/spark/rapids/window/GpuUnboundedToUnboundedAggWindowSuite.scala	Updated to use new constructor syntax (case→class) for GpuUnboundedToUnboundedAggStages and SecondPassAggResult.
sql-plugin/src/main/spark400/scala/com/nvidia/spark/rapids/shims/CudfUnsafeRow.scala	Adds blank-line padding before companion object for binary-dedupe alignment with pre-Spark-4 shims.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[PR #15038: Remove old Spark 4 shim sources] --> B[Deletions from sql-plugin]
    A --> C[In-place mutations in sql-plugin]
    A --> D[Test updates]

    B --> B1[SparkShimServiceProvider ×5\nspark400/401/402/400db173/411]
    B --> B2[Utility shims ×7\nDateTimeUtilsShims, OriginContextShim\nSparkSessionUtils, TrampolineConnectShims\nShuffleManagerShims, ShuffleClientShims\nFileCommitProtocolShims]

    B1 --> E[Replaced in sql-plugin-shims module]
    B2 --> F[Replaced in sql-plugin-shims\nor unused — safe to drop]

    C --> C1[InvokeExprMeta\ncase class → class\nExplicit lambda constructor in callers]
    C --> C2[GpuGroupedPythonRunnerFactory\ncase class → class + Serializable\nargNames default removed]
    C --> C3[TimeAddShims\nempty Map → TimestampAddInterval\nGPU override for 400db173 + 411]
    C --> C4[NullIntolerantShim\nAdds GpuLiteralShim helper]
    C --> C5[Blank-line padding\nBinary-dedupe alignment]

    D --> D1[GpuSortRetrySuite\nPositional → named params]
    D --> D2[GpuUnboundedToUnboundedAggWindowSuite\nnew-keyword constructors]

Loading

_{Reviews (1): Last reviewed commit: "Update tests for shared helper layout" | Re-trigger Greptile}

[greptile-apps]

Missing TimestampNTZType case in jsonFields

GpuLiteralShim.jsonFields handles TimestampType (LTZ) via DateTimeUtils.toJavaTimestamp, but TimestampNTZType falls through to the catch-all other branch, which calls .toString on the raw Long microsecond value. Any GpuLiteral with a TIMESTAMP_NTZ type will therefore serialize as a bare integer in explain plans and plan JSON output rather than a human-readable timestamp string. The parallel path for DateType already shows the expected pattern — TIMESTAMP_NTZ would need something like DateTimeUtils.microsToLocalDateTime instead of toJavaTimestamp (which would incorrectly apply a timezone offset).

[greptile-apps]

TIMESTAMP_NTZ not covered by the new override

ExprChecks.binaryProject is registered with TypeSig.TIMESTAMP only, so TimestampAddInterval on TIMESTAMP_NTZ columns will silently fall back to CPU. In Spark 4.1 / DB 17.3, TimestampAddInterval supports both timestamp variants — the CPU implementation is timezone-ignorant for TIMESTAMP_NTZ, making it straightforward to add TypeSig.TIMESTAMP_NTZ to both the result and the start input sig. Worth confirming whether this omission is intentional (matching old TimeAdd shim behaviour) or an oversight given the new override. Is the omission of TIMESTAMP_NTZ intentional here (preserving prior TimeAdd behaviour) or should TypeSig.TIMESTAMP_NTZ be added to both the result and start signatures?