Support Spark 4.x

[pang-wu]

This PR adapt raydp with Spark 4.x but leave the following work for future improvement:

1. Support tensorflow 2.16+ (see https://keras.io/getting_started/#tensorflow--keras-2-backwards-compatibility) and numpy 2.x
2. Support python 3.12 - we deprecated Python 3.9 because it is no longer supported by Spark. Need to modernize python build system.
3. Deprecate Ray AIR.

To make the tests pass, the PR is based on #458. Once PR#458 is merged this PR should rebase again.

[rexminnis]

Thanks for putting this together — the CommandLineUtilsBridge pattern and the SparkSubmit rework are clean solutions to the cross-version API drift. A few things I noticed:

1. Bug: spark340/SparkSqlUtils.toArrowRDD has infinite recursion (see inline comment)
2. Java target: maven.compiler.source is still 1.8 — worth bumping to 17?
3. Spark version: spark410.version targets 4.1.0 — consider 4.1.1 (current release)

Happy to help with testing or any of the shim work. I have a working Spark 4.1.1 setup locally and have been validating the Arrow conversion paths end-to-end.

[rexminnis]

Bug — this is infinitely recursive. SparkSqlUtils.toArrowRDD calls itself:

def toArrowRDD(dataFrame: DataFrame, sparkSession: SparkSession): RDD[Array[Byte]] = {
    SparkSqlUtils.toArrowRDD(dataFrame, dataFrame.sparkSession)
}

This will StackOverflowError at runtime. Should be dataFrame.toArrowBatchRdd like the other shims (spark322, spark330, spark350).

[pang-wu]

good catch, fixed.

[rexminnis]

The Maven compiler source/target is still 1.8. Since Spark 4.x requires Java 17 at runtime and CI now uses JDK 17, should we bump the compile target to 17 as well? This would catch any bytecode-level incompatibilities at compile time rather than runtime.

[rexminnis]

Minor: spark410.version is 4.1.0 — worth bumping to 4.1.1 (current release)? The SparkShimProvider already covers it at runtime, but compiling against the latest patch would catch any API changes at build time.

[pang-wu]

I would keep it 4.1.0 -- the idea is we should support the minimum API from the initial version otherwise the lib might introduce broken changes between Spark's patch versions (Spark is supposed to be backward compatible on patch versions)