[GH-1918] Spark 4 support

[Kimahriman]

Did you read the Contributor Guide?

• Yes, I have read the Contributor Rules and Contributor Development Guide

Is this PR related to a ticket?

• Yes, and the PR name follows the format [GH-XXX] my subject.

What changes were proposed in this PR?

Add support for Spark 4.

This required several updates:

• A new profile is added for Spark 4
• The spark/common module has source directories for Spark 3 and 4 respectively. I had to do it in the common module because things in the common module depend on the version specific shims. The main breaking changes that required this are:
  • Column objects are no longer wrappers around Expression objects, but a new ColumnNode construct for Spark Connect support. Supporting the expression wrapping requires a different setup. Initially I started working on this through reflection, but this got pretty messy and this will require different artifacts anyway, so I added the conditional source directories.
  • Creating a DataFrame from an RDD has to use the new location of the "classic" DataFrame class.
  • The NullIntolerant trait no longer exists, instead it's a an overridable function on an expression
• jt-jiffle-language and it's antlr dependency have to be shaded into the common module for Spark 4 to work. This is because in antlr 4.10 there was some internal version bump such that dependencies compiled with antlr < 4.10 can't run at runtime with >= 4.10. I think jt-jiffle-language has an Apache license so I think this is ok? Currently it's a provided dependency that comes with the external geotools-wrapper. But need some verification here or thoughts on any alternative approach.
• I copied the spark-3.5 module as is to spark-4.0. The only changes I had to make were to the new Arrow UDF stuff that was added recently. Could these also just be moved as conditional source directories in spark/common?
• DBSCAN tests are ignored on Spark 4 because the current graphframes dependency does not support Spark 4. I've been messing around with getting graphframes updated as well.

How was this patch tested?

Existing UTs.

Did this PR include necessary documentation updates?

• No, this PR does not affect any public API so no need to change the documentation.

Maybe supported versions need to change? Haven't looked at the docs yet.

[Kimahriman]

This was the only way I could figure out to get the scala docs to be aware of the additional source directory

[Kimahriman]

This arrow eval is the only thing I had to update from the spark-3.5 module to the spark-4.0 module due to some API changes. It looks like starting in 4.1 they added support for UDTs in arrow UDFs

[Kontinuation]

Can we shade it in geotools-wrapper so that no dependency reduced pom will be generated when building sedona-common? @jiayuasu

[Kimahriman]

It definitely needs to be shaded locally for the tests to work. I'm not 100% sure if the release could just be shaded into geotools-wrapper or not. My concern was if you somehow have jiffle as a separate dependency, those classes would be used with the provided antlr and not the relocated antlr dependency