[FEATURE] Add Apache Spark 4.0 support

[lawofcycles]

Is your feature request related to a problem?

Apache Spark 4.0.0 was released in December 2024 with breaking changes (Scala 2.12 dropped, Java 17+ required, javax→jakarta migration). Users who upgrade to Spark 4.0 cannot use opensearch-hadoop as it currently only supports up to Spark 3.x.

What solution would you like?

Add a new opensearch-spark-40 module following the existing pattern of opensearch-spark-20 and opensearch-spark-30

• New module: spark/sql-40/
• Scala 2.13 support (Spark 4.0 requirement)
• Java 17+ runtime
• Updated dependencies (jakarta.servlet, jakarta.xml.bind, etc.)
• CI workflow for Spark 4.0 builds

What alternatives have you considered?

Users can continue using Spark 2.x or 3.x with the existing modules.

Do you have any additional context?

Key Spark 4.0 changes

• SPARK-45314: Drop Scala 2.12
• SPARK-45315: Drop JDK 8/11
• Servlet API: javax.servlet → jakarta.servlet

Reference: Spark 4.0.0 Release Notes

[andrross]

Catch All Triage - 1 2

Thanks @lawofcycles. Any interest in putting together a PR?

[lawofcycles]

@andrross Yes, I'll raise PR.

[kylejablon]

+1 would love to hear if there are any updates on this

[lawofcycles]

I opened PR #684

[giladbu]

thanks

[lawofcycles]

@kylejablon @giladbu
If you find anything or have any feedback as you use it, please feel free to share.