[SPARK-60068] Run unit tests on each matrix JDK and skip linters in build-test CI

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to run the build-test CI unit tests on each matrix JDK (21 / 25 / 26 / 27) instead of repeating the full build in every leg.

• Add an opt-in Gradle property, -PtestJavaVersion=<N>, to run the test tasks on JDK <N>. Compilation still uses the JDK 27 toolchain.
• Run ./gradlew test -PtestJavaVersion=${{ matrix.java-version }} instead of ./gradlew build in the build-test matrix. The linters keep running once in the lint job.

Why are the changes needed?

Currently, all matrix legs run the unit tests on the auto-provisioned JDK 27 toolchain, so Java 21, 25, and 26 are never tested. Also, every leg repeats the linters, which already run in the lint job.

Does this PR introduce any user-facing change?

No. The default behavior is unchanged without the new property.

How was this patch tested?

Pass the CIs. I also verified locally that all unit tests pass on JDK 21, 25, and 26 with the new property.

$ ./gradlew test -PtestJavaVersion=21

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

[dongjoon-hyun]

Since this is a CI-only patch, let me merge this to improve JDK coverage.

[dongjoon-hyun]

Merged to main

[peter-toth]

Late LGTM.

[dongjoon-hyun]

Thank you always, @peter-toth !