[BUG] [AutoSparkUT][spark330] RapidsSQLQuerySuite.scala 'SPARK-33084: Add jar support Ivy URI in SQL' fails intermitently

[parthosa]

Describe the bug
Build: rapids_nightly-dev-github/1696

The spark 330 unit test job against cuda12 failed on worker w0. Maven reported 1 failed test out of 4987 in module rapids-4-spark-tests_2.12. The failing suite is RapidsParquetCompressionCodecPrecedenceSuite, test 'Rapids - Create parquet table with compression'. The log shows repeated IllegalArgumentException 'Codec [aa] is not available' thrown from GpuParquetFileFormat.tagGpuSupport via GpuOverrideUtil while applying GPU overrides. Other parallel shims in the same nightly build passed.

Error logs:

- Test `spark.sql.parquet.compression.codec` config
- [SPARK-21786] Test Acquiring 'compressionCodecClassName' for parquet in right order.
- Create parquet table with compression !!! IGNORED !!!
ERROR GpuOverrideUtil: Encountered an exception applying GPU overrides java.lang.IllegalArgumentException: Codec [aa] is not available. Available codecs are brotli, uncompressed, lz4, gzip, lzo, snappy, none, zstd.
java.lang.IllegalArgumentException: Codec [aa] is not available. Available codecs are brotli, uncompressed, lz4, gzip, lzo, snappy, none, zstd.
    at org.apache.spark.sql.execution.datasources.parquet.ParquetOptions.<init>(ParquetOptions.scala:59)
    at com.nvidia.spark.rapids.GpuParquetFileFormat$.tagGpuSupport(GpuParquetFileFormat.scala:54)
    at com.nvidia.spark.rapids.shims.CreateDataSourceTableAsSelectCommandMeta.tagSelfForGpuInternal(CreateDataSourceTableAsSelectCommandMetaShims.scala:64)
    ...
    at o.a.s.sql.execution.datasources.parquet.ParquetCompressionCodecPrecedenceSuite.checkCompressionCodec(ParquetCompressionCodecPrecedenceSuite.scala:95)
- Rapids - Create parquet table with compression
Run completed in 57 minutes, 19 seconds.
Total number of tests run: 4987
Suites: completed 252, aborted 0
Tests: succeeded 4986, failed 1, canceled 87, ignored 605, pending 0
*** 1 TEST FAILED ***
[INFO] RAPIDS Accelerator for Apache Spark Tests .......... FAILURE [59:32 min]
[ERROR] Failed to execute goal org.scalatest:scalatest-maven-plugin:2.0.2:test (test) on project rapids-4-spark-tests_2.12: There are test failures

Environment details

• Shim: spark330
• CUDA: cuda12
• JDK: 17 (OpenJDK 17.0.18)
• Scala: 2.12
• Failing module: rapids-4-spark-tests_2.12

[wjxiz1992]

Thanks for reporting this! let me take a look, it should be related to a recent data correctness fix.

[wjxiz1992]

Correction: I checked the RAPIDS test framework more carefully.

The inherited Spark test Create parquet table with compression is indeed excluded in RapidsTestSettings, and that is the test shown as !!! IGNORED !!! in the log:
https://github.com/NVIDIA/spark-rapids/blob/main/tests/src/test/spark330/scala/org/apache/spark/sql/rapids/utils/RapidsTestSettings.scala#L149

However, the failing test reported here is Rapids - Create parquet table with compression, which is the GPU-specific testRapids("Create parquet table with compression") replacement:
https://github.com/NVIDIA/spark-rapids/blob/main/tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsParquetCompressionCodecPrecedenceSuite.scala#L40

That replacement test is expected to run. testRapids prefixes the name with Rapids - , so the unprefixed .exclude("Create parquet table with compression", ...) only skips the inherited Spark test, not the RAPIDS replacement.

Also, the Codec [aa] is not available log likely comes from the inherited negative test Create table with unknown compression, which intentionally uses invalid codec aa. GpuOverrideUtil can log the exception while applying GPU overrides even if the test later intercepts the expected Spark exception, so that ERROR line alone does not prove the ignored test body ran.

So my earlier conclusion was incomplete: the original Spark test is skipped, but the RAPIDS replacement does run. To identify the real failure, we need the full ScalaTest failure stack for Rapids - Create parquet table with compression, not just the nearby GpuOverrideUtil ERROR log.

[wjxiz1992]

The Maven/ScalaTest summary does show *** 1 TEST FAILED ***, so there is still a real failed test in this run.

The confusing part is that the nearby Create parquet table with compression !!! IGNORED !!! line only shows the inherited Spark test was skipped, and the Codec [aa] is not available ERROR may come from the separate negative test that intentionally uses codec aa.

Could you please check the full ScalaTest failure section and share the actual failed test case name plus its failure stack? In particular, please look for the detailed block after the summary line or any *** FAILED *** entry, not only the nearby GpuOverrideUtil ERROR log. That should tell us whether the failing test is really Rapids - Create parquet table with compression or another test in the same suite.

[wjxiz1992]

Update: I found the real failed test case in the full log:

SPARK-33084: Add jar support Ivy URI in SQL

This is not the same test as the one discussed earlier:

• SPARK-33084: Add jar support Ivy URI in SQL -- jar contains udf class is excluded in RapidsTestSettings and replaced by a testRapids(...) version.
• SPARK-33084: Add jar support Ivy URI in SQL (without the -- jar contains udf class suffix) is a different inherited Spark SQLQuerySuite test and is currently not excluded/replaced.

That failing test exercises SQL ADD JAR ivy://... dependency resolution and then checks SparkContext.listJars(), e.g. spring-core, awaitility, hamcrest, and junit-jupiter jars. It does not look like a parquet compression or GPU data correctness test.

So the current failure should be investigated around the exact assertion/exception from this SPARK-33084 ADD JAR test: whether Ivy dependency resolution failed, a resolved jar name differed from the expected assertion, or SparkContext.listJars() did not contain the expected jars. The previous Codec [aa] is not available log is likely unrelated noise from a nearby negative test.

Could you please share the stack trace directly under:

- SPARK-33084: Add jar support Ivy URI in SQL *** FAILED ***

That should tell us which ADD JAR step or listJars() assertion is failing.

[wjxiz1992]

Thanks for the full failure block. This identifies the real root cause more clearly.

The failed test is:

SPARK-33084: Add jar support Ivy URI in SQL

The failure happens while Spark resolves an ADD JAR ivy://... dependency:

java.lang.RuntimeException: [download failed: ...]
  at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates
  at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies
  at org.apache.spark.sql.internal.SessionResourceLoader.resolveJars
  at org.apache.spark.sql.internal.SessionResourceLoader.addJar
  at org.apache.spark.sql.execution.command.AddJarsCommand.run

So this is not a Parquet compression failure, and it is not a GPU execution/data correctness failure. Spark fails before the query body can proceed, during Maven/Ivy artifact download.

The failed artifacts are mostly Hive/Hadoop transitive dependencies (hadoop-common, hive-llap-common, hive-service-rpc, parquet-hadoop-bundle, etc.), which points to the Spark SPARK-33084 ADD JAR ivy://... coverage. In Spark 3.3.0 this test exercises Hive-related Ivy coordinates such as hive-hcatalog-core / hive-contrib and then validates LIST JARS / SparkContext.listJars().

This looks like an external dependency resolution / repository availability issue in the CI environment, or a brittle inherited Spark test for the RAPIDS test job. It should be investigated separately from the earlier GpuOverrideUtil codec log. The Codec [aa] is not available message is unrelated nearby noise.

If this keeps recurring in RAPIDS spark330 tests, the practical fix is probably to exclude or adjust this inherited Spark test from the RAPIDS suite, because it validates Spark SQL ADD JAR ivy://... infrastructure rather than RAPIDS GPU behavior.

[wjxiz1992]

Additional data point: this did not reproduce in the next CI run, rapids_nightly-dev-github/1697.

Given that the failure in 1696 was a Maven/Ivy download failed error while resolving external artifacts, and 1697 passed without the same error, this now looks most likely to be a transient network / repository availability issue during dependency download rather than a deterministic RAPIDS test failure.