[BUG] Java test fails on GB100 and CUDA 12.8 (branch-25.02)

[ttnghia]

When running Java test (branch 25.02) on GB100 and CUDA 12.8, the following tests failed:

[ERROR] Errors: 
[ERROR]   TableTest.testReadAvro:1788 » CudaFatal reduce failed to synchronize: cudaErro...
[INFO] 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

[ERROR] Errors: 
[ERROR]   TableTest.testReadAvroFromDataSource:1808 » CudaFatal reduce failed to synchronize: cudaErro...
[INFO] 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

[ERROR] Errors: 
[ERROR]   TableTest.testReadAvroFull:1848 » CudaFatal reduce failed to synchronize: cuda...
[INFO] 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

[ERROR] Errors: 
[ERROR]   TableTest.testReadAvroBuffer:1827 » CudaFatal reduce failed to synchronize: cu...
[INFO] 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

I tried to run these individual tests one at a time to make sure such failures are not affected by the previous test and found this:

Note that I didn't test with other cudf branches nor other CUDA versions nor other similar GPU such as GB200.

[ttnghia]

The failure of readAvro is due to crashing when calling stream.synchronize() after nvcompBatchedSnappyDecompressAsync (nvcomp 4.2.0.11). Before calling to it, I also have stream.synchronize(). I intentionally inserted stream.synchronize() everywhere to debug, and caught this.

When replacing nvcomp 4.2 by nvcomp 4.1:

[INFO] Results:
[INFO] 
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] 

So definitely it's nvcomp 4.2 issue.

[ttnghia]

Here is a simple cudf test to reproduce the issue:

#include <cudf_test/base_fixture.hpp>
#include <cudf_test/testing_main.hpp>
#include <cudf_test/debug_utilities.hpp>

#include <cudf/io/avro.hpp>
#include <cudf/utilities/default_stream.hpp>
#include <cudf/utilities/memory_resource.hpp>

#include <rmm/device_buffer.hpp>
#include <rmm/device_uvector.hpp>

struct Test : public cudf::test::BaseFixture { };

TEST_F(Test, Decompression)
{
  auto stream = cudf::get_default_stream();
  auto mr = cudf::get_current_device_resource_ref();
  auto source = cudf::io::source_info("alltypes_plain.avro");

  cudf::io::avro_reader_options opts = cudf::io::avro_reader_options::builder(source)
                                           .columns({"bool_col", "int_col", "timestamp_col"})
                                           .build();
  auto result = cudf::io::read_avro(opts, stream, mr);
  cudf::test::print(result.tbl->get_column(0).view());
}

Where the file alltypes_plain.avro is copied into the working folder from java/src/test/resources/alltypes_plain.avro.

To reproduce, one can use docker with image from spark-rapids-jni: https://github.com/NVIDIA/spark-rapids-jni/blob/branch-25.04/ci/Dockerfile, but with the CUDA_VERSION changed to 12.8. After running a container from it:

source scl_source enable gcc-toolset-11

cd cudf
cd cpp
mkdir build
cd build
cmake .. -G Ninja    -DBUILD_BENCHMARKS=OFF     -DBUILD_SHARED_LIBS=OFF     -DBUILD_TESTS=ON     -DCMAKE_BUILD_TYPE=Release     -DCMAKE_CUDA_ARCHITECTURES=100     -DCUDA_STATIC_CUFILE=ON     -DCUDA_STATIC_RUNTIME=ON     -DCUDF_USE_PER_THREAD_DEFAULT_STREAM=ON     -DCUDF_KVIKIO_REMOTE_IO=OFF     -DCUDF_LARGE_STRINGS_DISABLED=ON -Dcuda.version=cuda12
ninja