Skip to content

Commit dbd2599

Browse files
NvTimLiuNvTimLiu
authored
Merge branch-24.12 into main (#11951)
Change version to 24.12.1 Note: merge this PR with **Create a merge commit to merge**
2 parents 9be8079 + 2bbc361 commit dbd2599

File tree

68 files changed

+302
-204
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

68 files changed

+302
-204
lines changed

‎CHANGELOG.md

+8-3
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# Change log
2-
Generated on 2024-12-16
2+
Generated on 2025-01-10
33

44
## Release 24.12
55

@@ -24,7 +24,10 @@ Generated on 2024-12-16
2424
### Bugs Fixed
2525
|||
2626
|:---|:---|
27+
|[#11842](https://github.com/NVIDIA/spark-rapids/issues/11842)|[BUG] udf-examples-native case failed core dump|
28+
|[#11718](https://github.com/NVIDIA/spark-rapids/issues/11718)|[BUG] update date/time APIs in CUDF java to avoid deprecated functions|
2729
|[#10907](https://github.com/NVIDIA/spark-rapids/issues/10907)|from_json function parses a column containing an empty array, throws an exception.|
30+
|[#11807](https://github.com/NVIDIA/spark-rapids/issues/11807)|[BUG] mismatched cpu and gpu result in test_lead_lag_for_structs_with_arrays intermittently|
2831
|[#11793](https://github.com/NVIDIA/spark-rapids/issues/11793)|[BUG] "Time in Heuristic" should not include previous operator's compute time|
2932
|[#11798](https://github.com/NVIDIA/spark-rapids/issues/11798)|[BUG] mismatch CPU and GPU result in test_months_between_first_day[DATAGEN_SEED=1733006411, TZ=Africa/Casablanca]|
3033
|[#11790](https://github.com/NVIDIA/spark-rapids/issues/11790)|[BUG] test_hash_* failed "java.util.NoSuchElementException: head of empty list" or "Too many times of repartition, may hit a bug?"|
@@ -72,6 +75,10 @@ Generated on 2024-12-16
7275
### PRs
7376
|||
7477
|:---|:---|
78+
|[#11947](https://github.com/NVIDIA/spark-rapids/pull/11947)|Update version to 24.12.1-SNAPSHOT [skip ci]|
79+
|[#11943](https://github.com/NVIDIA/spark-rapids/pull/11943)|Update rapids JNI dependency to 24.12.1|
80+
|[#11944](https://github.com/NVIDIA/spark-rapids/pull/11944)|Update download page for 24.12.1 hot fix release [skip ci]|
81+
|[#11876](https://github.com/NVIDIA/spark-rapids/pull/11876)|Update latest changelog [skip ci]|
7582
|[#11874](https://github.com/NVIDIA/spark-rapids/pull/11874)|Remove 350db143 shim's build [skip ci]|
7683
|[#11851](https://github.com/NVIDIA/spark-rapids/pull/11851)|Update latest changelog [skip ci]|
7784
|[#11849](https://github.com/NVIDIA/spark-rapids/pull/11849)|Update rapids JNI and private dependency to 24.12.0|
@@ -236,8 +243,6 @@ Generated on 2024-12-16
236243
|[#11183](https://github.com/NVIDIA/spark-rapids/issues/11183)|[BUG] Failed to split an empty string with error "ai.rapids.cudf.CudfException: parallel_for failed: cudaErrorInvalidDevice: invalid device ordinal"|
237244
|[#11008](https://github.com/NVIDIA/spark-rapids/issues/11008)|Fix tests failures in ast_test.py|
238245
|[#11265](https://github.com/NVIDIA/spark-rapids/issues/11265)|[BUG] segfaults seen in cuDF after prefetch calls intermittently|
239-
|[#11025](https://github.com/NVIDIA/spark-rapids/issues/11025)|Fix tests failures in date_time_test.py|
240-
|[#11065](https://github.com/NVIDIA/spark-rapids/issues/11065)|[BUG] Spark Connect Server (3.5.1) Can Not Running Correctly|
241246

242247
### PRs
243248
|||

‎CONTRIBUTING.md

+4-4
Original file line numberDiff line numberDiff line change
@@ -127,15 +127,15 @@ mvn -pl dist -PnoSnapshots package -DskipTests
127127
Verify that shim-specific classes are hidden from a conventional classloader.
128128

129129
```bash
130-
$ javap -cp dist/target/rapids-4-spark_2.12-24.12.0-cuda11.jar com.nvidia.spark.rapids.shims.SparkShimImpl
130+
$ javap -cp dist/target/rapids-4-spark_2.12-24.12.1-cuda11.jar com.nvidia.spark.rapids.shims.SparkShimImpl
131131
Error: class not found: com.nvidia.spark.rapids.shims.SparkShimImpl
132132
```
133133

134134
However, its bytecode can be loaded if prefixed with `spark3XY` not contained in the package name
135135

136136
```bash
137-
$ javap -cp dist/target/rapids-4-spark_2.12-24.12.0-cuda11.jar spark320.com.nvidia.spark.rapids.shims.SparkShimImpl | head -2
138-
Warning: File dist/target/rapids-4-spark_2.12-24.12.0-cuda11.jar(/spark320/com/nvidia/spark/rapids/shims/SparkShimImpl.class) does not contain class spark320.com.nvidia.spark.rapids.shims.SparkShimImpl
137+
$ javap -cp dist/target/rapids-4-spark_2.12-24.12.1-cuda11.jar spark320.com.nvidia.spark.rapids.shims.SparkShimImpl | head -2
138+
Warning: File dist/target/rapids-4-spark_2.12-24.12.1-cuda11.jar(/spark320/com/nvidia/spark/rapids/shims/SparkShimImpl.class) does not contain class spark320.com.nvidia.spark.rapids.shims.SparkShimImpl
139139
Compiled from "SparkShims.scala"
140140
public final class com.nvidia.spark.rapids.shims.SparkShimImpl {
141141
```
@@ -178,7 +178,7 @@ mvn package -pl dist -am -Dbuildver=340 -DallowConventionalDistJar=true
178178
Verify `com.nvidia.spark.rapids.shims.SparkShimImpl` is conventionally loadable:
179179
180180
```bash
181-
$ javap -cp dist/target/rapids-4-spark_2.12-24.12.0-cuda11.jar com.nvidia.spark.rapids.shims.SparkShimImpl | head -2
181+
$ javap -cp dist/target/rapids-4-spark_2.12-24.12.1-cuda11.jar com.nvidia.spark.rapids.shims.SparkShimImpl | head -2
182182
Compiled from "SparkShims.scala"
183183
public final class com.nvidia.spark.rapids.shims.SparkShimImpl {
184184
```

‎README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@ as a `provided` dependency.
7373
<dependency>
7474
<groupId>com.nvidia</groupId>
7575
<artifactId>rapids-4-spark_2.12</artifactId>
76-
<version>24.12.0</version>
76+
<version>24.12.1</version>
7777
<scope>provided</scope>
7878
</dependency>
7979
```

‎aggregator/pom.xml

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
<?xml version="1.0" encoding="UTF-8"?>
22
<!--
3-
Copyright (c) 2021-2024, NVIDIA CORPORATION.
3+
Copyright (c) 2021-2025, NVIDIA CORPORATION.
44
55
Licensed under the Apache License, Version 2.0 (the "License");
66
you may not use this file except in compliance with the License.
@@ -22,13 +22,13 @@
2222
<parent>
2323
<groupId>com.nvidia</groupId>
2424
<artifactId>rapids-4-spark-jdk-profiles_2.12</artifactId>
25-
<version>24.12.0</version>
25+
<version>24.12.1</version>
2626
<relativePath>../jdk-profiles/pom.xml</relativePath>
2727
</parent>
2828
<artifactId>rapids-4-spark-aggregator_2.12</artifactId>
2929
<name>RAPIDS Accelerator for Apache Spark Aggregator</name>
3030
<description>Creates an aggregated shaded package of the RAPIDS plugin for Apache Spark</description>
31-
<version>24.12.0</version>
31+
<version>24.12.1</version>
3232

3333
<properties>
3434
<rapids.module>aggregator</rapids.module>

‎api_validation/pom.xml

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
<?xml version="1.0" encoding="UTF-8"?>
22
<!--
3-
Copyright (c) 2020-2024, NVIDIA CORPORATION.
3+
Copyright (c) 2020-2025, NVIDIA CORPORATION.
44
55
Licensed under the Apache License, Version 2.0 (the "License");
66
you may not use this file except in compliance with the License.
@@ -22,11 +22,11 @@
2222
<parent>
2323
<groupId>com.nvidia</groupId>
2424
<artifactId>rapids-4-spark-shim-deps-parent_2.12</artifactId>
25-
<version>24.12.0</version>
25+
<version>24.12.1</version>
2626
<relativePath>../shim-deps/pom.xml</relativePath>
2727
</parent>
2828
<artifactId>rapids-4-spark-api-validation_2.12</artifactId>
29-
<version>24.12.0</version>
29+
<version>24.12.1</version>
3030

3131
<properties>
3232
<rapids.module>api_validation</rapids.module>

‎datagen/README.md

+3-3
Original file line numberDiff line numberDiff line change
@@ -24,12 +24,12 @@ Where `$SPARK_VERSION` is a compressed version number, like 330 for Spark 3.3.0.
2424

2525
After this the jar should be at
2626
`target/datagen_2.12-$PLUGIN_VERSION-spark$SPARK_VERSION.jar`
27-
for example a Spark 3.3.0 jar for the 24.12.0 release would be
28-
`target/datagen_2.12-24.12.0-spark330.jar`
27+
for example a Spark 3.3.0 jar for the 24.12.1 release would be
28+
`target/datagen_2.12-24.12.1-spark330.jar`
2929

3030
To get a spark shell with this you can run
3131
```shell
32-
spark-shell --jars target/datagen_2.12-24.12.0-spark330.jar
32+
spark-shell --jars target/datagen_2.12-24.12.1-spark330.jar
3333
```
3434

3535
After that you should be good to go.

‎datagen/ScaleTest.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ $SPARK_HOME/bin/spark-submit \
4444
--conf spark.sql.parquet.datetimeRebaseModeInWrite=CORRECTED \
4545
--class com.nvidia.rapids.tests.scaletest.ScaleTestDataGen \ # the main class
4646
--jars $SPARK_HOME/examples/jars/scopt_2.12-3.7.1.jar \ # one dependency jar just shipped with Spark under $SPARK_HOME
47-
./target/datagen_2.12-24.12.0-spark332.jar \
47+
./target/datagen_2.12-24.12.1-spark332.jar \
4848
1 \
4949
10 \
5050
parquet \

‎datagen/pom.xml

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
<?xml version="1.0" encoding="UTF-8"?>
22
<!--
3-
Copyright (c) 2023-2024 NVIDIA CORPORATION.
3+
Copyright (c) 2023-2025 NVIDIA CORPORATION.
44
55
Licensed under the Apache License, Version 2.0 (the "License");
66
you may not use this file except in compliance with the License.
@@ -21,13 +21,13 @@
2121
<parent>
2222
<groupId>com.nvidia</groupId>
2323
<artifactId>rapids-4-spark-shim-deps-parent_2.12</artifactId>
24-
<version>24.12.0</version>
24+
<version>24.12.1</version>
2525
<relativePath>../shim-deps/pom.xml</relativePath>
2626
</parent>
2727
<artifactId>datagen_2.12</artifactId>
2828
<name>Data Generator</name>
2929
<description>Tools for generating large amounts of data</description>
30-
<version>24.12.0</version>
30+
<version>24.12.1</version>
3131
<properties>
3232
<rapids.module>datagen</rapids.module>
3333
<target.classifier/>

‎delta-lake/delta-20x/pom.xml

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
<?xml version="1.0" encoding="UTF-8"?>
22
<!--
3-
Copyright (c) 2023-2024 NVIDIA CORPORATION.
3+
Copyright (c) 2023-2025 NVIDIA CORPORATION.
44
55
Licensed under the Apache License, Version 2.0 (the "License");
66
you may not use this file except in compliance with the License.
@@ -22,14 +22,14 @@
2222
<parent>
2323
<groupId>com.nvidia</groupId>
2424
<artifactId>rapids-4-spark-jdk-profiles_2.12</artifactId>
25-
<version>24.12.0</version>
25+
<version>24.12.1</version>
2626
<relativePath>../../jdk-profiles/pom.xml</relativePath>
2727
</parent>
2828

2929
<artifactId>rapids-4-spark-delta-20x_2.12</artifactId>
3030
<name>RAPIDS Accelerator for Apache Spark Delta Lake 2.0.x Support</name>
3131
<description>Delta Lake 2.0.x support for the RAPIDS Accelerator for Apache Spark</description>
32-
<version>24.12.0</version>
32+
<version>24.12.1</version>
3333

3434
<properties>
3535
<rapids.module>../delta-lake/delta-20x</rapids.module>

‎delta-lake/delta-21x/pom.xml

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
<?xml version="1.0" encoding="UTF-8"?>
22
<!--
3-
Copyright (c) 2023-2024 NVIDIA CORPORATION.
3+
Copyright (c) 2023-2025 NVIDIA CORPORATION.
44
55
Licensed under the Apache License, Version 2.0 (the "License");
66
you may not use this file except in compliance with the License.
@@ -22,14 +22,14 @@
2222
<parent>
2323
<groupId>com.nvidia</groupId>
2424
<artifactId>rapids-4-spark-jdk-profiles_2.12</artifactId>
25-
<version>24.12.0</version>
25+
<version>24.12.1</version>
2626
<relativePath>../../jdk-profiles/pom.xml</relativePath>
2727
</parent>
2828

2929
<artifactId>rapids-4-spark-delta-21x_2.12</artifactId>
3030
<name>RAPIDS Accelerator for Apache Spark Delta Lake 2.1.x Support</name>
3131
<description>Delta Lake 2.1.x support for the RAPIDS Accelerator for Apache Spark</description>
32-
<version>24.12.0</version>
32+
<version>24.12.1</version>
3333

3434
<properties>
3535
<rapids.module>../delta-lake/delta-21x</rapids.module>

‎delta-lake/delta-22x/pom.xml

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
<?xml version="1.0" encoding="UTF-8"?>
22
<!--
3-
Copyright (c) 2023-2024 NVIDIA CORPORATION.
3+
Copyright (c) 2023-2025 NVIDIA CORPORATION.
44
55
Licensed under the Apache License, Version 2.0 (the "License");
66
you may not use this file except in compliance with the License.
@@ -22,14 +22,14 @@
2222
<parent>
2323
<groupId>com.nvidia</groupId>
2424
<artifactId>rapids-4-spark-jdk-profiles_2.12</artifactId>
25-
<version>24.12.0</version>
25+
<version>24.12.1</version>
2626
<relativePath>../../jdk-profiles/pom.xml</relativePath>
2727
</parent>
2828

2929
<artifactId>rapids-4-spark-delta-22x_2.12</artifactId>
3030
<name>RAPIDS Accelerator for Apache Spark Delta Lake 2.2.x Support</name>
3131
<description>Delta Lake 2.2.x support for the RAPIDS Accelerator for Apache Spark</description>
32-
<version>24.12.0</version>
32+
<version>24.12.1</version>
3333

3434
<properties>
3535
<rapids.module>../delta-lake/delta-22x</rapids.module>

‎delta-lake/delta-23x/pom.xml

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
<?xml version="1.0" encoding="UTF-8"?>
22
<!--
3-
Copyright (c) 2023-2024 NVIDIA CORPORATION.
3+
Copyright (c) 2023-2025 NVIDIA CORPORATION.
44
55
Licensed under the Apache License, Version 2.0 (the "License");
66
you may not use this file except in compliance with the License.
@@ -22,14 +22,14 @@
2222
<parent>
2323
<groupId>com.nvidia</groupId>
2424
<artifactId>rapids-4-spark-parent_2.12</artifactId>
25-
<version>24.12.0</version>
25+
<version>24.12.1</version>
2626
<relativePath>../../pom.xml</relativePath>
2727
</parent>
2828

2929
<artifactId>rapids-4-spark-delta-23x_2.12</artifactId>
3030
<name>RAPIDS Accelerator for Apache Spark Delta Lake 2.3.x Support</name>
3131
<description>Delta Lake 2.3.x support for the RAPIDS Accelerator for Apache Spark</description>
32-
<version>24.12.0</version>
32+
<version>24.12.1</version>
3333

3434
<properties>
3535
<rapids.module>../delta-lake/delta-23x</rapids.module>

‎delta-lake/delta-24x/pom.xml

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
<?xml version="1.0" encoding="UTF-8"?>
22
<!--
3-
Copyright (c) 2023-2024 NVIDIA CORPORATION.
3+
Copyright (c) 2023-2025 NVIDIA CORPORATION.
44
55
Licensed under the Apache License, Version 2.0 (the "License");
66
you may not use this file except in compliance with the License.
@@ -22,14 +22,14 @@
2222
<parent>
2323
<groupId>com.nvidia</groupId>
2424
<artifactId>rapids-4-spark-jdk-profiles_2.12</artifactId>
25-
<version>24.12.0</version>
25+
<version>24.12.1</version>
2626
<relativePath>../../jdk-profiles/pom.xml</relativePath>
2727
</parent>
2828

2929
<artifactId>rapids-4-spark-delta-24x_2.12</artifactId>
3030
<name>RAPIDS Accelerator for Apache Spark Delta Lake 2.4.x Support</name>
3131
<description>Delta Lake 2.4.x support for the RAPIDS Accelerator for Apache Spark</description>
32-
<version>24.12.0</version>
32+
<version>24.12.1</version>
3333

3434
<properties>
3535
<rapids.module>../delta-lake/delta-24x</rapids.module>

‎delta-lake/delta-spark330db/pom.xml

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
<?xml version="1.0" encoding="UTF-8"?>
22
<!--
3-
Copyright (c) 2023-2024 NVIDIA CORPORATION.
3+
Copyright (c) 2023-2025 NVIDIA CORPORATION.
44
55
Licensed under the Apache License, Version 2.0 (the "License");
66
you may not use this file except in compliance with the License.
@@ -22,14 +22,14 @@
2222
<parent>
2323
<groupId>com.nvidia</groupId>
2424
<artifactId>rapids-4-spark-shim-deps-parent_2.12</artifactId>
25-
<version>24.12.0</version>
25+
<version>24.12.1</version>
2626
<relativePath>../../shim-deps/pom.xml</relativePath>
2727
</parent>
2828

2929
<artifactId>rapids-4-spark-delta-spark330db_2.12</artifactId>
3030
<name>RAPIDS Accelerator for Apache Spark Databricks 11.3 Delta Lake Support</name>
3131
<description>Databricks 11.3 Delta Lake support for the RAPIDS Accelerator for Apache Spark</description>
32-
<version>24.12.0</version>
32+
<version>24.12.1</version>
3333

3434
<properties>
3535
<rapids.module>../delta-lake/delta-spark330db</rapids.module>

‎delta-lake/delta-spark332db/pom.xml

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
<?xml version="1.0" encoding="UTF-8"?>
22
<!--
3-
Copyright (c) 2023-2024 NVIDIA CORPORATION.
3+
Copyright (c) 2023-2025 NVIDIA CORPORATION.
44
55
Licensed under the Apache License, Version 2.0 (the "License");
66
you may not use this file except in compliance with the License.
@@ -22,14 +22,14 @@
2222
<parent>
2323
<groupId>com.nvidia</groupId>
2424
<artifactId>rapids-4-spark-shim-deps-parent_2.12</artifactId>
25-
<version>24.12.0</version>
25+
<version>24.12.1</version>
2626
<relativePath>../../shim-deps/pom.xml</relativePath>
2727
</parent>
2828

2929
<artifactId>rapids-4-spark-delta-spark332db_2.12</artifactId>
3030
<name>RAPIDS Accelerator for Apache Spark Databricks 12.2 Delta Lake Support</name>
3131
<description>Databricks 12.2 Delta Lake support for the RAPIDS Accelerator for Apache Spark</description>
32-
<version>24.12.0</version>
32+
<version>24.12.1</version>
3333

3434
<properties>
3535
<rapids.module>../delta-lake/delta-spark332db</rapids.module>

‎delta-lake/delta-spark341db/pom.xml

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
<?xml version="1.0" encoding="UTF-8"?>
22
<!--
3-
Copyright (c) 2023-2024 NVIDIA CORPORATION.
3+
Copyright (c) 2023-2025 NVIDIA CORPORATION.
44
55
Licensed under the Apache License, Version 2.0 (the "License");
66
you may not use this file except in compliance with the License.
@@ -22,14 +22,14 @@
2222
<parent>
2323
<groupId>com.nvidia</groupId>
2424
<artifactId>rapids-4-spark-shim-deps-parent_2.12</artifactId>
25-
<version>24.12.0</version>
25+
<version>24.12.1</version>
2626
<relativePath>../../shim-deps/pom.xml</relativePath>
2727
</parent>
2828

2929
<artifactId>rapids-4-spark-delta-spark341db_2.12</artifactId>
3030
<name>RAPIDS Accelerator for Apache Spark Databricks 13.3 Delta Lake Support</name>
3131
<description>Databricks 13.3 Delta Lake support for the RAPIDS Accelerator for Apache Spark</description>
32-
<version>24.12.0</version>
32+
<version>24.12.1</version>
3333

3434
<properties>
3535
<rapids.compressed.artifact>false</rapids.compressed.artifact>

‎delta-lake/delta-spark350db143/pom.xml

+3-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
<?xml version="1.0" encoding="UTF-8"?>
22
<!--
3-
Copyright (c) 2023-2024 NVIDIA CORPORATION.
3+
Copyright (c) 2023-2025 NVIDIA CORPORATION.
44
55
Licensed under the Apache License, Version 2.0 (the "License");
66
you may not use this file except in compliance with the License.
@@ -22,14 +22,14 @@
2222
<parent>
2323
<groupId>com.nvidia</groupId>
2424
<artifactId>rapids-4-spark-shim-deps-parent_2.12</artifactId>
25-
<version>24.12.0</version>
25+
<version>24.12.1</version>
2626
<relativePath>../../shim-deps/pom.xml</relativePath>
2727
</parent>
2828

2929
<artifactId>rapids-4-spark-delta-spark350db143_2.12</artifactId>
3030
<name>RAPIDS Accelerator for Apache Spark Databricks 13.3 Delta Lake Support</name>
3131
<description>Databricks 13.3 Delta Lake support for the RAPIDS Accelerator for Apache Spark</description>
32-
<version>24.12.0</version>
32+
<version>24.12.1</version>
3333

3434
<properties>
3535
<rapids.compressed.artifact>false</rapids.compressed.artifact>

0 commit comments

Comments
 (0)