Skip to content

Commit 795edf5

Browse files
dk1844dk1844
authored
#74 spark 3.3 added to project matrix (#75)
* #74 Spark 3.3 added to project matrix, info added to README.md
1 parent 1b80a0c commit 795edf5

File tree

3 files changed

+37
-7
lines changed

3 files changed

+37
-7
lines changed

‎README.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@
66

77
`spark-commons` is a library offering commonly needed routines, classes and functionality. It consists of three modules.
88
* spark-commons-spark2.4
9-
* spark-commons-spark3.7
9+
* spark-commons-spark3.2
10+
* spark-commons-spark3.3
1011
* spark-commons-test
1112

1213
**spark2-commons** and **spark3-commons** both offer the same logic for the respective major versions of Spark addressing
@@ -16,10 +17,10 @@ usual needs of Spark applications.
1617
version used)
1718

1819

19-
| | spark-commons-spark2.4 | spark-commons-spark3.2 | spark-commons-test |
20-
|--------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
21-
| _Scala 2.11_ | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark2.4_2.11/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark2.4_2.11) | | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-test_2.11/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-test_2.11) |
22-
| _Scala 2.12_ | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark2.4_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark2.4_2.12) | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.2_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.2_2.12) | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-test_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-test_2.12) |
20+
| | spark-commons-spark2.4 | spark-commons-spark3.2 | spark-commons-spark3.3 | spark-commons-test |
21+
|--------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
22+
| _Scala 2.11_ | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark2.4_2.11/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark2.4_2.11) | | | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-test_2.11/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-test_2.11) |
23+
| _Scala 2.12_ | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark2.4_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark2.4_2.12) | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.2_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.2_2.12) | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.3_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-spark3.3_2.12) | [![Maven Central](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-test_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/za.co.absa/spark-commons-test_2.12) |
2324

2425
## Spark-Commons
2526

‎build.sbt‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,8 @@ ThisBuild / organization := "za.co.absa"
1818
lazy val scala211 = "2.11.12"
1919
lazy val scala212 = "2.12.12"
2020
lazy val spark2 = "2.4.7"
21-
lazy val spark3 = "3.2.1"
21+
lazy val spark32 = "3.2.1"
22+
lazy val spark33 = "3.3.1"
2223

2324
import Dependencies._
2425
import SparkVersionAxis._
@@ -61,7 +62,8 @@ lazy val `sparkCommons` = (projectMatrix in file("spark-commons"))
6162
)
6263
)
6364
.sparkRow(SparkVersionAxis(spark2), scalaVersions = Seq(scala211, scala212))
64-
.sparkRow(SparkVersionAxis(spark3), scalaVersions = Seq(scala212))
65+
.sparkRow(SparkVersionAxis(spark32), scalaVersions = Seq(scala212))
66+
.sparkRow(SparkVersionAxis(spark33), scalaVersions = Seq(scala212))
6567
.dependsOn(sparkCommonsTest % "test")
6668

6769
lazy val sparkCommonsTest = (projectMatrix in file("spark-commons-test"))

‎spark-commons/src/main/scala-spark3.3-jvm/za/co/absa/spark/commons/adapters/TransformAdapter.scala‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
/*
2+
* Copyright 2021 ABSA Group Limited
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
package za.co.absa.spark.commons.adapters
18+
19+
import org.apache.spark.sql.Column
20+
import org.apache.spark.sql.functions.{transform => sparkTransform}
21+
22+
trait TransformAdapter {
23+
/**
24+
* For Spark versions prior 3.0.0, delegates to {{{hofs.transform()}}}
25+
*/
26+
def transform: (Column, Column => Column) => Column = sparkTransform
27+
}

0 commit comments

Comments
 (0)