Skip to content

Commit de3cba2

Browse files
zkaoudizkaoudi
authored
Merge pull request #560 from xristlamp/main
Update to Java 17
2 parents d55652b + 368a695 commit de3cba2

32 files changed

Lines changed: 661 additions & 104 deletions

File tree

‎.github/workflows/backend.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,11 +40,11 @@ jobs:
4040
runs-on: ubuntu-latest
4141
steps:
4242
- uses: actions/checkout@v2
43-
- name: Set up JDK 11
43+
- name: Set up JDK 17
4444
uses: actions/setup-java@v2
4545
with:
46-
java-version: 11
47-
distribution: 'adopt'
46+
java-version: 17
47+
distribution: 'temurin'
4848
- name: Install Protoc
4949
run: sudo apt install -y protobuf-compiler
5050
- uses: actions/cache@v4

‎.github/workflows/codeql.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -36,11 +36,11 @@ jobs:
3636
uses: actions/checkout@v2
3737
with:
3838
submodules: true
39-
- name: Set up JDK 11
39+
- name: Set up JDK 17
4040
uses: actions/setup-java@v2
4141
with:
42-
java-version: 11
43-
distribution: 'adopt'
42+
java-version: 17
43+
distribution: 'temurin'
4444
- name: Install Protoc
4545
run: sudo apt install -y protobuf-compiler
4646
- uses: actions/cache@v4

‎.github/workflows/documentation.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -36,11 +36,11 @@ jobs:
3636
API_KEY: $HOME/gems
3737
steps:
3838
- uses: actions/checkout@v2
39-
- name: Set up JDK 1.8
39+
- name: Set up JDK 17
4040
uses: actions/setup-java@v2
4141
with:
42-
java-version: 8
43-
distribution: 'adopt'
42+
java-version: 17
43+
distribution: 'temurin'
4444
- name: Install ruby
4545
run: |
4646
sudo apt install -y ruby-full build-essential zlib1g-dev

‎README.md‎

Lines changed: 48 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -81,11 +81,13 @@ source ~/.zshrc
8181

8282
### Requirements at Runtime
8383

84-
Since Apache Wayang (incubating) is not an execution engine itself but rather manages the execution engines for you, it is important to have the necessary requirements installed.
84+
Apache Wayang (incubating) relies on external execution engines and Java to function correctly. Below are the updated runtime requirements:
8585

86-
- Apache Wayang supports Java versions 8 and above. However, the Wayang team recommends using Java version 11. Don’t forget to set the `JAVA_HOME` environment variable.
87-
- You need to install Apache Spark version 3 or higher. Don’t forget to set the `SPARK_HOME` environment variable.
88-
- You need to install Apache Hadoop version 3 or higher. Don’t forget to set the `HADOOP_HOME` environment variable.
86+
- **Java 17**: Make sure `JAVA_HOME` is correctly set to your Java 17 installation.
87+
- **Apache Spark 3.4.4**: Compatible with Scala 2.12. Set the `SPARK_HOME` environment variable.
88+
- **Apache Hadoop 3+**: Set the `HADOOP_HOME` environment variable.
89+
90+
> 🛠️ **Note:** When using Java 17, you _must_ add JVM flags to allow Wayang and Spark to access internal Java APIs, or you will encounter `IllegalAccessError`. See below.
8991
9092
### Validating the installation
9193

@@ -95,6 +97,26 @@ To execute your first application with Apache Wayang, you need to execute your p
9597
bin/wayang-submit org.apache.wayang.apps.wordcount.Main java file://$(pwd)/README.md
9698
```
9799

100+
### ⚙️ Java 17 Compatibility
101+
102+
When running Wayang applications using Java 17 (especially with Spark), you must add JVM flags to open specific internal Java modules. These flags resolve access issues with `sun.nio.ch.DirectBuffer` and others.
103+
104+
Update your `wayang-submit` (wayang-assembly/target/wayang-1.0.1-SNAPSHOT/bin/wayang-submit) script (or command) with:
105+
106+
```bash
107+
eval "$RUNNER \
108+
--add-exports=java.base/sun.nio.ch=ALL-UNNAMED \
109+
--add-opens=java.base/java.nio=ALL-UNNAMED \
110+
--add-opens=java.base/java.lang=ALL-UNNAMED \
111+
--add-opens=java.base/java.util=ALL-UNNAMED \
112+
--add-opens=java.base/java.io=ALL-UNNAMED \
113+
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED \
114+
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED \
115+
--add-opens=java.base/java.net=ALL-UNNAMED \
116+
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED \
117+
$FLAGS -cp \"${WAYANG_CLASSPATH}\" $CLASS ${ARGS}"
118+
```
119+
98120
## Getting Started
99121

100122
Wayang is available via Maven Central. To use it with Maven, include the following code snippet into your POM file:
@@ -128,15 +150,17 @@ In addition, you can obtain the most recent snapshot version of Wayang via Sonat
128150
```
129151

130152
### Prerequisites
131-
Apache Wayang (incubating) is built with Java 11 and Scala 2.12. However, to run Apache Wayang it is sufficient to have just Java 11 installed. Please also consider that processing platforms employed by Wayang might have further requirements.
153+
Apache Wayang (incubating) is built with Java 17 and Scala 2.12. However, to run Apache Wayang it is sufficient to have just Java 17 installed. Please also consider that processing platforms employed by Wayang might have further requirements.
132154
```
133-
Java 11
134-
[Scala 2.12]
155+
Java 17
156+
Scala 2.12.17
157+
Spark 3.4.4, Compatible with Scala 2.12.
158+
Maven
135159
```
136160

137161
> **NOTE:** In windows, you need to define the variable `HADOOP_HOME` with the winutils.exe, an not official option to obtain [this repository](https://github.com/steveloughran/winutils), or you can generate your winutils.exe following the instructions in the repository. Also, you may need to install [msvcr100.dll](https://www.microsoft.com/en-us/download/details.aspx?id=26999)
138162
139-
> **NOTE:** Make sure that the JAVA_HOME environment variable is set correctly to Java 11 as the prerequisite checker script currently supports up to Java 11 and checks the latest version of Java if you have higher version installed. In Linux, it is preferably to use the export JAVA_HOME method inside the project folder. It is also recommended running './mvnw clean install' before opening the project using IntelliJ.
163+
> **NOTE:** Make sure that the JAVA_HOME environment variable is set correctly to Java 17 as the prerequisite checker script currently supports up to Java 17 and checks the latest version of Java if you have higher version installed. In Linux, it is preferably to use the export JAVA_HOME method inside the project folder. It is also recommended running './mvnw clean install' before opening the project using IntelliJ.
140164
141165

142166
### Building
@@ -152,11 +176,24 @@ If you need to rebuild Wayang, e.g., to use a different Scala version, you can s
152176
```
153177
> **NOTE:** If you receive an error about not finding `MathExBaseVisitor`, then the problem might be that you are trying to build from IntelliJ, without Maven. MathExBaseVisitor is generated code, and a Maven build should generate it automatically.
154178

155-
> **NOTE:** In the current Maven setup, the version of scala is tied to the Java version, you can compile the profile `scala-11` with Java 8 and profile `scala-12` with Java 11.
179+
> **NOTE:**: In the current Maven setup, Wayang supports Java 17. The default Scala version is 2.12.17, which is compatible with Java 17. Ensure that your Spark distribution is also built with Scala 2.12 (e.g., `spark-3.4.4-bin-hadoop3-scala2.12`).
156180

157181
> **NOTE:** For compiling and testing the code it is required to have Hadoop installed on your machine.
158182

159183
> **NOTE:** the `standalone` profile to fix Hadoop and Spark versions, so that Wayang apps do not explicitly need to declare the corresponding dependencies.
184+
185+
> **NOTE**: When running applications (e.g., WordCount) with Java 17, you must pass additional flags to allow internal module access:
186+
187+
>--add-exports=java.base/sun.nio.ch=ALL-UNNAMED \
188+
--add-opens=java.base/java.nio=ALL-UNNAMED \
189+
--add-opens=java.base/java.lang=ALL-UNNAMED \
190+
--add-opens=java.base/java.util=ALL-UNNAMED \
191+
--add-opens=java.base/java.io=ALL-UNNAMED \
192+
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED \
193+
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED \
194+
--add-opens=java.base/java.net=ALL-UNNAMED \
195+
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED \
196+
160197
>
161198
> Also, note the `distro` profile, which assembles a binary Wayang distribution.
162199
To activate these profiles, you need to specify them when running maven, i.e.,
@@ -176,8 +213,8 @@ You can see examples on how to start using Wayang [here](guides/wayang-examples.
176213

177214
## Built With
178215

179-
* [Java 11](https://www.oracle.com/de/java/technologies/javase/jdk11-archive-downloads.html)
180-
* [Scala 2.12](https://www.scala-lang.org/download/2.12.0.html)
216+
* [Java 17](https://www.oracle.com/java/technologies/javase/17-0-14-relnotes.html)
217+
* [Scala 2.12.17](https://www.scala-lang.org/download/2.12.17.html)
181218
* [Maven](https://maven.apache.org/)
182219

183220
## Contributing

‎guides/tutorial.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,23 @@ To execute the WordCount example with Apache Wayang, you need to execute your pr
6363
cd wayang-1.0.1-SNAPSHOT
6464
./bin/wayang-submit org.apache.wayang.apps.wordcount.Main java file://$(pwd)/README.md
6565
```
66+
## If you're using Java 17, add the following JVM flags:
67+
Update your `wayang-submit` (wayang-assembly/target/wayang-1.0.1-SNAPSHOT/bin/wayang-submit) script (or command) with:
68+
69+
```shell
70+
eval "$RUNNER \
71+
--add-exports=java.base/sun.nio.ch=ALL-UNNAMED \
72+
--add-opens=java.base/java.nio=ALL-UNNAMED \
73+
--add-opens=java.base/java.lang=ALL-UNNAMED \
74+
--add-opens=java.base/java.util=ALL-UNNAMED \
75+
--add-opens=java.base/java.io=ALL-UNNAMED \
76+
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED \
77+
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED \
78+
--add-opens=java.base/java.net=ALL-UNNAMED \
79+
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED \
80+
$FLAGS -cp \"${WAYANG_CLASSPATH}\" $CLASS ${ARGS}"
81+
```
82+
6683
Then you should be able to see outputs like this:
6784

6885
![img.png](../images/wordcount_result.png)

‎pom.xml‎

Lines changed: 47 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -102,33 +102,33 @@
102102
<!-- Timestamp for the reproducible builds -->
103103
<project.build.outputTimestamp>2025-02-04T20:09:34Z</project.build.outputTimestamp>
104104

105-
<scala.version>2.12.12</scala.version>
105+
<scala.version>2.12.17</scala.version>
106106
<scala.mayor.version>2.12</scala.mayor.version>
107-
<spark.version>3.1.2</spark.version>
107+
<spark.version>3.4.4</spark.version>
108108
<flink.version>1.20.0</flink.version>
109109

110-
<java.version>11</java.version>
111-
<source.level>11</source.level>
112-
<maven.compiler.source>11</maven.compiler.source>
113-
<maven.compiler.target>11</maven.compiler.target>
114-
<maven.compiler.release>11</maven.compiler.release>
110+
<java.version>17</java.version>
111+
<source.level>17</source.level>
112+
<maven.compiler.source>17</maven.compiler.source>
113+
<maven.compiler.target>17</maven.compiler.target>
114+
<maven.compiler.release>17</maven.compiler.release>
115115
<jaxb.version>2.2.11</jaxb.version>
116116
<java-activation.version>1.1.1</java-activation.version>
117117
<javax-annotation-api>1.3.2</javax-annotation-api>
118118

119-
<assertj.version>3.17.2</assertj.version>
120-
<commons-io.version>2.5</commons-io.version>
121-
<guava.version>19.0</guava.version>
122-
<hamcrest.version>1.3</hamcrest.version>
123-
<jackson.version>2.10.2</jackson.version>
124-
<jacoco.version>0.8.5</jacoco.version>
119+
<assertj.version>3.25.3</assertj.version>
120+
<commons-io.version>2.15.1</commons-io.version>
121+
<guava.version>32.1.3-jre</guava.version>
122+
<hamcrest.version>2.2</hamcrest.version>
123+
<jackson.version>2.15.4</jackson.version>
124+
<jacoco.version>0.8.9</jacoco.version>
125125
<jodatime.version>2.10.6</jodatime.version>
126126
<jsonpath.version>2.4.0</jsonpath.version>
127-
<junit5.version>5.6.1</junit5.version>
128-
<mockito.version>3.5.10</mockito.version>
127+
<junit5.version>5.10.2</junit5.version>
128+
<mockito.version>5.11.0</mockito.version>
129129
<mockk.version>1.10.0</mockk.version>
130130
<external.platforms.scope>provided</external.platforms.scope>
131-
<hadoop.version>3.1.2</hadoop.version>
131+
<hadoop.version>3.3.6</hadoop.version>
132132
<!-- To be overridden by individual modules -->
133133
<java-module-name>org.apache.wayang.default</java-module-name>
134134
<code.coverage.project.folder>${basedir}/</code.coverage.project.folder>
@@ -286,16 +286,16 @@
286286

287287
<!-- Used to build Java 11+ -->
288288
<profile>
289-
<id>java11</id>
289+
<id>java17</id>
290290
<activation>
291-
<jdk>11</jdk>
291+
<jdk>17</jdk>
292292
</activation>
293293
<properties>
294-
<java.version>11</java.version>
295-
<source.level>11</source.level>
296-
<maven.compiler.source>11</maven.compiler.source>
297-
<maven.compiler.target>11</maven.compiler.target>
298-
<maven.compiler.release>11</maven.compiler.release>
294+
<java.version>17</java.version>
295+
<source.level>17</source.level>
296+
<maven.compiler.source>17</maven.compiler.source>
297+
<maven.compiler.target>17</maven.compiler.target>
298+
<maven.compiler.release>17</maven.compiler.release>
299299
<jaxb.version>2.2.11</jaxb.version>
300300
<java-activation.version>1.1.1</java-activation.version>
301301
<javax-annotation-api>1.3.2</javax-annotation-api>
@@ -314,7 +314,7 @@
314314
<configuration>
315315
<rules>
316316
<requireJavaVersion>
317-
<version>11</version>
317+
<version>17</version>
318318
</requireJavaVersion>
319319
</rules>
320320
</configuration>
@@ -325,7 +325,7 @@
325325
<groupId>org.apache.maven.plugins</groupId>
326326
<artifactId>maven-javadoc-plugin</artifactId>
327327
<configuration>
328-
<source>11</source>
328+
<source>17</source>
329329
<detectJavaApiLink>false</detectJavaApiLink>
330330
</configuration>
331331
</plugin>
@@ -393,7 +393,7 @@
393393
<plugin>
394394
<groupId>net.alchim31.maven</groupId>
395395
<artifactId>scala-maven-plugin</artifactId>
396-
<version>4.8.1</version>
396+
<version>4.9.5</version>
397397
<executions>
398398
<execution>
399399
<id>compile-scala</id>
@@ -447,9 +447,9 @@
447447
</file>
448448
</activation>
449449
<properties>
450-
<scala.version>2.12.12</scala.version>
450+
<scala.version>2.12.17</scala.version>
451451
<scala.mayor.version>2.12</scala.mayor.version>
452-
<spark.version>3.1.2</spark.version>
452+
<spark.version>3.4.4</spark.version>
453453
</properties>
454454
<build>
455455
<plugins>
@@ -555,7 +555,7 @@
555555
</file>
556556
</activation>
557557
<properties>
558-
<antlr.version>4.9.1</antlr.version>
558+
<antlr.version>4.13.1</antlr.version>
559559
</properties>
560560
<build>
561561
<pluginManagement>
@@ -775,7 +775,7 @@
775775
<dependency>
776776
<groupId>org.apache.logging.log4j</groupId>
777777
<artifactId>log4j-bom</artifactId>
778-
<version>2.17.0</version>
778+
<version>2.20.0</version>
779779
<scope>import</scope>
780780
<type>pom</type>
781781
</dependency>
@@ -959,8 +959,22 @@
959959
<plugin>
960960
<groupId>org.apache.maven.plugins</groupId>
961961
<artifactId>maven-surefire-plugin</artifactId>
962-
<version>3.0.0-M5</version>
962+
<version>3.0.0</version>
963963
<configuration>
964+
<argLine>
965+
--add-exports=java.base/sun.nio.ch=ALL-UNNAMED
966+
--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED
967+
--add-opens=java.base/java.nio=ALL-UNNAMED
968+
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
969+
--add-opens=java.base/java.lang=ALL-UNNAMED
970+
--add-opens=java.base/java.util=ALL-UNNAMED
971+
--add-opens=java.base/java.io=ALL-UNNAMED
972+
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
973+
--add-opens=java.base/sun.reflect.annotation=ALL-UNNAMED
974+
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
975+
--add-opens=java.base/java.net=ALL-UNNAMED
976+
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
977+
</argLine>
964978
<useSystemClassLoader>false</useSystemClassLoader>
965979
</configuration>
966980
</plugin>
@@ -1242,7 +1256,7 @@
12421256
<plugin>
12431257
<groupId>org.apache.maven.plugins</groupId>
12441258
<artifactId>maven-enforcer-plugin</artifactId>
1245-
<version>3.0.0</version>
1259+
<version>3.0.0-M3</version>
12461260
<executions>
12471261
<execution>
12481262
<id>enforce-maven-version</id>
@@ -1263,7 +1277,7 @@
12631277
<plugin>
12641278
<groupId>org.jacoco</groupId>
12651279
<artifactId>jacoco-maven-plugin</artifactId>
1266-
<version>0.8.3</version>
1280+
<version>0.8.9</version>
12671281
<executions>
12681282
<execution>
12691283
<id>coverage-initialize</id>

‎src/main/script/prerequisiteCheck.groovy‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -190,7 +190,7 @@ if (os == "windows") {
190190
// profiles.
191191
/////////////////////////////////////////////////////
192192

193-
checkJavaVersion("1.8", "11")
193+
checkJavaVersion("1.8", "17")
194194

195195
// Check if hadoop is available
196196
// It seems that this is only required on Windows systems.

‎wayang-api/wayang-api-json/pom.xml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -34,10 +34,10 @@
3434

3535
<properties>
3636
<java-module-name>org.apache.wayang.api.json</java-module-name>
37-
<maven.compiler.source>1.8</maven.compiler.source>
38-
<maven.compiler.target>1.8</maven.compiler.target>
37+
<maven.compiler.source>17</maven.compiler.source>
38+
<maven.compiler.target>17</maven.compiler.target>
3939
<encoding>UTF-8</encoding>
40-
<scala.version>2.12.10</scala.version>
40+
<scala.version>2.12.17</scala.version>
4141
<scala.compat.version>2.12</scala.compat.version>
4242
<spec2.version>4.2.0</spec2.version>
4343
</properties>
@@ -125,7 +125,7 @@
125125
<dependency>
126126
<groupId>com.fasterxml.jackson.module</groupId>
127127
<artifactId>jackson-module-scala_2.12</artifactId>
128-
<version>2.13.1</version>
128+
<version>2.15.4</version>
129129
</dependency>
130130

131131

0 commit comments

Comments
 (0)