Skip to content

[Gradle] Refactor Java toolchain selection using Gradle toolchain management #34915

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 29 commits into
base: master
Choose a base branch
from
Open

[Gradle] Refactor Java toolchain selection using Gradle toolchain management #34915

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
96f6c82
Refactor Java toolchain selection using Gradle toolchain management
sjvanrossum May 12, 2025
357cadd
Fix workflow configurations and build scripts with compatibility issues
sjvanrossum May 12, 2025
2ed0a97
Target test created by task instead of test creation task
sjvanrossum May 12, 2025
49740ac
Revert remnant of Python toolchain management refactoring
sjvanrossum May 12, 2025
185a651
Remove remnants of project.ext.currentJavaVersion
sjvanrossum May 12, 2025
5772557
Merge remote-tracking branch 'upstream/master' into gradle-toolchain-…
sjvanrossum May 12, 2025
f849bd6
Fix path for :sdks:java:container:java[VERSION] targets
sjvanrossum May 12, 2025
fb5d0fc
Remove forkJavaVersion
sjvanrossum May 12, 2025
71dcc2d
Add java home paths for arm64
sjvanrossum May 12, 2025
c447529
Apply javaLauncher to test instead of aggregate task
sjvanrossum May 12, 2025
50733cc
Add missing task prefix
sjvanrossum May 12, 2025
97b012b
Run all HCatalog tests with JRE 8
sjvanrossum May 12, 2025
4fbb93b
Temporarily remove launcher overrides for Samza
sjvanrossum May 12, 2025
918c068
Remove outdated comment
sjvanrossum May 12, 2025
1579cc1
Run all Spark 3 tests with JRE 8
sjvanrossum May 12, 2025
1ac8f88
Remove remnant of getSupportedJavaVersion
sjvanrossum May 12, 2025
99975ce
Remove remnant of getSupportedJavaVersion
sjvanrossum May 12, 2025
4aae884
Run all Spark 3 tests with default toolchain
sjvanrossum May 12, 2025
4cbeb65
Remove verifyTestCodeIsCompiledWithJava[VERSION] tests
sjvanrossum May 12, 2025
a98a16f
Explicitly set javaVersion
sjvanrossum May 12, 2025
0a8fffc
Spotless
sjvanrossum May 12, 2025
e8ffce4
Add java as prefix to container image names
sjvanrossum May 12, 2025
1f0d535
Skip task if not supported instead of omitting registration
sjvanrossum May 13, 2025
caf3f63
Defer environment variable expansion for toolchain paths
sjvanrossum May 13, 2025
28ad2f6
Defer environment variable expansion for toolchain paths
sjvanrossum May 14, 2025
c86952a
Add missing javaVersion
sjvanrossum May 14, 2025
af56d3c
Explicitly set auto-download to false in test script
sjvanrossum May 14, 2025
1724dc9
Use gradle-command-self-hosted-action instead of custom steps
sjvanrossum May 15, 2025
4cd0819
Attempt to recover from toolchain incompatibility by selecting minimu…
sjvanrossum May 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/actions/gradle-command-self-hosted-action/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,5 +42,9 @@ runs:
rm ~/.m2/settings.xml
fi
./gradlew ${{ inputs.gradle-command }} --max-workers=${{ inputs.max-workers }} --continue \
-PjavaVersion=11 \
-Porg.gradle.java.installations.auto-detect=false \
-Porg.gradle.java.installations.auto-download=false \
-Porg.gradle.java.installations.fromEnv=JAVA_HOME_11_ARM64,JAVA_HOME_17_ARM64,JAVA_HOME_21_ARM64,JAVA_HOME_8_X64,JAVA_HOME_11_X64,JAVA_HOME_17_X64,JAVA_HOME_21_X64 \
-Dorg.gradle.jvmargs=-Xms2g -Dorg.gradle.jvmargs=-Xmx6g -Dorg.gradle.vfs.watch=false -Pdocker-pull-licenses \
${{ inputs.arguments }}
11 changes: 8 additions & 3 deletions .github/actions/setup-environment-action/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ inputs:
java-version:
required: false
description: 'Install Java version'
default: ''
default: 'default'
go-version:
required: false
description: 'Install Go version'
Expand Down Expand Up @@ -70,11 +70,16 @@ runs:
tox-${{ runner.os }}-py${{ inputs.python-version == 'default' && '39' || inputs.python-version }}-

- name: Install Java
if: ${{ inputs.java-version != '' }}
if: ${{ inputs.java-version != 'default' && '11' }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self-hosted runner comes with a java in PATH. We don't install java for most workflow runs, if such workflow not explicitly asked to do

Copy link
Contributor Author

@sjvanrossum sjvanrossum May 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm aware and while it is just another build of OpenJDK it's also subtly different from the Temurin release which is packaged in the default container images. On top of that, the default JDK version on Ubuntu 24.04 is their build of OpenJDK 21 and decoupling this build configuration's toolchains from the system toolchains makes it easier to upgrade machine images if needed.

Ubuntu 20.04's OpenJDK 11 build is configured with (build logs):

using configure arguments '--with-jvm-variants=server --with-boot-jdk=/usr/lib/jvm/java-11-openjdk-amd64 --disable-precompiled-headers --with-jvm-features=zgc,shenandoahgc --with-extra-cflags='-Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -fno-stack-protector -Wno-deprecated-declarations -Wdate-time -D_FORTIFY_SOURCE=2' --with-extra-cxxflags='-Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -fno-stack-protector -Wno-deprecated-declarations -Wno-deprecated-declarations' --with-extra-ldflags='-Xlinker -z -Xlinker relro -Xlinker -Bsymbolic-functions -Xlinker --no-as-needed' --enable-dtrace --disable-ccache --with-jtreg=/usr/share/jtreg --with-vendor-name=Ubuntu --with-vendor-url=https://ubuntu.com/ --with-vendor-bug-url=https://bugs.launchpad.net/ubuntu/+source/openjdk-lts --with-vendor-vm-bug-url=https://bugs.launchpad.net/ubuntu/+source/openjdk-lts --with-version-pre= --with-version-build=6 --with-version-opt='post-Ubuntu-0ubuntu1~20.04' --with-copyright-year=2025 --with-debug-level=release --with-native-debug-symbols=external --enable-unlimited-crypto --with-zlib=system --with-giflib=system --with-libpng=system --with-libjpeg=system --with-lcms=system --with-pcsclite=system --disable-warnings-as-errors --disable-javac-server --with-harfbuzz=system --with-stdc++lib=dynamic --with-num-cores=4'.

Adoptium's Temurin OpenJDK 11 build is configured with (build logs):

using configure arguments '--verbose --with-vendor-name='Eclipse Adoptium' --with-vendor-url=https://adoptium.net/ --with-vendor-bug-url=https://github.com/adoptium/adoptium-support/issues --with-vendor-vm-bug-url=https://github.com/adoptium/adoptium-support/issues --with-version-opt=202504181134 --with-version-pre=beta --with-version-build=6 --with-vendor-version-string=Temurin-11.0.27+6-202504181134 --with-boot-jdk=/usr/lib/jvm/jdk10 --with-jvm-features=shenandoahgc --with-debug-level=release --enable-ccache --with-alsa=/home/jenkins/workspace/./build//installedalsa --with-jvm-variants=server --with-cacerts-file=/home/jenkins/sbin/../security/cacerts --disable-warnings-as-errors --with-native-debug-symbols=none --with-zlib=bundled'.

Most of this shouldn't matter, but it's hard to tell whether Ubuntu's choice to mostly use system libraries, compile with GCC 10 and dynamically linking to stdc++ compared to Temurin's choice to bundle zlib, compile with GCC 7.5 and statically link stdc++ would ever matter in Beam's test suites. If the default JDK for Beam's provided container images is Temurin, then the test suites should at least target that same JDK distribution.

Copy link
Contributor

@Abacn Abacn May 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's using zulu jdk 11 actually -

beam/.github/gh-actions-self-hosted-runners/arc/images/Dockerfile

Line 43 in cbe100a

mv /usr/local/zulu11.74.15-ca-jdk11.0.24-linux_x64 /usr/local/java

OpenJDK it's also subtly different from the Temurin release

Yeah this is a good point. Strictly speaking we should use tumerin there

But Beam repo has many other tests doesn't require JDK. It justs need a java runtime to run gradle. I don't think we want to install JDK on every single workflow run (e.g. those python/go/website tests). Arguably Java package is much heavier than Python/Go

uses: actions/setup-java@v3
with:
distribution: 'temurin'
java-version: ${{ inputs.java-version }}
- name: Install Java 11 as default for Gradle and build targets
uses: actions/setup-java@v3
with:
distribution: 'temurin'
java-version: ${{ inputs.java-version == 'default' && '11' || inputs.java-version }}
java-version: '11'
- name: Setup Gradle
uses: gradle/gradle-build-action@v2
with:
Expand Down
12 changes: 4 additions & 8 deletions .github/workflows/beam_LoadTests_Java_CoGBK_Dataflow_V2_Batch_JavaVersions.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,8 +95,7 @@ jobs:
with:
gradle-command: :sdks:java:testing:load-tests:run
arguments: |
-PtestJavaVersion=${{ matrix.java_version }} \
-Pjava${{ matrix.java_version }}Home=$JAVA_HOME_${{ matrix.java_version }}_X64 \
-PjavaVersion=${{ matrix.java_version }} \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does -PjavaVersion affects the JDK used to compile main sources? Tests running on Java8/17/21 JVM should still have main sources compiled on JDK11, as we only ship jars compiled on java other than JDK11. Relevant test case is running JDK11 compiled classes on different JVM

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, if we keep "testJavaVersion" property these tests are going to pass on current PR

Copy link
Contributor Author

@sjvanrossum sjvanrossum May 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does -PjavaVersion affects the JDK used to compile main sources? Tests running on Java8/17/21 JVM should still have main sources compiled on JDK11, as we only ship jars compiled on java other than JDK11. Relevant test case is running JDK11 compiled classes on different JVM

The way I wired it up makes it affect the JDK selection for all JavaCompile, JavaExec and Test tasks, but not the JDK selection used for buildSrc since that uses the same JDK as the running Gradle process.

BeamModulePlugin.groovy L1023:

project.java {
  toolchain {
    languageVersion = JavaLanguageVersion.of(project.javaVersion)
  }
}

Much like Maven's toolchain plugin this results in the task requesting the Gradle process to select or discover a JDK fitting the toolchain spec, which could also include requests about the vendor (e.g. Adoptium, IBM, Oracle) or implementation (e.g. J9).
The javaVersion property sets the default for all compilations and executions, but individual tasks can modify the toolchain spec as needed.

runners/google-cloud-dataflow-java/examples/build.gradle L150:

task postCommitLegacyWorkerJava8(type: Test) {
  javaLauncher = javaToolchains.launcherFor {
    languageVersion = JavaLanguageVersion.of(8)
  }

  dependsOn ":runners:google-cloud-dataflow-java:worker:shadowJar"
  def dataflowWorkerJar = project.findProperty('dataflowWorkerJar') ?: project(":runners:google-cloud-dataflow-java:worker").shadowJar.archivePath
  with commonConfig(dataflowWorkerJar: dataflowWorkerJar, runWordCount: 'only')
}

These version-specific tasks aren't necessary, I've only found it necessary to override this property at the tesk level to deal with the HCatalog IO project due to incompatibilities with Java 11.

sdks/java/io/hcatalog/build.gradle L109:

tasks.withType(Test).configureEach {
    // TODO(https://github.com/apache/beam/issues/32189) remove when embedded hive supports Java11
    // Also change .github/workflows/beam_PostCommit_Java_Hadoop_Versions.yml
    javaLauncher = javaToolchains.launcherFor {
      languageVersion = JavaLanguageVersion.of(8)
    }
}

Given that the distributed JARs target Java 8 as the minimum supported language version the compilation output for any variation of javaVersion should be close to the output generated by the compiler that shipped in JDK 8, aside from compiler bugs like JDK-8204322. Executing those JARs might behave differently for variations of javaVersion (e.g.

beam/sdks/java/core/src/test/java/org/apache/beam/sdk/util/common/ReflectHelpersTest.java

Line 141 in 5f9cd73

equalTo("Default.String(value=\"package.OuterClass$InnerClass#method()\")")));
) and I think the next step would be to use the test suite plugin to copy the default test test suite target and configure each Test task to override javaLauncher, producing :path:to:project:test8 and :path:to:project:test11 for example.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does -PjavaVersion affects the JDK used to compile main sources? Tests running on Java8/17/21 JVM should still have main sources compiled on JDK11, as we only ship jars compiled on java other than JDK11. Relevant test case is running JDK11 compiled classes on different JVM

On the other point regarding the current binary distribution process and hence only targeting the compiler that ships with JDK 11, that seems like a sizable blind spot in the testing process to me. While the --release flag should make the current compiler behave much like the targeted release compiler, it may still generate additional warnings (which are treated as errors by this build configuration) or trip on a compiler bug. Using a single compiler version would hide these issues until the next time this build configuration changes the minimum supported JDK version.

It also makes it slightly harder for the few (none?) users who may opt to build Beam from source using their own toolchains and container images in organizations with restricted or centralized build infrastructure when they trip on these issues because these workflows don't cover Beam's stated version support range.

Copy link
Contributor

@Abacn Abacn May 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it affect the JDK selection for all JavaCompile, JavaExec and Test tasks

Using a single compiler version would hide these issues until the next time this build configuration changes the minimum supported JDK version.

Said differently, after this change it adds requirements for Beam to build on all supported Java versions (for components with versioning tests. Previously, it only requires tests build and run on supported Java versions, that is aligned with user adoption scenarios). This significantly increases scope of maintainence. Most importantly it would make new JDK support harder and delayed - when I did JDK21 support, at first Gradle 8.3- didn't support build on JDK21, but the work wasn't blocked by Gradle 8.4 release, and can proceed because we only need to make sure Beam runs on JDK21 fine. It won't be possible after this change.

Copy link
Contributor

@Abacn Abacn May 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the next step would be to use the test suite plugin to copy the default test test suite target and configure each Test task to override javaLauncher, producing :path:to:project:test8 and :path:to:project:test11 for example.

Yeah, this sounds a good idea replacing -PtestJavaVersion=xx -PjavaxxHome that is used currently in java github action workflows involving testing java version

-PloadTest.mainClass=org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
'-PloadTest.args=${{ env.beam_LoadTests_Java_CoGBK_Dataflow_V2_Batch_JavaVersions_test_arguments_1 }} --appName=load_tests_Java${{ matrix.java_version }}_Dataflow_V2_batch_CoGBK_1' \
Expand All @@ -105,8 +104,7 @@ jobs:
with:
gradle-command: :sdks:java:testing:load-tests:run
arguments: |
-PtestJavaVersion=${{ matrix.java_version }} \
-Pjava${{ matrix.java_version }}Home=$JAVA_HOME_${{ matrix.java_version }}_X64 \
-PjavaVersion=${{ matrix.java_version }} \
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
'-PloadTest.args=${{ env.beam_LoadTests_Java_CoGBK_Dataflow_V2_Batch_JavaVersions_test_arguments_2 }} --appName=load_tests_Java${{ matrix.java_version }}_Dataflow_V2_batch_CoGBK_2' \
Expand All @@ -115,8 +113,7 @@ jobs:
with:
gradle-command: :sdks:java:testing:load-tests:run
arguments: |
-PtestJavaVersion=${{ matrix.java_version }} \
-Pjava${{ matrix.java_version }}Home=$JAVA_HOME_${{ matrix.java_version }}_X64 \
-PjavaVersion=${{ matrix.java_version }} \
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
'-PloadTest.args=${{ env.beam_LoadTests_Java_CoGBK_Dataflow_V2_Batch_JavaVersions_test_arguments_3 }} --appName=load_tests_Java${{ matrix.java_version }}_Dataflow_V2_batch_CoGBK_3' \
Expand All @@ -125,8 +122,7 @@ jobs:
with:
gradle-command: :sdks:java:testing:load-tests:run
arguments: |
-PtestJavaVersion=${{ matrix.java_version }} \
-Pjava${{ matrix.java_version }}Home=$JAVA_HOME_${{ matrix.java_version }}_X64 \
-PjavaVersion=${{ matrix.java_version }} \
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
'-PloadTest.args=${{ env.beam_LoadTests_Java_CoGBK_Dataflow_V2_Batch_JavaVersions_test_arguments_4 }} --appName=load_tests_Java${{ matrix.java_version }}_Dataflow_V2_batch_CoGBK_4'
12 changes: 4 additions & 8 deletions .github/workflows/beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_JavaVersions.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,8 +95,7 @@ jobs:
with:
gradle-command: :sdks:java:testing:load-tests:run
arguments: |
-PtestJavaVersion=${{ matrix.java_version }} \
-Pjava${{ matrix.java_version }}Home=$JAVA_HOME_${{ matrix.java_version }}_X64 \
-PjavaVersion=${{ matrix.java_version }} \
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
'-PloadTest.args=${{ env.beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_JavaVersions_test_arguments_1 }} --appName=load_tests_Java${{ matrix.java_version }}_Dataflow_V2_streaming_CoGBK_1' \
Expand All @@ -105,8 +104,7 @@ jobs:
with:
gradle-command: :sdks:java:testing:load-tests:run
arguments: |
-PtestJavaVersion=${{ matrix.java_version }} \
-Pjava${{ matrix.java_version }}Home=$JAVA_HOME_${{ matrix.java_version }}_X64 \
-PjavaVersion=${{ matrix.java_version }} \
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
'-PloadTest.args=${{ env.beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_JavaVersions_test_arguments_2 }} --appName=load_tests_Java${{ matrix.java_version }}_Dataflow_V2_streaming_CoGBK_2' \
Expand All @@ -115,8 +113,7 @@ jobs:
with:
gradle-command: :sdks:java:testing:load-tests:run
arguments: |
-PtestJavaVersion=${{ matrix.java_version }} \
-Pjava${{ matrix.java_version }}Home=$JAVA_HOME_${{ matrix.java_version }}_X64 \
-PjavaVersion=${{ matrix.java_version }} \
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
'-PloadTest.args=${{ env.beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_JavaVersions_test_arguments_3 }} --appName=load_tests_Java${{ matrix.java_version }}_Dataflow_V2_streaming_CoGBK_3' \
Expand All @@ -125,8 +122,7 @@ jobs:
with:
gradle-command: :sdks:java:testing:load-tests:run
arguments: |
-PtestJavaVersion=${{ matrix.java_version }} \
-Pjava${{ matrix.java_version }}Home=$JAVA_HOME_${{ matrix.java_version }}_X64 \
-PjavaVersion=${{ matrix.java_version }} \
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.CoGroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
'-PloadTest.args=${{ env.beam_LoadTests_Java_CoGBK_Dataflow_V2_Streaming_JavaVersions_test_arguments_4 }} --appName=load_tests_Java${{ matrix.java_version }}_Dataflow_V2_streaming_CoGBK_4'
25 changes: 8 additions & 17 deletions .github/workflows/beam_LoadTests_Java_GBK_Dataflow_V2_Batch_Java17.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,9 +72,7 @@ jobs:
- name: Setup environment
uses: ./.github/actions/setup-environment-action
with:
java-version: |
17
8
java-version: 17
- name: Prepare test arguments
uses: ./.github/actions/test-arguments-action
with:
Expand All @@ -97,8 +95,7 @@ jobs:
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.GroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
-Prunner.version=V2 \
-PtestJavaVersion=17 \
-Pjava17Home=$JAVA_HOME_17_X64 \
-PjavaVersion=17 \
'-PloadTest.args=${{ env.beam_LoadTests_Java_GBK_Dataflow_V2_Batch_Java17_test_arguments_1 }}' \
- name: run Load test 2GB of 100B records
uses: ./.github/actions/gradle-command-self-hosted-action
Expand All @@ -108,8 +105,7 @@ jobs:
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.GroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
-Prunner.version=V2 \
-PtestJavaVersion=17 \
-Pjava17Home=$JAVA_HOME_17_X64 \
-PjavaVersion=17 \
'-PloadTest.args=${{ env.beam_LoadTests_Java_GBK_Dataflow_V2_Batch_Java17_test_arguments_2 }}' \
- name: run Load test 2GB of 100kB records
uses: ./.github/actions/gradle-command-self-hosted-action
Expand All @@ -119,8 +115,7 @@ jobs:
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.GroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
-Prunner.version=V2 \
-PtestJavaVersion=17 \
-Pjava17Home=$JAVA_HOME_17_X64 \
-PjavaVersion=17 \
'-PloadTest.args=${{ env.beam_LoadTests_Java_GBK_Dataflow_V2_Batch_Java17_test_arguments_3 }}' \
- name: run Load test fanout 4 times with 2GB 10-byte records total
uses: ./.github/actions/gradle-command-self-hosted-action
Expand All @@ -130,8 +125,7 @@ jobs:
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.GroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
-Prunner.version=V2 \
-PtestJavaVersion=17 \
-Pjava17Home=$JAVA_HOME_17_X64 \
-PjavaVersion=17 \
'-PloadTest.args=${{ env.beam_LoadTests_Java_GBK_Dataflow_V2_Batch_Java17_test_arguments_4 }}' \
- name: run Load test fanout 8 times with 2GB 10-byte records total
uses: ./.github/actions/gradle-command-self-hosted-action
Expand All @@ -141,8 +135,7 @@ jobs:
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.GroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
-Prunner.version=V2 \
-PtestJavaVersion=17 \
-Pjava17Home=$JAVA_HOME_17_X64 \
-PjavaVersion=17 \
'-PloadTest.args=${{ env.beam_LoadTests_Java_GBK_Dataflow_V2_Batch_Java17_test_arguments_5 }}' \
- name: run Load test reiterate 4 times 10kB values
uses: ./.github/actions/gradle-command-self-hosted-action
Expand All @@ -152,8 +145,7 @@ jobs:
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.GroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
-Prunner.version=V2 \
-PtestJavaVersion=17 \
-Pjava17Home=$JAVA_HOME_17_X64 \
-PjavaVersion=17 \
'-PloadTest.args=${{ env.beam_LoadTests_Java_GBK_Dataflow_V2_Batch_Java17_test_arguments_6 }}' \
- name: run Load test reiterate 4 times 2MB values
uses: ./.github/actions/gradle-command-self-hosted-action
Expand All @@ -163,6 +155,5 @@ jobs:
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.GroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
-Prunner.version=V2 \
-PtestJavaVersion=17 \
-Pjava17Home=$JAVA_HOME_17_X64 \
-PjavaVersion=17 \
'-PloadTest.args=${{ env.beam_LoadTests_Java_GBK_Dataflow_V2_Batch_Java17_test_arguments_7 }}'
25 changes: 8 additions & 17 deletions .github/workflows/beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java17.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,9 +72,7 @@ jobs:
- name: Setup environment
uses: ./.github/actions/setup-environment-action
with:
java-version: |
17
8
java-version: 17
- name: Prepare test arguments
uses: ./.github/actions/test-arguments-action
with:
Expand All @@ -97,8 +95,7 @@ jobs:
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.GroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
-Prunner.version=V2 \
-PtestJavaVersion=17 \
-Pjava17Home=$JAVA_HOME_17_X64 \
-PjavaVersion=17 \
'-PloadTest.args=${{ env.beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java17_test_arguments_1 }}' \
- name: run Load test 2GB of 100B records
uses: ./.github/actions/gradle-command-self-hosted-action
Expand All @@ -108,8 +105,7 @@ jobs:
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.GroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
-Prunner.version=V2 \
-PtestJavaVersion=17 \
-Pjava17Home=$JAVA_HOME_17_X64 \
-PjavaVersion=17 \
'-PloadTest.args=${{ env.beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java17_test_arguments_2 }}' \
- name: run Load test 2GB of 100kB records
uses: ./.github/actions/gradle-command-self-hosted-action
Expand All @@ -119,8 +115,7 @@ jobs:
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.GroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
-Prunner.version=V2 \
-PtestJavaVersion=17 \
-Pjava17Home=$JAVA_HOME_17_X64 \
-PjavaVersion=17 \
'-PloadTest.args=${{ env.beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java17_test_arguments_3 }}' \
- name: run Load test fanout 4 times with 2GB 10-byte records total
uses: ./.github/actions/gradle-command-self-hosted-action
Expand All @@ -130,8 +125,7 @@ jobs:
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.GroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
-Prunner.version=V2 \
-PtestJavaVersion=17 \
-Pjava17Home=$JAVA_HOME_17_X64 \
-PjavaVersion=17 \
'-PloadTest.args=${{ env.beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java17_test_arguments_4 }}' \
- name: run Load test fanout 8 times with 2GB 10-byte records total
uses: ./.github/actions/gradle-command-self-hosted-action
Expand All @@ -141,8 +135,7 @@ jobs:
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.GroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
-Prunner.version=V2 \
-PtestJavaVersion=17 \
-Pjava17Home=$JAVA_HOME_17_X64 \
-PjavaVersion=17 \
'-PloadTest.args=${{ env.beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java17_test_arguments_5 }}' \
- name: run Load test reiterate 4 times 10kB values
uses: ./.github/actions/gradle-command-self-hosted-action
Expand All @@ -152,8 +145,7 @@ jobs:
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.GroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
-Prunner.version=V2 \
-PtestJavaVersion=17 \
-Pjava17Home=$JAVA_HOME_17_X64 \
-PjavaVersion=17 \
'-PloadTest.args=${{ env.beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java17_test_arguments_6 }}' \
- name: run Load test reiterate 4 times 2MB values
uses: ./.github/actions/gradle-command-self-hosted-action
Expand All @@ -163,6 +155,5 @@ jobs:
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.GroupByKeyLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
-Prunner.version=V2 \
-PtestJavaVersion=17 \
-Pjava17Home=$JAVA_HOME_17_X64 \
-PjavaVersion=17 \
'-PloadTest.args=${{ env.beam_LoadTests_Java_GBK_Dataflow_V2_Streaming_Java17_test_arguments_7 }}'
12 changes: 4 additions & 8 deletions .github/workflows/beam_LoadTests_Java_ParDo_Dataflow_V2_Batch_JavaVersions.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,8 +98,7 @@ jobs:
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.ParDoLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
-Prunner.version=V2 \
-PtestJavaVersion=${{ matrix.java_version }} \
-Pjava${{ matrix.java_version }}Home=$JAVA_HOME_${{ matrix.java_version }}_X64 \
-PjavaVersion=${{ matrix.java_version }} \
'-PloadTest.args=${{ env.beam_LoadTests_Java_ParDo_Dataflow_V2_Batch_JavaVersions_test_arguments_1 }} --appName=load_tests_Java${{ matrix.java_version }}_Dataflow_V2_batch_ParDo_1' \
- name: run ParDo Dataflow V2 Batch Java Load Test 2 (200 times)
uses: ./.github/actions/gradle-command-self-hosted-action
Expand All @@ -109,8 +108,7 @@ jobs:
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.ParDoLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
-Prunner.version=V2 \
-PtestJavaVersion=${{ matrix.java_version }} \
-Pjava${{ matrix.java_version }}Home=$JAVA_HOME_${{ matrix.java_version }}_X64 \
-PjavaVersion=${{ matrix.java_version }} \
'-PloadTest.args=${{ env.beam_LoadTests_Java_ParDo_Dataflow_V2_Batch_JavaVersions_test_arguments_2 }} --appName=load_tests_Java${{ matrix.java_version }}_Dataflow_V2_batch_ParDo_2' \
- name: run ParDo Dataflow V2 Batch Java Load Test 3 (10 counters)
uses: ./.github/actions/gradle-command-self-hosted-action
Expand All @@ -120,8 +118,7 @@ jobs:
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.ParDoLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
-Prunner.version=V2 \
-PtestJavaVersion=${{ matrix.java_version }} \
-Pjava${{ matrix.java_version }}Home=$JAVA_HOME_${{ matrix.java_version }}_X64 \
-PjavaVersion=${{ matrix.java_version }} \
'-PloadTest.args=${{ env.beam_LoadTests_Java_ParDo_Dataflow_V2_Batch_JavaVersions_test_arguments_3 }} --appName=load_tests_Java${{ matrix.java_version }}_Dataflow_V2_batch_ParDo_3' \
- name: run ParDo Dataflow V2 Batch Java Load Test 4 (100 counters)
uses: ./.github/actions/gradle-command-self-hosted-action
Expand All @@ -131,6 +128,5 @@ jobs:
-PloadTest.mainClass=org.apache.beam.sdk.loadtests.ParDoLoadTest \
-Prunner=:runners:google-cloud-dataflow-java \
-Prunner.version=V2 \
-PtestJavaVersion=${{ matrix.java_version }} \
-Pjava${{ matrix.java_version }}Home=$JAVA_HOME_${{ matrix.java_version }}_X64 \
-PjavaVersion=${{ matrix.java_version }} \
'-PloadTest.args=${{ env.beam_LoadTests_Java_ParDo_Dataflow_V2_Batch_JavaVersions_test_arguments_4 }} --appName=load_tests_Java${{ matrix.java_version }}_Dataflow_V2_batch_ParDo_4'
Loading
Loading