-
Notifications
You must be signed in to change notification settings - Fork 98
[CI] Enable Java test in CI workflow #805
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
rhdong
wants to merge
45
commits into
rapidsai:branch-25.06
Choose a base branch
from
rhdong:rhdong/java-ci
base: branch-25.06
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,494
−20,015
Open
Changes from all commits
Commits
Show all changes
45 commits
Select commit
Hold shift + click to select a range
d81ce11
[CI] Enable Java test in CI workflow
rhdong 1363b16
confusing but had to change
rhdong d23578e
Remove the redundancy test
rhdong 0a11a6e
retry with -X when fail
rhdong ab2610f
fix directory for mvn test
rhdong 101e166
add dump file commands
rhdong 5c8c905
switch to nn-descent
rhdong f46ec06
set the signature only under deploy
rhdong 25579b2
remove gpg plugin
rhdong 3c0ef89
upload to https://downloads.rapids.ai/
rhdong e4a8d7e
Merge branch 'branch-25.06' into rhdong/java-ci
rhdong 832555b
add build for nightlies & releases
rhdong f43a7d3
Merge branch 'branch-25.06' into rhdong/java-ci
rhdong 1cd8d67
initial setup
48f878f
refactor script and fix comment
9a8c606
update script - add logic to detect and update panama java files
83d1ae2
update comment/messages in the script
4dcda3c
plug in the bindings script into the java build script
5a388c7
update comment
07e16d0
update script to remove last blank line from generated bindings
fcef00b
update script - renaming and conditional delete
4327f63
update script - handle exceptions
b23aa36
update script - trim last blank line for all generated files
271dee5
remove manually generated bindings with the new ones
cc745b7
plug in new binding classes
20063eb
Merge branch 'branch-25.06' into vivek/automate-panama-bindings
d347610
bump up javadoc plugin version and fix javadoc
6021c09
update readme and simplify logging in examples
df1f6f2
Enable ivf-pq index and search parameter configuration via the cuvs-j…
064760d
remove panama bindings
37cc0d4
update gitignore
02cc69f
update gitignore, add source gpuinfo struct, and simplify script
53fddbb
add gpuinfo header and simplify script
806af86
remove license header and a gitignore file
b715805
Merge branch 'branch-25.06' into rhdong/java-ci
rhdong 9fc2702
Merge branch 'branch-25.06' into rhdong/java-ci
rhdong 6db6cbb
Merge remote-tracking branch 'origin/branch-25.06' into rhdong/java-c…
rhdong 2a253b8
Merge remote-tracking branch 'rhdong/rhdong/java-ci' into rhdong/java-ci
rhdong 79a477e
Merge remote-tracking branch 'SearchScale/vivek/automate-panama-bindi…
rhdong 050defc
Merge branch 'branch-25.06' into vivek/automate-panama-bindings
narangvivek10 b516348
adding jextract to prerequisites in readme
570fa2a
Download jextract-22 if not already present
b9531c0
Merge branch 'branch-25.06' into rhdong/java-ci
rhdong 04f0bba
Merge remote-tracking branch 'SearchScale/vivek/automate-panama-bindi…
rhdong cb5d1ba
find correctly CUDA header
rhdong File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -64,6 +64,20 @@ jobs: | |
node_type: "gpu-l4-latest-1" | ||
run_script: "ci/build_go.sh" | ||
sha: ${{ inputs.sha }} | ||
java-build: | ||
needs: cpp-build | ||
secrets: inherit | ||
uses: rapidsai/shared-workflows/.github/workflows/[email protected] | ||
with: | ||
build_type: ${{ inputs.build_type || 'branch' }} | ||
branch: ${{ inputs.branch }} | ||
arch: "amd64" | ||
date: ${{ inputs.date }} | ||
container_image: "rapidsai/ci-conda:latest" | ||
node_type: "gpu-l4-latest-1" | ||
run_script: "ci/build_java.sh" | ||
file_to_upload: "java/cuvs-java/target/" | ||
sha: ${{ inputs.sha }} | ||
python-build: | ||
needs: [cpp-build] | ||
secrets: inherit | ||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -20,6 +20,7 @@ jobs: | |
- conda-cpp-checks | ||
- conda-python-build | ||
- conda-python-tests | ||
- conda-java-tests | ||
- docs-build | ||
- rust-build | ||
- go-build | ||
|
@@ -73,6 +74,18 @@ jobs: | |
- '!rust/**' | ||
- '!go/**' | ||
- '!thirdparty/LICENSES/**' | ||
test_java: | ||
- '**' | ||
- '!.devcontainer/**' | ||
- '!.pre-commit-config.yaml' | ||
- '!README.md' | ||
- '!docs/**' | ||
- '!img/**' | ||
- '!notebooks/**' | ||
- '!python/**' | ||
- '!rust/**' | ||
- '!go/**' | ||
- '!thirdparty/LICENSES/**' | ||
test_notebooks: | ||
- '**' | ||
- '!.devcontainer/**' | ||
|
@@ -134,6 +147,18 @@ jobs: | |
if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python | ||
with: | ||
build_type: pull-request | ||
conda-java-tests: | ||
needs: [conda-cpp-build, changed-files] | ||
secrets: inherit | ||
uses: rapidsai/shared-workflows/.github/workflows/[email protected] | ||
if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_java | ||
with: | ||
build_type: pull-request | ||
node_type: "gpu-l4-latest-1" | ||
arch: "amd64" | ||
container_image: "rapidsai/ci-conda:latest" | ||
run_script: "ci/test_java.sh" | ||
file_to_upload: "java/cuvs-java/target/" | ||
docs-build: | ||
needs: conda-python-build | ||
secrets: inherit | ||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -43,6 +43,18 @@ jobs: | |
branch: ${{ inputs.branch }} | ||
date: ${{ inputs.date }} | ||
sha: ${{ inputs.sha }} | ||
conda-java-tests: | ||
secrets: inherit | ||
uses: rapidsai/shared-workflows/.github/workflows/[email protected] | ||
with: | ||
build_type: ${{ inputs.build_type }} | ||
branch: ${{ inputs.branch }} | ||
date: ${{ inputs.date }} | ||
sha: ${{ inputs.sha }} | ||
node_type: "gpu-l4-latest-1" | ||
arch: "amd64" | ||
container_image: "rapidsai/ci-conda:latest" | ||
run_script: "ci/test_java.sh" | ||
wheel-tests-cuvs: | ||
secrets: inherit | ||
uses: rapidsai/shared-workflows/.github/workflows/[email protected] | ||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,44 @@ | ||
#!/bin/bash | ||
# Copyright (c) 2025, NVIDIA CORPORATION. | ||
|
||
set -euo pipefail | ||
|
||
. /opt/conda/etc/profile.d/conda.sh | ||
|
||
rapids-logger "Downloading artifacts from previous jobs" | ||
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp) | ||
|
||
rapids-logger "Generate Java testing dependencies" | ||
|
||
ENV_YAML_DIR="$(mktemp -d)" | ||
|
||
rapids-dependency-file-generator \ | ||
--output conda \ | ||
--file-key java \ | ||
--prepend-channel "${CPP_CHANNEL}" \ | ||
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch)" | tee "${ENV_YAML_DIR}/env.yaml" | ||
|
||
rapids-mamba-retry env create --yes -f "${ENV_YAML_DIR}/env.yaml" -n java | ||
|
||
export CMAKE_GENERATOR=Ninja | ||
|
||
# Temporarily allow unbound variables for conda activation. | ||
set +u | ||
conda activate java | ||
set -u | ||
|
||
rapids-print-env | ||
|
||
rapids-logger "Check GPU usage" | ||
nvidia-smi | ||
|
||
EXITCODE=0 | ||
trap "EXITCODE=1" ERR | ||
set +e | ||
|
||
rapids-logger "Run Java build and tests" | ||
|
||
bash ./build.sh java | ||
|
||
rapids-logger "Test script exiting with value: $EXITCODE" | ||
exit ${EXITCODE} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,58 @@ | ||
#!/bin/bash | ||
# Copyright (c) 2022-2025, NVIDIA CORPORATION. | ||
|
||
set -euo pipefail | ||
|
||
. /opt/conda/etc/profile.d/conda.sh | ||
|
||
rapids-logger "Downloading artifacts from previous jobs" | ||
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp) | ||
|
||
rapids-logger "Generate Java testing dependencies" | ||
|
||
ENV_YAML_DIR="$(mktemp -d)" | ||
|
||
rapids-dependency-file-generator \ | ||
--output conda \ | ||
--file-key test_java \ | ||
--prepend-channel "${CPP_CHANNEL}" \ | ||
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch)" | tee "${ENV_YAML_DIR}/env.yaml" | ||
|
||
rapids-mamba-retry env create --yes -f "${ENV_YAML_DIR}/env.yaml" -n test | ||
|
||
export CMAKE_GENERATOR=Ninja | ||
|
||
# Temporarily allow unbound variables for conda activation. | ||
set +u | ||
conda activate test | ||
set -u | ||
|
||
rapids-print-env | ||
|
||
rapids-logger "Check GPU usage" | ||
nvidia-smi | ||
|
||
EXITCODE=0 | ||
trap "EXITCODE=1" ERR | ||
set +e | ||
|
||
rapids-logger "Run Java build and tests" | ||
|
||
bash ./build.sh java | ||
if [[ $? -ne 0 ]]; then | ||
rapids-logger "Initial Java build & test failed. Retrying with 'mvn clean verify -X'" | ||
pushd java/cuvs-java/ | ||
mvn clean verify -X | ||
popd | ||
DUMP_DIR="/__w/cuvs/cuvs/java/cuvs-java/target/failsafe-reports" | ||
for file in "$DUMP_DIR"/*.dumpstream; do | ||
if [ -f "$file" ]; then | ||
echo "======= Dump file: $file =======" | ||
cat "$file" | ||
echo "======= End of dump file =======" | ||
fi | ||
done | ||
fi | ||
|
||
rapids-logger "Test script exiting with value: $EXITCODE" | ||
exit ${EXITCODE} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
# cuvs-java | ||
/cuvs-java/target/ | ||
/cuvs-java/bin/ | ||
/cuvs-java/src/main/java22/com/nvidia/cuvs/internal/panama/ | ||
/cuvs-java/*.cag | ||
# internal | ||
/internal/.ninja_deps | ||
/internal/.ninja_log | ||
/internal/CMakeCache.txt | ||
/internal/CMakeFiles/ | ||
/internal/CPM_modules/ | ||
/internal/_deps/ | ||
/internal/build.ninja | ||
/internal/cmake/ | ||
/internal/cmake_install.cmake | ||
/internal/cpm-package-lock.cmake | ||
/internal/Makefile | ||
# examples | ||
/examples/target/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,14 +1,27 @@ | ||
Prerequisites | ||
------------- | ||
# CuVS Java API | ||
|
||
* JDK 22 | ||
* Maven 3.9.6 or later | ||
|
||
To build this API, please do `./build.sh java` in the top level directory. Since this API is dependent on `libcuvs` it must be noted that `libcuvs` gets built automatically before building this API. | ||
CuVS Java API provides a Java based simple, efficient, and a robust vector search API. | ||
|
||
Alternatively, please build libcuvs (`./build.sh libcuvs` from top level directory) before building the Java API with `./build.sh` from this directory. | ||
> [!CAUTION] | ||
> CuVS 25.06 contains an experimental version and updates to this API are expected in the coming release. | ||
|
||
Building | ||
-------- | ||
## Prerequisites | ||
|
||
`./build.sh` will generate the `libcuvs_java.so` file in the `internal/` directory, and then build the final jar file for the cuVS Java API in the `cuvs-java/` directory. | ||
- [CuVS libraries](https://docs.rapids.ai/api/cuvs/stable/build/#build-from-source) | ||
- [maven 3.9.6 or above](https://maven.apache.org/download.cgi) | ||
- [JDK 22](https://openjdk.org/projects/jdk/22/) | ||
- [jextract for JDK 22](https://jdk.java.net/jextract/) (If not already installed, the build script downloads it) | ||
|
||
|
||
## Building | ||
|
||
The libcuvs C and C++ libraries are needed for this API. If libcuvs libraries have not been built and installed, use `./build.sh libcuvs java` in the top level directory to build this API. | ||
|
||
Alternatively, if libcuvs libraries are already built and you just want to build this API, please | ||
do `./build.sh java` in the top level directory or just do `./build.sh` in this directory. | ||
|
||
|
||
## Examples | ||
|
||
A few starter examples of CAGRA, HNSW, and Bruteforce index are provided in the `examples` directory. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this actually run tests requiring a GPU? If yes, can be changed so that only the PR & test workflows run tests?
If it doesn't run tests, then would be good update this line and also to switch the node type for the build workflow from
gpu-l4-latest-1
to a CPU runner.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It will run tests.