Skip to content

[CI] Enable Java test in CI workflow #805

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 45 commits into
base: branch-25.06
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
d81ce11
[CI] Enable Java test in CI workflow
rhdong Apr 3, 2025
1363b16
confusing but had to change
rhdong Apr 3, 2025
d23578e
Remove the redundancy test
rhdong Apr 3, 2025
0a11a6e
retry with -X when fail
rhdong Apr 3, 2025
ab2610f
fix directory for mvn test
rhdong Apr 4, 2025
101e166
add dump file commands
rhdong Apr 4, 2025
5c8c905
switch to nn-descent
rhdong Apr 7, 2025
f46ec06
set the signature only under deploy
rhdong Apr 7, 2025
25579b2
remove gpg plugin
rhdong Apr 8, 2025
3c0ef89
upload to https://downloads.rapids.ai/
rhdong Apr 8, 2025
e4a8d7e
Merge branch 'branch-25.06' into rhdong/java-ci
rhdong Apr 8, 2025
832555b
add build for nightlies & releases
rhdong Apr 9, 2025
f43a7d3
Merge branch 'branch-25.06' into rhdong/java-ci
rhdong Apr 9, 2025
1cd8d67
initial setup
Apr 10, 2025
48f878f
refactor script and fix comment
Apr 10, 2025
9a8c606
update script - add logic to detect and update panama java files
Apr 10, 2025
83d1ae2
update comment/messages in the script
Apr 10, 2025
4dcda3c
plug in the bindings script into the java build script
Apr 11, 2025
5a388c7
update comment
Apr 11, 2025
07e16d0
update script to remove last blank line from generated bindings
Apr 11, 2025
fcef00b
update script - renaming and conditional delete
Apr 11, 2025
4327f63
update script - handle exceptions
Apr 11, 2025
b23aa36
update script - trim last blank line for all generated files
Apr 11, 2025
271dee5
remove manually generated bindings with the new ones
Apr 11, 2025
cc745b7
plug in new binding classes
Apr 11, 2025
20063eb
Merge branch 'branch-25.06' into vivek/automate-panama-bindings
Apr 19, 2025
d347610
bump up javadoc plugin version and fix javadoc
Apr 19, 2025
6021c09
update readme and simplify logging in examples
Apr 19, 2025
df1f6f2
Enable ivf-pq index and search parameter configuration via the cuvs-j…
Apr 20, 2025
064760d
remove panama bindings
Apr 21, 2025
37cc0d4
update gitignore
Apr 21, 2025
02cc69f
update gitignore, add source gpuinfo struct, and simplify script
Apr 21, 2025
53fddbb
add gpuinfo header and simplify script
Apr 21, 2025
806af86
remove license header and a gitignore file
Apr 21, 2025
b715805
Merge branch 'branch-25.06' into rhdong/java-ci
rhdong Apr 22, 2025
9fc2702
Merge branch 'branch-25.06' into rhdong/java-ci
rhdong Apr 22, 2025
6db6cbb
Merge remote-tracking branch 'origin/branch-25.06' into rhdong/java-c…
rhdong Apr 22, 2025
2a253b8
Merge remote-tracking branch 'rhdong/rhdong/java-ci' into rhdong/java-ci
rhdong Apr 22, 2025
79a477e
Merge remote-tracking branch 'SearchScale/vivek/automate-panama-bindi…
rhdong Apr 22, 2025
050defc
Merge branch 'branch-25.06' into vivek/automate-panama-bindings
narangvivek10 Apr 24, 2025
b516348
adding jextract to prerequisites in readme
Apr 24, 2025
570fa2a
Download jextract-22 if not already present
Apr 24, 2025
b9531c0
Merge branch 'branch-25.06' into rhdong/java-ci
rhdong Apr 24, 2025
04f0bba
Merge remote-tracking branch 'SearchScale/vivek/automate-panama-bindi…
rhdong Apr 24, 2025
cb5d1ba
find correctly CUDA header
rhdong Apr 24, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .github/workflows/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,20 @@ jobs:
node_type: "gpu-l4-latest-1"
run_script: "ci/build_go.sh"
sha: ${{ inputs.sha }}
java-build:
needs: cpp-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/[email protected]
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
arch: "amd64"
date: ${{ inputs.date }}
container_image: "rapidsai/ci-conda:latest"
node_type: "gpu-l4-latest-1"
run_script: "ci/build_java.sh"
file_to_upload: "java/cuvs-java/target/"
sha: ${{ inputs.sha }}
python-build:
needs: [cpp-build]
secrets: inherit
Expand Down
25 changes: 25 additions & 0 deletions .github/workflows/pr.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ jobs:
- conda-cpp-checks
- conda-python-build
- conda-python-tests
- conda-java-tests
- docs-build
- rust-build
- go-build
Expand Down Expand Up @@ -73,6 +74,18 @@ jobs:
- '!rust/**'
- '!go/**'
- '!thirdparty/LICENSES/**'
test_java:
- '**'
- '!.devcontainer/**'
- '!.pre-commit-config.yaml'
- '!README.md'
- '!docs/**'
- '!img/**'
- '!notebooks/**'
- '!python/**'
- '!rust/**'
- '!go/**'
- '!thirdparty/LICENSES/**'
test_notebooks:
- '**'
- '!.devcontainer/**'
Expand Down Expand Up @@ -134,6 +147,18 @@ jobs:
if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_python
with:
build_type: pull-request
conda-java-tests:
needs: [conda-cpp-build, changed-files]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/[email protected]
if: fromJSON(needs.changed-files.outputs.changed_file_groups).test_java
with:
build_type: pull-request
node_type: "gpu-l4-latest-1"
arch: "amd64"
container_image: "rapidsai/ci-conda:latest"
run_script: "ci/test_java.sh"
file_to_upload: "java/cuvs-java/target/"
docs-build:
needs: conda-python-build
secrets: inherit
Expand Down
12 changes: 12 additions & 0 deletions .github/workflows/test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,18 @@ jobs:
branch: ${{ inputs.branch }}
date: ${{ inputs.date }}
sha: ${{ inputs.sha }}
conda-java-tests:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/[email protected]
with:
build_type: ${{ inputs.build_type }}
branch: ${{ inputs.branch }}
date: ${{ inputs.date }}
sha: ${{ inputs.sha }}
node_type: "gpu-l4-latest-1"
arch: "amd64"
container_image: "rapidsai/ci-conda:latest"
run_script: "ci/test_java.sh"
wheel-tests-cuvs:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/[email protected]
Expand Down
44 changes: 44 additions & 0 deletions ci/build_java.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
#!/bin/bash
# Copyright (c) 2025, NVIDIA CORPORATION.

set -euo pipefail

. /opt/conda/etc/profile.d/conda.sh

rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)

rapids-logger "Generate Java testing dependencies"

ENV_YAML_DIR="$(mktemp -d)"

rapids-dependency-file-generator \
--output conda \
--file-key java \
--prepend-channel "${CPP_CHANNEL}" \
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch)" | tee "${ENV_YAML_DIR}/env.yaml"

rapids-mamba-retry env create --yes -f "${ENV_YAML_DIR}/env.yaml" -n java

export CMAKE_GENERATOR=Ninja

# Temporarily allow unbound variables for conda activation.
set +u
conda activate java
set -u

rapids-print-env

rapids-logger "Check GPU usage"
nvidia-smi

EXITCODE=0
trap "EXITCODE=1" ERR
set +e

rapids-logger "Run Java build and tests"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this actually run tests requiring a GPU? If yes, can be changed so that only the PR & test workflows run tests?

If it doesn't run tests, then would be good update this line and also to switch the node type for the build workflow from gpu-l4-latest-1 to a CPU runner.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will run tests.


bash ./build.sh java

rapids-logger "Test script exiting with value: $EXITCODE"
exit ${EXITCODE}
58 changes: 58 additions & 0 deletions ci/test_java.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
#!/bin/bash
# Copyright (c) 2022-2025, NVIDIA CORPORATION.

set -euo pipefail

. /opt/conda/etc/profile.d/conda.sh

rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)

rapids-logger "Generate Java testing dependencies"

ENV_YAML_DIR="$(mktemp -d)"

rapids-dependency-file-generator \
--output conda \
--file-key test_java \
--prepend-channel "${CPP_CHANNEL}" \
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch)" | tee "${ENV_YAML_DIR}/env.yaml"

rapids-mamba-retry env create --yes -f "${ENV_YAML_DIR}/env.yaml" -n test

export CMAKE_GENERATOR=Ninja

# Temporarily allow unbound variables for conda activation.
set +u
conda activate test
set -u

rapids-print-env

rapids-logger "Check GPU usage"
nvidia-smi

EXITCODE=0
trap "EXITCODE=1" ERR
set +e

rapids-logger "Run Java build and tests"

bash ./build.sh java
if [[ $? -ne 0 ]]; then
rapids-logger "Initial Java build & test failed. Retrying with 'mvn clean verify -X'"
pushd java/cuvs-java/
mvn clean verify -X
popd
DUMP_DIR="/__w/cuvs/cuvs/java/cuvs-java/target/failsafe-reports"
for file in "$DUMP_DIR"/*.dumpstream; do
if [ -f "$file" ]; then
echo "======= Dump file: $file ======="
cat "$file"
echo "======= End of dump file ======="
fi
done
fi

rapids-logger "Test script exiting with value: $EXITCODE"
exit ${EXITCODE}
24 changes: 24 additions & 0 deletions dependencies.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,14 @@ files:
- depends_on_libcuvs
- depends_on_cuvs
- depends_on_cuvs_bench
test_java:
output: none
includes:
- cuda
- cuda_version
- rapids_build
- depends_on_libcuvs
- java
checks:
output: none
includes:
Expand Down Expand Up @@ -111,6 +119,14 @@ files:
- depends_on_libcuvs
- depends_on_libraft
- depends_on_nccl
java:
output: none
includes:
- cuda
- cuda_version
- rapids_build
- depends_on_libcuvs
- java
py_build_libcuvs:
output: pyproject
pyproject_dir: python/libcuvs
Expand Down Expand Up @@ -563,6 +579,14 @@ dependencies:
- output_types: [conda, requirements, pyproject]
packages:
- scikit-learn
java:
common:
- output_types: conda
packages:
- *cmake_ver
- maven
- openjdk=22.*
- boost
bench:
common:
- output_types: [conda, pyproject, requirements]
Expand Down
19 changes: 19 additions & 0 deletions java/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# cuvs-java
/cuvs-java/target/
/cuvs-java/bin/
/cuvs-java/src/main/java22/com/nvidia/cuvs/internal/panama/
/cuvs-java/*.cag
# internal
/internal/.ninja_deps
/internal/.ninja_log
/internal/CMakeCache.txt
/internal/CMakeFiles/
/internal/CPM_modules/
/internal/_deps/
/internal/build.ninja
/internal/cmake/
/internal/cmake_install.cmake
/internal/cpm-package-lock.cmake
/internal/Makefile
# examples
/examples/target/
31 changes: 22 additions & 9 deletions java/README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,27 @@
Prerequisites
-------------
# CuVS Java API

* JDK 22
* Maven 3.9.6 or later

To build this API, please do `./build.sh java` in the top level directory. Since this API is dependent on `libcuvs` it must be noted that `libcuvs` gets built automatically before building this API.
CuVS Java API provides a Java based simple, efficient, and a robust vector search API.

Alternatively, please build libcuvs (`./build.sh libcuvs` from top level directory) before building the Java API with `./build.sh` from this directory.
> [!CAUTION]
> CuVS 25.06 contains an experimental version and updates to this API are expected in the coming release.

Building
--------
## Prerequisites

`./build.sh` will generate the `libcuvs_java.so` file in the `internal/` directory, and then build the final jar file for the cuVS Java API in the `cuvs-java/` directory.
- [CuVS libraries](https://docs.rapids.ai/api/cuvs/stable/build/#build-from-source)
- [maven 3.9.6 or above](https://maven.apache.org/download.cgi)
- [JDK 22](https://openjdk.org/projects/jdk/22/)
- [jextract for JDK 22](https://jdk.java.net/jextract/) (If not already installed, the build script downloads it)


## Building

The libcuvs C and C++ libraries are needed for this API. If libcuvs libraries have not been built and installed, use `./build.sh libcuvs java` in the top level directory to build this API.

Alternatively, if libcuvs libraries are already built and you just want to build this API, please
do `./build.sh java` in the top level directory or just do `./build.sh` in this directory.


## Examples

A few starter examples of CAGRA, HNSW, and Bruteforce index are provided in the `examples` directory.
11 changes: 11 additions & 0 deletions java/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,17 @@ if [ -z "$CMAKE_PREFIX_PATH" ]; then
export CMAKE_PREFIX_PATH=`pwd`/../cpp/build
fi

# Generate Panama FFM API bindings and update (if any of them changed)
/bin/bash panama-bindings/generate-bindings.sh

BINDINGS_GENERATION_RETURN_VALUE=$?
if [ $BINDINGS_GENERATION_RETURN_VALUE != 0 ]
then
echo "Bindings generation did not complete normally (returned value ${BINDINGS_GENERATION_RETURN_VALUE})"
echo "Forcing this build process to abort"
exit 1
fi

cd internal && cmake . && cmake --build . \
&& cd .. \
&& mvn install:install-file -DgroupId=$GROUP_ID -DartifactId=cuvs-java-internal -Dversion=$VERSION -Dpackaging=so -Dfile=$SO_FILE_PATH/libcuvs_java.so \
Expand Down
1 change: 0 additions & 1 deletion java/cuvs-java/.gitignore

This file was deleted.

20 changes: 1 addition & 19 deletions java/cuvs-java/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,6 @@
<maven.compiler.source>22</maven.compiler.source>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<gpg.passphrase>${GPG_PASSPHRASE}</gpg.passphrase>
</properties>

<distributionManagement>
Expand Down Expand Up @@ -258,7 +257,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.9.1</version>
<version>3.11.2</version>
<executions>
<execution>
<id>attach-javadocs</id>
Expand All @@ -271,20 +270,6 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-gpg-plugin</artifactId>
<version>1.5</version>
<executions>
<execution>
<id>sign-artifacts</id>
<phase>verify</phase>
<goals>
<goal>sign</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.sonatype.plugins</groupId>
<artifactId>nexus-staging-maven-plugin</artifactId>
Expand All @@ -298,7 +283,4 @@
</plugin>
</plugins>
</build>



</project>
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,12 @@

package com.nvidia.cuvs;

import com.nvidia.cuvs.spi.CuVSProvider;

import java.io.InputStream;
import java.io.OutputStream;
import java.nio.file.Path;
import java.util.Objects;
import java.util.UUID;

import com.nvidia.cuvs.spi.CuVSProvider;

/**
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ public class BruteForceQuery {
* @param queryVectors 2D float query vector array
* @param mapping an instance of ID mapping
* @param topK the top k results to return
* @param prefilter the prefilter data to use while searching the BRUTEFORCE
* @param prefilters the prefilters data to use while searching the BRUTEFORCE
* index
* @param numDocs Maximum of bits in each prefilter, representing number of documents in this index.
* Used only when prefilter(s) is/are passed.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@
import java.io.OutputStream;
import java.nio.file.Path;
import java.util.Objects;
import java.util.UUID;

import com.nvidia.cuvs.spi.CuVSProvider;

Expand Down
Loading
Loading