Skip to content

Commit c854ea3

Browse files
authored
fix(java): use Maven mirror, remove unnecessary plugins, other CI-stability improvements (#2253)
Contributes to NVIDIA/cuvs-lucene#145 Java builds have been failing like this: ```text Error: Unresolveable build extension: Plugin org.sonatype.plugins:nexus-staging-maven-plugin:1.6.7 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.sonatype.plugins:nexus-staging-maven-plugin:jar:1.6.7 ``` This is hiding the true issue ... **we're getting rate-limited by Maven Central**, with requests hitting `429 (Too Many Requests)`. This proposes the following changes to fix that: * switching to the unofficial Maven Central read-only mirror on GCS * removing unnecessary dependencies from `pom.xml` * configuring Maven to download more slowly and wait longer between retries * fine-tuning `changed-files` rules to reduce how often Java CI runs on PRs And these other changes relevant to the Java builds: * validating XML in `pre-commit` hooks (this did catch a syntax error in the Java examples' `pom.xml`!) * removing unnecessary configuration for snapshots (this project doesn't publish or download snapshots) * reformatting Java READMEs per https://yihui.org/en/2021/06/markdown-breath/ See inline comments for more details. ## Notes for Reviewers ### Impact I don't believe there are any breaking changes in this PR. Our existing publishing process for `cuvs-java` should be unaffected and CI should become faster and more reliable. If anything breaks because of it, assume that was unintentional. ### AI use I'm not very familiar with Java packaging, so heavily relied on back-and-forth with an agent for help with this. I wrote this PR description, code comments, and all inline review comments here myself. My motivation is to unblock CI here and in `cuvs-lucene`. If a more qualified reviewer wants to close this and implement a different fix, please do. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Corey J. Nolet (https://github.com/cjnolet) - Tim Liu (https://github.com/NvTimLiu) URL: #2253
1 parent 808b2f4 commit c854ea3

9 files changed

Lines changed: 209 additions & 79 deletions

File tree

.github/workflows/pr.yaml

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -160,6 +160,10 @@ jobs:
160160
- '!.github/workflows/trigger-breaking-change-alert.yaml'
161161
- '!.pre-commit-config.yaml'
162162
- '!.shellcheckrc'
163+
- '!CODEOWNERS'
164+
- '!Dockerfile'
165+
- '!README.md'
166+
- '!SECURITY.md'
163167
- '!ci/build_docs.sh'
164168
- '!ci/build_go.sh'
165169
- '!ci/build_python.sh'
@@ -171,14 +175,19 @@ jobs:
171175
- '!ci/test_python.sh'
172176
- '!ci/test_wheel_cuvs.sh'
173177
- '!ci/validate_wheel.sh'
174-
- '!Dockerfile'
178+
- '!conda/environments/**'
179+
- '!conda/recipes/cuvs/**'
180+
- '!conda/recipes/cuvs-bench/**'
181+
- '!conda/recipes/cuvs-bench-cpu/**'
175182
- '!docs/**'
183+
- '!examples/**'
176184
- '!fern/**'
185+
- '!go/**'
177186
- '!img/**'
178187
- '!notebooks/**'
188+
- '!pyproject.toml'
179189
- '!python/**'
180190
- '!rust/**'
181-
- '!go/**'
182191
- '!thirdparty/LICENSES/**'
183192
test_python_conda:
184193
- '**'
@@ -192,8 +201,12 @@ jobs:
192201
- '!.github/labeler.yml'
193202
- '!.github/ops-bot.yaml'
194203
- '!.github/release.yml'
204+
- '!.github/workflows/check-c-abi.yaml'
205+
- '!.github/workflows/labeler.yml'
195206
- '!.github/workflows/publish-rust.yaml'
207+
- '!.github/workflows/store-c-abi-baseline.yaml'
196208
- '!.github/workflows/trigger-breaking-change-alert.yaml'
209+
- '!.github/workflows/update-c-abi-baseline.yaml'
197210
- '!.pre-commit-config.yaml'
198211
- '!.shellcheckrc'
199212
- '!ci/build_go.sh'
@@ -210,6 +223,7 @@ jobs:
210223
- '!cpp/.clang-format'
211224
- '!cpp/.clang-tidy'
212225
- '!cpp/doxygen/**'
226+
- '!CODEOWNERS'
213227
- '!Dockerfile'
214228
- '!README.md'
215229
- '!SECURITY.md'
@@ -252,6 +266,7 @@ jobs:
252266
- '!ci/test_python.sh'
253267
- '!ci/test_standalone_c.sh'
254268
- '!ci/validate_wheel.sh'
269+
- '!CODEOWNERS'
255270
- '!Dockerfile'
256271
- '!README.md'
257272
- '!SECURITY.md'
@@ -283,6 +298,7 @@ jobs:
283298
- '!.github/workflows/trigger-breaking-change-alert.yaml'
284299
- '!.pre-commit-config.yaml'
285300
- '!.shellcheckrc'
301+
- '!CODEOWNERS'
286302
- '!Dockerfile'
287303
- '!README.md'
288304
- '!SECURITY.md'
@@ -324,6 +340,7 @@ jobs:
324340
- '!.github/workflows/trigger-breaking-change-alert.yaml'
325341
- '!.pre-commit-config.yaml'
326342
- '!.shellcheckrc'
343+
- '!CODEOWNERS'
327344
- '!Dockerfile'
328345
- '!README.md'
329346
- '!SECURITY.md'

.pre-commit-config.yaml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,13 @@
33

44
repos:
55
- repo: https://github.com/pre-commit/pre-commit-hooks
6-
rev: v5.0.0
6+
rev: v6.0.0
77
hooks:
88
- id: check-json
99
- id: trailing-whitespace
1010
- id: end-of-file-fixer
1111
- id: check-symlinks
12+
- id: check-xml
1213
- repo: https://github.com/PyCQA/isort
1314
rev: 5.12.0
1415
hooks:

java/README.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
# cuVS Java API
22

3-
43
cuVS Java API provides a Java based simple, efficient, and a robust vector search API.
54

65
> [!CAUTION]
@@ -13,7 +12,6 @@ cuVS Java API provides a Java based simple, efficient, and a robust vector searc
1312
- [JDK 22](https://openjdk.org/projects/jdk/22/)
1413
- [jextract for JDK 22](https://jdk.java.net/jextract/) (If not already installed, the build script downloads it)
1514

16-
1715
## Building
1816

1917
The libcuvs C and C++ libraries are needed for this API. If libcuvs libraries have not been built and installed, use `./build.sh libcuvs java` in the top level directory to build this API.
@@ -26,17 +24,22 @@ do `./build.sh java` in the top level directory or just do `./build.sh` in this
2624
Run `./build.sh --run-java-tests` from this directory.
2725

2826
To run a single test suite:
27+
2928
```sh
3029
cd cuvs-java/
3130
mvn clean integration-test -Dit.test=com.nvidia.cuvs.CagraBuildAndSearchIT
3231
```
32+
3333
or, for a single test:
34+
3435
```sh
3536
mvn clean integration-test -Dit.test=com.nvidia.cuvs.CagraBuildAndSearchIT#testMergeStrategies
3637
```
38+
3739
Be sure to set (manually, if needed) your `LD_LIBRARY_PATH` to include the directory with the appropriate (matching)
3840
version of `libcuvs.so`.
3941
Also, ensure that your panama bindings are up-to-date. They can be re-generated by running:
42+
4043
```sh
4144
./panama-bindings/generate-bindings.sh
4245
```
@@ -45,6 +48,7 @@ Tests run using a randomized runner. Specific failures can be reproduced running
4548
by passing `-Dtests.seed=42FC5CC6B4C6BA8E` (where `42FC5CC6B4C6BA8E` has to be
4649
replaced with your specific seed). It also possible to re-run a single test, but
4750
in this case it's necessary to pass the extended seed (suite:method), e.g.
51+
4852
```shell
4953
mvn integration-test -Dit.test=com.nvidia.cuvs.CagraBuildAndSearchIT#testFloatIndexing -Dtests.seed=66039A8CAFB9D3C9:449B6310296799E0
5054
```

java/benchmarks/README.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,25 +3,32 @@
33
This maven project contains JMH benchmarks for the CAGRA Java API.
44

55
## Prerequisites
6+
67
- [CuVS libraries](https://docs.rapids.ai/api/cuvs/stable/build/#build-from-source)
78
- Build the CuVS-Java API (`./build.sh` from the parent directory)
89

910
## Run benchmarks
1011

1112
Build:
13+
1214
```shell
1315
mvn clean verify
1416
```
17+
1518
Run:
19+
1620
```shell
1721
export RAFT_DEBUG_LOG_FILE=/dev/null
1822
java -jar target/benchmarks.jar
1923
```
24+
2025
The environment variable is needed to silence RAFT logging; RAFT emits some logs at INFO level when
2126
building indices and queries, and writing them to stdout (the default) influences benchmark results.
2227

2328
It is possible to change the dataset size and the vectors dimension via 2 parameters:
29+
2430
```shell
2531
java -jar target/benchmarks.jar -p size=4 -p dims=4
2632
```
33+
2734
Use `java -jar target/benchmarks.jar -h` for details on the options to fine-tune your benchmark runs.

java/benchmarks/pom.xml

Lines changed: 54 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,60 @@
1515

1616
<name>cuvs-java-benchmarks</name>
1717

18+
<repositories>
19+
<!-- Prefer Google Cloud mirror, which has higher rate limits -->
20+
<repository>
21+
<id>gcs-maven-central-mirror</id>
22+
<name>GCS Maven Central mirror</name>
23+
<url>https://maven-central.storage-download.googleapis.com/maven2/</url>
24+
<releases>
25+
<enabled>true</enabled>
26+
</releases>
27+
<snapshots>
28+
<enabled>false</enabled>
29+
</snapshots>
30+
</repository>
31+
<!-- Fall back to Maven upstream for packages not found in GCS mirror -->
32+
<repository>
33+
<id>central</id>
34+
<name>Maven Central</name>
35+
<url>https://repo.maven.apache.org/maven2</url>
36+
<releases>
37+
<enabled>true</enabled>
38+
</releases>
39+
<snapshots>
40+
<enabled>false</enabled>
41+
</snapshots>
42+
</repository>
43+
</repositories>
44+
45+
<pluginRepositories>
46+
<!-- Prefer Google Cloud mirror, which has higher rate limits -->
47+
<pluginRepository>
48+
<id>gcs-maven-central-mirror</id>
49+
<name>GCS Maven Central mirror</name>
50+
<url>https://maven-central.storage-download.googleapis.com/maven2/</url>
51+
<releases>
52+
<enabled>true</enabled>
53+
</releases>
54+
<snapshots>
55+
<enabled>false</enabled>
56+
</snapshots>
57+
</pluginRepository>
58+
<!-- Fall back to Maven upstream for packages not found in GCS mirror -->
59+
<pluginRepository>
60+
<id>central</id>
61+
<name>Maven Plugin Repository</name>
62+
<url>https://repo.maven.apache.org/maven2</url>
63+
<releases>
64+
<enabled>true</enabled>
65+
</releases>
66+
<snapshots>
67+
<enabled>false</enabled>
68+
</snapshots>
69+
</pluginRepository>
70+
</pluginRepositories>
71+
1872
<dependencies>
1973
<dependency>
2074
<groupId>org.openjdk.jmh</groupId>
@@ -104,38 +158,14 @@
104158
<artifactId>maven-clean-plugin</artifactId>
105159
<version>2.5</version>
106160
</plugin>
107-
<plugin>
108-
<artifactId>maven-deploy-plugin</artifactId>
109-
<version>2.8.1</version>
110-
</plugin>
111-
<plugin>
112-
<artifactId>maven-install-plugin</artifactId>
113-
<version>2.5.1</version>
114-
</plugin>
115161
<plugin>
116162
<artifactId>maven-jar-plugin</artifactId>
117163
<version>2.4</version>
118164
</plugin>
119-
<plugin>
120-
<artifactId>maven-javadoc-plugin</artifactId>
121-
<version>2.9.1</version>
122-
</plugin>
123165
<plugin>
124166
<artifactId>maven-resources-plugin</artifactId>
125167
<version>2.6</version>
126168
</plugin>
127-
<plugin>
128-
<artifactId>maven-site-plugin</artifactId>
129-
<version>3.3</version>
130-
</plugin>
131-
<plugin>
132-
<artifactId>maven-source-plugin</artifactId>
133-
<version>2.2.1</version>
134-
</plugin>
135-
<plugin>
136-
<artifactId>maven-surefire-plugin</artifactId>
137-
<version>2.17</version>
138-
</plugin>
139169
</plugins>
140170
</pluginManagement>
141171
</build>

java/cuvs-java/.mvn/maven.config

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
-e
2+
-B
3+
-Daether.connector.basic.downstreamThreads=1
4+
-Daether.transport.http.retryHandler.count=5
5+
-Daether.transport.http.retryHandler.interval=10000
6+
-Dmaven.wagon.http.retryHandler.count=5

java/cuvs-java/pom.xml

Lines changed: 54 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,6 @@
11
<!--
2-
/*
3-
* SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION.
4-
* SPDX-License-Identifier: Apache-2.0
5-
*/
2+
SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
SPDX-License-Identifier: Apache-2.0
64
-->
75
<project xmlns="http://maven.apache.org/POM/4.0.0"
86
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
@@ -54,13 +52,59 @@
5452
<native.build.path>${project.build.directory}/../../../cpp/build</native.build.path>
5553
</properties>
5654

57-
<distributionManagement>
58-
<snapshotRepository>
59-
<id>ossrh</id>
60-
<url>https://oss.sonatype.org/content/repositories/snapshots</url>
61-
</snapshotRepository>
62-
</distributionManagement>
55+
<repositories>
56+
<!-- Prefer Google Cloud mirror, which has higher rate limits -->
57+
<repository>
58+
<id>gcs-maven-central-mirror</id>
59+
<name>GCS Maven Central mirror</name>
60+
<url>https://maven-central.storage-download.googleapis.com/maven2/</url>
61+
<releases>
62+
<enabled>true</enabled>
63+
</releases>
64+
<snapshots>
65+
<enabled>false</enabled>
66+
</snapshots>
67+
</repository>
68+
<!-- Fall back to Maven upstream for packages not found in GCS mirror -->
69+
<repository>
70+
<id>central</id>
71+
<name>Maven Central</name>
72+
<url>https://repo.maven.apache.org/maven2</url>
73+
<releases>
74+
<enabled>true</enabled>
75+
</releases>
76+
<snapshots>
77+
<enabled>false</enabled>
78+
</snapshots>
79+
</repository>
80+
</repositories>
6381

82+
<pluginRepositories>
83+
<!-- Prefer Google Cloud mirror, which has higher rate limits -->
84+
<pluginRepository>
85+
<id>gcs-maven-central-mirror</id>
86+
<name>GCS Maven Central mirror</name>
87+
<url>https://maven-central.storage-download.googleapis.com/maven2/</url>
88+
<releases>
89+
<enabled>true</enabled>
90+
</releases>
91+
<snapshots>
92+
<enabled>false</enabled>
93+
</snapshots>
94+
</pluginRepository>
95+
<!-- Fall back to Maven upstream for packages not found in GCS mirror -->
96+
<pluginRepository>
97+
<id>central</id>
98+
<name>Maven Plugin Repository</name>
99+
<url>https://repo.maven.apache.org/maven2</url>
100+
<releases>
101+
<enabled>true</enabled>
102+
</releases>
103+
<snapshots>
104+
<enabled>false</enabled>
105+
</snapshots>
106+
</pluginRepository>
107+
</pluginRepositories>
64108

65109
<dependencies>
66110

@@ -204,17 +248,6 @@
204248
</execution>
205249
</executions>
206250
</plugin>
207-
<plugin>
208-
<groupId>org.sonatype.plugins</groupId>
209-
<artifactId>nexus-staging-maven-plugin</artifactId>
210-
<version>1.6.7</version>
211-
<extensions>true</extensions>
212-
<configuration>
213-
<serverId>ossrh</serverId>
214-
<nexusUrl>https://oss.sonatype.org/</nexusUrl>
215-
<autoReleaseAfterClose>false</autoReleaseAfterClose>
216-
</configuration>
217-
</plugin>
218251
<plugin>
219252
<groupId>com.diffplug.spotless</groupId>
220253
<artifactId>spotless-maven-plugin</artifactId>

0 commit comments

Comments
 (0)