You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
description: Upgrade Elasticsearch end-to-end: bump versions, check Lucene, compile, unit test, open PR, monitor CI and fix failures.
4
+
argument-hint: "[target version, e.g. 8.18.5]"
5
+
---
6
+
7
+
Upgrade Elasticsearch to the version specified by the user. If no version is specified, run `gh release list --repo elastic/elasticsearch --limit 60` and pick the closest available version to the current one (i.e., the next patch or minor release — minimize the version jump). Follow these steps in order, fixing any issues before moving to the next step:
8
+
9
+
1.**Bump versions** in all six places:
10
+
-`ElasticsearchVersion` in `build.sbt`
11
+
- Image tag in `docker/Dockerfile`
12
+
- Image tag and release download URL in `docs/pages/installation.md`
13
+
-`version` file (format: `X.Y.Z.0`)
14
+
-`elasticsearch` pin in `client-python/requirements.txt` — upgrade to the latest available version whose minor version does not exceed the new ES minor version. The Python client does **not** publish patch releases (e.g. there is no `8.18.6`), so check available versions first: `pip index versions elasticsearch` or check PyPI.
15
+
-`Elastic4sVersion` in `build.sbt` — upgrade to the latest available `nl.gn0s1s:elastic4s-client-esjava` version whose minor version does not exceed the new ES minor version. Check available versions on Maven Central: `curl -s "https://repo1.maven.org/maven2/nl/gn0s1s/elastic4s-client-esjava_3/maven-metadata.xml" | grep '<version>'`
16
+
17
+
2.**Check if LuceneVersion needs updating.** ES ships with a bundled Lucene. If it changed, the old pinned version will show as "evicted" in the dependency tree:
If evicted, bump `LuceneVersion` in `build.sbt` to match.
22
+
23
+
3.**Verify compilation:**`task jvmCompile`. This compiles all modules including integration test sources (`compile; Test/compile`). Fix any issues before proceeding.
24
+
25
+
4.**Verify unit tests:**`task jvmUnitTest`. Fix any failures before proceeding.
26
+
27
+
Once all steps pass, open a PR with the title `Dependencies: upgrade Elasticsearch to <new version>` (e.g. `Dependencies: upgrade Elasticsearch to 8.18.5`).
28
+
29
+
5.**Monitor CI and fix failures.** After the PR is open, poll GitHub Actions until CI completes or a job fails:
30
+
31
+
```bash
32
+
# Get the latest run ID for the PR branch
33
+
gh run list --branch <branch-name> --limit 5
34
+
35
+
# Watch a specific run (blocks until done, then prints summary)
36
+
gh run watch <run-id>
37
+
38
+
# If a job failed, get the logs
39
+
gh run view <run-id> --log-failed
40
+
```
41
+
42
+
For each failed job:
43
+
- Read the failure output carefully to identify the root cause.
44
+
- Fix the issue locally (edit code, push a new commit).
45
+
- Wait for a new CI run to start on that push, then repeat the watch/log loop.
46
+
- Continue until all jobs pass (green).
47
+
48
+
Do not close or abandon the PR — iterate until CI is green.
Integration tests require a Docker cluster (see `dockerRunTestingCluster` / `dockerStopTestingCluster` tasks in Taskfile.yaml).
24
+
25
+
### Java Version
26
+
27
+
The project uses Java 21 (see `.tool-versions`). Manage via asdf. Building with a newer JDK will embed that version in `plugin-descriptor.properties`, causing the plugin install to fail in the Docker container (which runs ES's bundled JDK).
28
+
29
+
## Project Structure
30
+
31
+
The project has seven SBT subprojects (Scala 3.3.6):
32
+
33
+
| Module | Role |
34
+
|--------|------|
35
+
|`elastiknn-api4s`| API types: (`Vec`, `Mapping`, `NearestNeighborsQuery`, etc.) XContent serialization |
36
+
|`elastiknn-models`| LSH model implementations and vector similarity math (Java + Panama SIMD) |
37
+
|`elastiknn-lucene`| Custom Lucene queries (`MatchHashesAndScoreQuery`) and field types |
38
+
|`elastiknn-plugin`| ES plugin: mappers, query builders, score functions |
39
+
|`elastiknn-plugin-integration-tests`| Full end-to-end tests against a live ES cluster |
40
+
|`elastiknn-client-elastic4s`| Scala client library using elastic4s |
ElasticsearchVersion// must match docker/Dockerfile and version file
49
+
LuceneVersion// must match the Lucene bundled in the ES release
50
+
ElastiknnVersion// read from the `version` file (e.g. 8.18.4.0)
51
+
```
52
+
53
+
When upgrading ES, update all four: `build.sbt`, `docker/Dockerfile`, `docs/pages/installation.md`, and `version`. Also check whether the bundled Lucene version changed (`sbt elastiknn-plugin/dependencyTree | grep lucene`).
54
+
55
+
## Dependency Upgrades
56
+
57
+
Use `/upgrade-elasticsearch` to perform an Elasticsearch upgrade end-to-end.
58
+
59
+
Other JVM dependencies are mostly handled by Scala Steward via automated PRs (see git log for the pattern).
60
+
61
+
## SIMD / Panama Vector API
62
+
63
+
`PanamaFloatVectorOps` uses `jdk.incubator.vector` for SIMD-accelerated dot product, cosine, L1, and L2 computations. It is enabled at runtime via the `elastiknn.jdk-incubator-vector.enabled` setting. The sbt server and ES container both need `--add-modules jdk.incubator.vector` (already configured in `.sbtopts` and `build.sbt`).
64
+
65
+
SIMD `reduceLanes(ADD)` can produce slightly different floating-point results before vs. after JIT compilation — relevant when writing determinism tests.
66
+
67
+
## CI
68
+
69
+
`.github/workflows/ci.yaml` runs: lint → compile → assemble → unit tests → start Docker cluster → integration tests. Scala compiler is strict (`CiMode`) when `CI=true`; locally it uses `DevMode` (relaxed).
The image version (`elasticsearch:A.B.C`) must match the plugin's version (e.g. `A.B.C.x/elastiknn-A.B.C.x`).
43
43
44
44
```docker
45
-
FROM docker.elastic.co/elasticsearch/elasticsearch:8.18.3
46
-
RUN elasticsearch-plugin install --batch https://github.com/alexklibisz/elastiknn/releases/download/8.18.3.0/elastiknn-8.18.3.0.zip
45
+
FROM docker.elastic.co/elasticsearch/elasticsearch:9.3.3
46
+
RUN elasticsearch-plugin install --batch https://github.com/alexklibisz/elastiknn/releases/download/9.3.3.0/elastiknn-9.3.3.0.zip
47
47
```
48
48
49
49
Build and run the Dockerfile. If you have any issues please refer to the [official docs.](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html)
0 commit comments