Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions .github/workflows/build-native.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
name: Build Native Image

on:
workflow_dispatch:

env:
GRAALVM_DISTRIBUTION: 'graalvm'
JAVA_VERSION: '21'

jobs:
build:
name: Build ${{ matrix.platform }}
runs-on: ${{ matrix.os }}
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
include:
- os: ubuntu-latest
platform: linux-amd64
- os: ubuntu-24.04-arm
platform: linux-arm64

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Setup GraalVM
uses: graalvm/setup-graalvm@v1
with:
java-version: ${{ env.JAVA_VERSION }}
distribution: ${{ env.GRAALVM_DISTRIBUTION }}
github-token: ${{ secrets.GITHUB_TOKEN }}

- name: Build native image
run: |
./native/build.sh

- name: Package binary
run: |
tar -czf nextflow-lsp-${{ matrix.platform }}.tar.gz \
-C build/native/nativeCompile nextflow-lsp

- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: nextflow-lsp-${{ matrix.platform }}
path: nextflow-lsp-${{ matrix.platform }}.tar.gz
retention-days: 7
9 changes: 9 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,14 @@ endif
@cp build/libs/language-server-all.jar ~/.nextflow/lsp/$(STABLE)/v$(VERSION).jar
@echo "installed at: ~/.nextflow/lsp/$(STABLE)/v$(VERSION).jar"

.PHONY: native
native:
./native/build.sh

install-native:
@mkdir -p ~/.local/bin
@cp build/native/nativeCompile/nextflow-lsp ~/.local/bin
@echo "installed at: ~/.local/bin/nextflow-lsp"

clean:
./gradlew clean
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,8 @@ A separate branch is maintained for each stable release, starting with `STABLE-2
To make a new release of the language server:

1. Checkout the appropriate `STABLE-*` branch for Nextflow.
2. Build the language server locally.
3. Create a new GitHub release with the language server JAR and a list of notable changes.
2. Build the language server JAR and native image.
3. Create a new GitHub release with the JAR, native binary, and a list of notable changes.

## Troubleshooting

Expand Down
95 changes: 95 additions & 0 deletions adr/20260804-graalvm-native-image-build.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# GraalVM native image build

- Authors: Ben Sherman
- Status: proposed
- Date: 2026-08-04

## Summary

Build the language server as a [GraalVM Native Image](https://www.graalvm.org/latest/reference-manual/native-image/), which reduces startup time from ~400 ms to ~10 ms and eliminates the JRE dependency.

## Problem Statement

A language server is launched once per editor session, and again after every crash or manual restart. The editor has no Nextflow features until it answers `initialize`. Measured locally on Linux x86_64, the JAR takes ~400 ms to answer `initialize`, while a native image takes ~10 ms. The JAR also requires a compatible JRE, which the user must provide.

Compiling this particular application ahead of time is not routine, because much of what it does is dynamic:

- Gson deserializes every LSP message type reflectively.
- Groovy's `Java8.configureClassNode` reads `getDeclaredFields`/`getDeclaredMethods` on the Nextflow DSL and value-type classes to build the `ClassNode`s that back completion, hover, and type checking.
- Groovy's `invokedynamic` call sites reach `MethodHandleNatives` methods that `native-image` cannot compile at all.

So the decision is not only "should we ship a binary" but "can we produce a binary that behaves identically to the JAR, and how would we know". The second half is where the risk concentrates: a native image missing reflection metadata does not fail to build and does not crash. It answers `initialize` and then returns empty — or plausible but wrong — results.

## Goals

- **Startup latency.** The user-visible cost of launching the server should be negligible.
- **No JRE prerequisite** for consumers of the binary.
- **Behavioral equivalence with the JAR.** A binary that differs from the JAR is worse than no binary, because the difference is invisible to whoever ships it.
- **No silent failure.** A metadata gap must fail the build or the test, not degrade the binary in production.

## Non-goals

- **Replacing the JAR.** It remains the primary artifact; the native build is an optional alternative.
- **Minimizing binary size.** 78 MB is accepted.
- **Runtime performance tuning.** `native-image` recommends G1GC, PGO and `-march=native`; none are applied, since startup was the objective and steady-state throughput is already adequate.

## Considered Options

### Keep the JAR only

- Good, because it is one artifact for all platforms and needs no per-platform CI.
- Good, because there is one execution model, so the existing tests remain representative.
- Bad, because ~400 ms of startup is paid on every session and every restart.
- Bad, because it makes the client responsible for finding a JRE.

### GraalVM `native-image`

- Good, because it removes essentially all startup cost (~400 ms → ~10 ms) and the JRE dependency.
- Good, because a single self-contained file is the simplest thing for a client to launch.
- Bad, because correctness now depends on reflection metadata declared ahead of time, and the failure mode is silent.
- Bad, because it depends on `--report-unsupported-elements-at-runtime`, which is deprecated upstream (see Residual risks).
- Bad, because the binary is 78 MB against a 15 MB JAR, per platform.

### AppCDS or CRaC

- Good, because it keeps one execution model and raises no reflection metadata problem.
- Bad, because it reduces JVM startup rather than removing it — not the order-of-magnitude change we want. Not benchmarked here, so this is a judgment rather than a measurement.
- Bad, because CRaC needs a specific JDK and a checkpoint step, and still requires a runtime on the user's machine.

### `jlink`/`jpackage`

- Good, because it removes the JRE prerequisite, with behavior identical to the JAR by construction.
- Bad, because startup remains JVM startup.
- Bad, because it produces a per-platform bundle anyway, so the CI cost is comparable without the latency benefit.

## Solution

Add a GraalVM 21 `native-image` build producing `nextflow-lsp` for linux-amd64 and linux-arm64. Derive the reflection metadata from the resolved classpath rather than a traced session. Verify the build with an equivalent check against the JAR baseline.

## Rationale & discussion

### Extracting reflection metadata

`generateNativeImageMetadata` scans the resolved runtime classpath and emits `reflect-config.json`, `resource-config.json` and `proxy-config.json`, registering ~630 classes. Registration flags are set per group rather than uniformly, since image size is driven by reachability; the per-group reasoning lives in comments in `build.gradle`. The tracing agent remains a secondary source, for the tail that resists enumeration: the reflective entries behind Groovy's indy call sites, and the `META-INF/services` files that the JDK and Groovy load through `ServiceLoader`. `native-image` merges the two, static config first.

A scan is exhaustive where a traced session is accidental. The reflected-over sets are *closed* — Gson reflects over every type in `org.eclipse.lsp4j`, and Groovy's `configureClassNode` over every class handed to `ClassHelper.makeCached` — and those are properties of the jars, so scanning enumerates them completely and picks up dependency upgrades automatically. Traced coverage is instead a function of the script: the session used here reaches 2 of the 35 `nextflow.script.types.**` classes, and whatever it misses is not reported at build time, because the `--report-unsupported-elements-at-runtime` that Groovy's indy call sites force turns what would be build errors into runtime no-ops.

### Verification

`native/verify.py` runs one scripted LSP session (`native/simulate.sh`) against the JAR and the binary and diffs the responses order-insensitively; `native/build.sh` fails if they differ, in CI and locally.

The JAR is the only available oracle. Unit tests run on the JVM and say nothing about the binary, and no self-contained assertion separates a correct response from a plausible wrong one, which is the failure mode a metadata gap produces. Differential testing also needs no expected values, so it stays valid as the language evolves. The gate was validated in both directions: it passes on a correct binary and fails on a deliberately under-registered one, which is what makes "no silent failure" a property of the build rather than an aspiration.

### Residual risks

- **`--report-unsupported-elements-at-runtime` is load-bearing and deprecated.** Removing it fails the build with `DeletedElementException` on `MethodHandleNatives.setCallSiteTargetNormal`, reached from Groovy's indy call sites. A future GraalVM that drops the flag will need a different answer; this is the largest strategic risk to the approach.

- **The registered package set is a judgment call.** It derives from call sites we read, so a future feature reflecting over a new package will not be registered automatically. The mitigation is `native/verify.py`: such a gap surfaces as a response divergence rather than as a user report.

- **A macOS-specific regression would go unnoticed**, since CI builds Linux only.

- **The plugin registry is not covered.** Resolving `include { ... } from 'plugin/...'` fetches over HTTPS, which brings in `HttpClient` and the JSSE provider graph — reflection-heavy territory, and no config directory registers anything for it today. The session deliberately stays offline, so this would first fail for a user rather than in CI.

## Links

- [GraalVM Native Image](https://www.graalvm.org/latest/reference-manual/native-image/)
64 changes: 64 additions & 0 deletions adr/YYYYMMDD-template-name.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# [short title of solved problem and solution]

- Authors: [who wrote the ADR]
- Status: [draft | proposed | rejected | accepted | deprecated | … | superseded by [xxx](xxx.md)]
- Date: [YYYY-MM-DD when the decision was last updated]

## Summary

Quick description of the problem and the context. Should not take more than 2-3 lines.

## Problem Statement

Description of the technical problem to solve or to decision to make. This should be concise but provide all required details and the context related to the technical decision to be taken.

## Goals

Depending the context define clearly what are the goals or what are the most important decision drivers.

- [driver 1, e.g., a force, facing concern, …]
- [driver 2, e.g., a force, facing concern, …]
- … <!-- numbers of drivers can vary -->

## Non-goals

Define what's out of the scope of this ADR.

## Considered Options <!-- optional -->

### [option 1]

[example | description | pointer to more information | …] <!-- optional -->

- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- … <!-- numbers of pros and cons can vary -->

### [option 2]

[example | description | pointer to more information | …] <!-- optional -->

- Good, because [argument a]
- Good, because [argument b]
- Bad, because [argument c]
- … <!-- numbers of pros and cons can vary -->


## Solution

Summarize the solution or decision outcome in one-two lines.

## Rationale & discussion

Describe the solution or the decision outcome discussing how decision drivers have been applied and how it matches the declared goals. This section is expected to be concise though providing comprehensive description of the technical solution and covering all uncertainty or ambiguous points.

## Links <!-- optional -->

- [Link type](link to adr) <!-- example: Refined by [xxx](yyyymmdd-xxx.md) -->
- … <!-- numbers of links can vary -->

## More information (remove)

- [What is an ADR and why should you use them](https://github.com/thomvaill/log4brains/tree/master#-what-is-an-adr-and-why-should-you-use-them)
- [ADR GitHub organization](https://adr.github.io/)
3 changes: 3 additions & 0 deletions build.gradle
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
plugins {
id 'com.gradleup.shadow' version '8.3.5'
id 'org.graalvm.buildtools.native' version '0.10.4'
id 'application'
id 'groovy'
id 'java'
Expand Down Expand Up @@ -107,3 +108,5 @@ application {
shadowJar {
archiveVersion = ''
}

apply from: "$projectDir/native/native.gradle"
Loading
Loading