Add GraalVM native image compilation support - #146
Open
pditommaso wants to merge 13 commits into
Open
Conversation
4 tasks
bentsherman
force-pushed
the
add-native-image-support
branch
from
August 4, 2026 16:26
ee11c67 to
02b4ebd
Compare
bentsherman
force-pushed
the
add-native-image-support
branch
from
August 4, 2026 19:19
f7919ba to
8218794
Compare
Add a native-image build for the language server binary, covering linux-amd64, linux-arm64, macos-intel and macos-silicon. The build runs the tracing agent against a simulated LSP session to capture reflection metadata, then feeds it to native-image along with manual configuration in conf/native-image/.
- Register conf/native-image as a nativeCompile input, so editing the reflection config actually triggers a rebuild instead of reporting UP-TO-DATE and shipping the previous binary. - Drop the graalvmNative agent block. It only takes effect with -Pagent, and generateNativeImageConfig does the tracing instead. - Make the simulator send workspace/didChangeConfiguration. This is what initializes the language services, so without it every language feature returned an empty result and the tracing agent observed almost nothing but lsp4j plumbing. The resulting binary answered "initialize" but was dead for real use: it could not deserialize DidChangeConfigurationParams or ExecuteCommandParams, and reported false errors such as "Unrecognized process input qualifier val". - Also send workspace/executeCommand, use a real temp workspace instead of hardcoded /Users paths, and wait for outstanding async requests before shutdown -- responses arrive out of order and "exit" was truncating them. - Strengthen the binary test. Checking only for an "initialize" response passes on a binary whose language features are all broken; also check for missing-metadata errors and for a non-empty documentSymbol result. - Fix the release job: it collected *.tar.gz while the build uploaded a bare binary, and read the version from refs/tags while the workflow ignored tags. Package the binary as a tarball and trigger the release on v* tags. - Limit push builds to main and tags, so a PR no longer builds four native images twice. Add timeout-minutes, and use the free macos-latest runner for Apple Silicon instead of the billed macos-latest-xlarge. - Rename the binary from nlsp to nextflow-lsp.
The reflection metadata was produced only by tracing a simulated LSP session, which records what that one session happened to touch. Anything it missed shipped a binary that failed at runtime, and failed silently -- the --report-unsupported-elements-at-runtime flag Groovy needs turns what would be build errors into runtime no-ops. Register the class sets that are reflected over wholesale instead, generating reflect/resource/proxy config from the resolved classpath in generateNativeImageMetadata. Tracing is kept for the tail that genuinely cannot be enumerated (Groovy indy call sites, JDK internals), but is no longer the only source. 629 classes are now registered statically versus 270 observed by tracing. Flags are per group rather than uniform, because cost here is reachability, not entry count: lsp4j's 538 message types need fields and the no-arg constructor that Gson calls, not their accessors, while the Nextflow DSL and value-type classes do need allDeclaredMethods -- Groovy's Java8.configureAnnotation invokes the annotation accessors to read @Description/@Constant/@ops members. Newly covered, none of which the traced session reached: - nextflow.script.types.** including the .shim value types and their package-private *Ops interfaces. These back completion and hover on String/List/Map/Path; tracing had registered 1 of 35. - nextflow.config.spec.**, which is the package the code actually imports. The prefix list previously named nextflow.config.schema -- a near-identical decoy package that nothing references. - nextflow.script.namespaces.** and nextflow.config.dsl.ConfigDsl. - spec/definitions.json. ConfigSpecFactory is the only classpath-resource read in the server and it only runs for .config files, which no session opened. - The LanguageClient/Endpoint proxy pair, previously supplied only by tracing. Dropping the traced config made the binary die on startup with MissingReflectionRegistrationError. Add verify-native.py, which runs the same LSP session against the jar and the binary and diffs the responses. No self-contained assertion can catch this class of bug: with the DSL classes unregistered the server reported "Unrecognized process input qualifier val" on a valid script while every response stayed well-formed and non-empty. The JVM build is the only available oracle. Verified in both directions -- it passes on a correct binary and fails on a deliberately under-registered one. It replaces the ad-hoc grep checks in build-native.sh, which it subsumes. Extend the simulator to open a nextflow.config as well, so both the tracing agent and the comparison exercise the config service and the definitions.json resource. LSP_SIM_WORKSPACE lets the comparison pin the workspace, since document URIs appear in the responses. Declare the generated config directories as nativeCompile inputs. dependsOn alone orders the tasks without invalidating the build, so a metadata change reported UP-TO-DATE and silently reused the previous binary. The config completion probe targets the docker scope, not process: the server's overload resolution for process directives is nondeterministic across runs -- arch alternates between String and Map, as do accelerator, clusterOptions and pod -- which would make the comparison flaky for reasons unrelated to the native image. Binary grows from 74MB to 78MB. Startup is unchanged at ~10ms versus ~400ms for the JVM build.
Records the decision to distribute prebuilt nextflow-lsp binaries alongside the JAR: the options considered, how the reflection metadata is obtained and why it is derived from the classpath rather than traced, how the binary is verified against the JAR, the platform matrix and its CI cost, release packaging, and the risks accepted -- notably that the build depends on a GraalVM flag that is deprecated upstream. Also adds the ADR template the directory was created with.
Consolidate the three helper scripts as native/build.sh, native/simulate.sh and native/verify.py. Remove the release job. It could not have worked -- it collected *.tar.gz while the build uploaded a bare binary, and read the version from refs/tags/ while the workflow ignored tags -- and attaching native binaries to releases is a distribution decision better taken on its own, once something consumes them. The v* tag trigger went with it, since it existed only to fire that job. Keep the tar step: upload-artifact does not preserve the executable bit, so an uploaded bare binary is not runnable when downloaded. Drop the macOS matrix entries. There is no free Intel macOS runner since macos-13 was retired, so macos-intel could only run on the per-minute macos-latest-large, and Apple Silicon adds a second build. Paying for two macOS builds per commit is not justified while nothing consumes the binary. native/build.sh still builds on macOS locally.
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
instructions Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
The session that drives both the tracing agent and the JVM-vs-native comparison left several code paths untouched, and four of its probes compared an empty result against an empty result: - completion on main.nf returned nothing, because member completion on a complete expression has no answer to give - previewDag was passed a process name where the second argument is a workflow name, so it returned null and neither DataflowVisitor nor MermaidRenderer ever ran - documentLink returned nothing, because links come from include statements and the workspace had a single file - documentSymbol on the config file can only ever return nothing, as the config service has no symbol provider So: point completion at positions that resolve (an identifier, the top level, and a member of the lowercase channel namespace), pass null to previewDag for the entry workflow, add a module that usage.nf includes, and drop the config documentSymbol probe. Also cover the handlers that were never exercised: didChange, incoming/outgoingCalls, and the three remaining commands. The comparison goes from 27 messages to 42. That immediately failed, which is the point of it: NoClassDefFoundError: Unable to configure java.nio.file.Path due to missing dependency java.nio.file.WatchEvent$Modifier Groovy configures the Path class node for the Path shim type and enumerates its methods, one of which is register(WatchService, Kind[], Modifier...). Nothing calls it, so native-image left the watch types out of the image entirely and the first completion request that resolved a Path failed with an internal error. Registering that closure fixes it. Also correct the ADR: tracing contributes the reflective entries behind Groovy's indy call sites and five META-INF/services files, not the JSSE provider graph or JDK icu resources -- no config directory registers anything for those. Since nothing exercises the plugin registry, the HTTPS path it needs is covered by neither source, which is now recorded as a residual risk.
bentsherman
force-pushed
the
add-native-image-support
branch
from
August 5, 2026 16:15
8751be1 to
f07b2ed
Compare
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Member
|
I took some time to clean up the PR:
I went ahead and updated the vs code extension to use the native binary ( I still need to figure out the distribution mechanism:
I'm still thinking about the simulate/verify scripts and whether/how they can be improved. Will likely do a few more iterations on this before merging I need to focus on other things now, but the native build is much closer to being mergeable |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add GraalVM native image compilation to deliver pre-built
nlspbinaries for:Build Process
The native image build uses a two-phase approach to handle Groovy's dynamic nature:
Tracing Phase: The
lsp-simulator.shscript simulates an LSP client session, sending JSON-RPC messages that exercise all major LSP operations (initialize, hover, completion, definition, references, formatting, etc.). This runs with the GraalVM tracing agent (-agentlib:native-image-agent) to capture reflection, resource, and serialization metadata.Compilation Phase: The captured metadata is fed to
native-imagealong with manual configuration inconf/native-image/to produce a self-contained binary.Changes
generateNativeImageConfigtask that runs the tracing agentTest plan
nlspbinary responds to LSP initialize request🤖 Generated with Claude Code