Skip to content

fix: parse LSP messages by byte length to fix multi-byte UTF-8 (CI + real-Zed verified) - #11

Merged
liuyanghejerry merged 7 commits into
mainfrom
fix/utf8-byte-based-lsp-parser
Jul 5, 2026
Merged

fix: parse LSP messages by byte length to fix multi-byte UTF-8 (CI + real-Zed verified)#11
liuyanghejerry merged 7 commits into
mainfrom
fix/utf8-byte-based-lsp-parser

Conversation

@liuyanghejerry

Copy link
Copy Markdown
Owner

Problem

CI (Test LSP Server job) was failing: two tests in lib/data-parser.test.js under UTF-8 byte length handling — the parse callback was never invoked.

Root cause: LSP Content-Length is defined in bytes, but data-parser.js had been reverted to parse by character (stdinBuffer = '', substring). Any message containing multi-byte UTF-8 (e.g. textDocument/didOpen for an .ets file with Chinese content) is mis-sliced — the parser grabs too many characters, pulls the next message's header into the JSON, and JSON.parse throws. The document is silently dropped, so the language server never receives it → dead LSP for those files. The tests asserted the correct byte-based behavior, so they (correctly) failed.

This has been reverted the wrong way several times (c6cee0c1083e2da88ed44, plus 3b04512/052b8be). Each revert "worked" only because LSP startup traffic is ASCII.

Fix

lib/data-parser.js — restore Buffer-based parsing (Content-Length counted in bytes) with all three pieces the earlier reverts never had together:

  1. Buffer buffer (Buffer.alloc(0), Buffer.concat, subarray, .toString('utf8'));
  2. keep process.stdin.setEncoding('utf8') in index.js (its StringDecoder reassembles multi-byte chars split across TCP chunks);
  3. a string→Buffer guard in parse() (Buffer.isBuffer(data) ? data : Buffer.from(data, 'utf8')) — because setEncoding yields strings and Buffer.concat([buf, string]) throws.

index.js — harden the initialize handler to forward the request as-is instead of crashing with Cannot read properties of undefined (reading 'tsdk') when initializationOptions isn't configured.

Verification

  • Unit + integration + formatting: all 63 tests pass.
  • Real Zed editor driving the real @arkts/language-server, opening a Chinese-heavy .ets file:
    • char-based → the 4698-char didOpen is dropped; wrapper logs Error parsing message: Unexpected non-whitespace character after JSON.
    • byte-based (this PR) → didOpen forwarded intact, 0 parse errors, no crash.

🤖 Generated with Claude Code

liuyanghejerry and others added 7 commits July 4, 2026 19:15
LSP Content-Length is defined in bytes, but data-parser.js parsed by
character. Any message containing multi-byte UTF-8 (e.g. a didOpen for an
.ets file with Chinese content) was mis-sliced: the parser grabbed too
many characters, pulling the next message's header into the JSON and
failing to parse — silently dropping the document so the language server
never received it, leaving the LSP dead for those files.

Restore Buffer-based parsing (Content-Length counted in bytes). Keep
setEncoding('utf8') in index.js and add a string->Buffer guard in parse()
so it works whether stdin yields strings or Buffers — the combination the
earlier reverts (a88ed44, 052b8be) never had together.

Verified in a real Zed editor against the real @arkts/language-server:
char-based drops the multi-byte didOpen; byte-based forwards it intact
with zero "Error parsing message" occurrences.

Also harden the initialize handler: forward the request as-is instead of
crashing with "Cannot read properties of undefined (reading 'tsdk')" when
initializationOptions is not configured.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Zed sends the lowercased language name ("arkts language") as the didOpen
languageId when no language_ids mapping is set. @arkts/language-server only
activates its ETS plugin for languageId === "ets" (and ts/js/json ids), so
every document was silently treated as plain text: definition, hover and
documentHighlight all returned empty results with no error. This was the
root cause of "go to definition does nothing" in Zed.

Verified end-to-end in real Zed: with the mapping, textDocument/definition
returns the correct LocationLink and the editor navigates to it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Zed only passes initializationOptions when the user configured
lsp.arkts-language-server.initialization_options in settings; without them
the server cannot finish initialize (it fails loading TypeScript) and Zed
reports "Failed to start language server". Fall back to env vars
(ZED_ETS_TSDK/TSDK, ZED_ETS_OHOS_SDK_PATH/OHOS_SDK_PATH) and then
auto-detect the tsdk from the ohos-typescript package installed next to
@arkts/language-server, so the server starts out of the box. A missing
ohosSdkPath now degrades ArkUI typings instead of blocking startup.

Also: swallow the response to the wrapper-injected
ets/waitForEtsConfigurationChangedRequested request instead of forwarding
it to the editor (which never issued it), and drop stdin setEncoding so
the byte-based parser receives raw Buffers end to end (chunk splits inside
multi-byte characters are covered by data-parser tests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…iers)

DefTest.ets places a resolvable reference at position 0:0 so a freshly
opened editor can exercise textDocument/definition without moving the
cursor; ChineseTest.ets covers multi-byte UTF-8 content including a
Chinese identifier as the definition target.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- step 6 was missing the compiled tree-sitter grammar wasm; without
  grammars/arkts.wasm Zed fails to load the language entirely (no
  highlighting, no LSP). Document compiling it with Zed's cached wasi-sdk.
- the Force* editor actions were never registered in element.rs, so the
  documented commands could not fire; document the registration step.
- document the new ZED_AUTO_CMD_FILE automation channel and the two ways
  to bypass the worktree-trust dialog for headless runs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
v1.3+ refuses to initialize unless ets.sdkPath is an existing directory
containing ets/build-tools/ets-loader/tsconfig.json (v1.2 accepted any
value and merely warned). When no ohosSdkPath is configured, create a
minimal placeholder skeleton under the OS temp dir instead of passing a
nonexistent path, so the server still starts on both v1.2 and v1.3;
ArkUI typings stay degraded until a real SDK path is configured.

Verified against @arkts/language-server 1.3.10: initialize succeeds and
definition/hover return correct results with the placeholder skeleton.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bump zed-ets-language-server to 3.0.0 and the extension to 0.3.0, and pin
LANGUAGE_SERVER_VERSION to "3" so each extension release states exactly
which wrapper major it is compatible with.

Rework the update check to compare within the pinned major instead of
against npm's overall latest: the old logic reinstalled the wrapper on
every startup whenever npm's latest belonged to a different major than
the pin (installed could never equal latest), adding a network round trip
to every language server start. Now a wrapper of the right major is kept
as-is unless a newer release exists within that same major, and a failed
latest-version lookup (e.g. offline) no longer prevents startup when a
compatible wrapper is already installed.

Note: publish zed-ets-language-server@3.0.0 to npm before shipping
extension 0.3.0, otherwise fresh installs have no 3.x to download.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@liuyanghejerry
liuyanghejerry merged commit ebab416 into main Jul 5, 2026
4 checks passed
@liuyanghejerry
liuyanghejerry deleted the fix/utf8-byte-based-lsp-parser branch July 5, 2026 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant