Length data underflow for multiline token in response for `textDocument/semanticTokens/full`

[onion108]

Version

0.78.5+6061841946fe2237452639750895878768f14e9f

Dotnet Info

.NET SDK:
Version: 9.0.109
Commit: a1e39f97e5
Workload version: 9.0.100-manifests.d3740a48
MSBuild version: 17.12.41+a1e39f97e

Runtime Environment:
OS Name: arch
OS Version:
OS Platform: Linux
RID: arch-x64
Base Path: /usr/share/dotnet/sdk/9.0.109/

.NET workloads installed:
There are no installed workloads to display.
Configured to use loose manifests when installing new manifests.

Host:
Version: 9.0.8
Architecture: x64
Commit: a1e39f97e5

.NET SDKs installed:
6.0.136 [/usr/share/dotnet/sdk]
8.0.119 [/usr/share/dotnet/sdk]
9.0.109 [/usr/share/dotnet/sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 8.0.19 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 9.0.8 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 6.0.36 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.19 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.8 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
None

Environment variables:
Not set

global.json file:
Not found

Learn more:
https://aka.ms/dotnet/info

Download .NET:
https://aka.ms/dotnet/download

Steps to reproduce

• Clone this repostiory: https://github.com/onion108/csharp-language-server/
• Navigate to tests/CSharpLanguageServer.Tests/CompletionTests.fs, using neovim with semantic token enable and also enable trace log
• neovim just freezes after seconds, and in the log the data returned for textDocument/semanticTokens/full has an abnormal data: 4294967290

Details

• Expected behavior: neovim doesn't freeze and all token lengths' data in the response of textDocument/semanticTokens/full should not be greater than the file size
• Actual behavior: the data underflows (4294967290, which is -6 in Int32)

Logs

[DEBUG][2025-09-12 17:12:18] /usr/share/nvim/runtime/lua/vim/lsp/log.lua:151 "rpc.receive" { id = 2, jsonrpc = "2.0", result = { data = { 0, 7, 20, 0, 0, 0, 21, 5, 0, 0, 0, 6, 15, 25, 0, 2, 5, 5, 0, 0, 0, 6, 9, 0, 0, 1, 5, 6, 0, 0, 0, 7, 22, 0, 0, 0, 23, 5, 0, 0, 2, 5, 20, 0, 0, 0, 21, 5, 0, 0, 0, 6, 7, 25, 0, 2, 2, 8, 1, 0, 1, 4, 19, 12, 0, 1, 8, 6, 8, 2048, 0, 9, 17, 12, 0, 0, 18, 20, 8, 0, 2, 4, 6, 8, 2048, 0, 6, 28, 12, 0, 4, 8, 19, 8, 0, 1, 8, 6, 8, 2048, 0, 6, 9, 12, 0, 0, 0, 4294967290, 9, 4, 0, 9, 21, 9, 4, 1, 8, 2, 21, 0, 0, 3, 6, 25, 0, 0, 6, 5, 12, 0, 0, 11, 1, 8, 0, 0, 5, 1, 8, 0, 0, 1, 19, 9, 4, 1, 8, 2, 21, 0, 0, 3, 6, 25, 0, 0, 6, 5, 12, 0, 0, 11, 1, 8, 0, 0, 5, 1, 8, 0, 0, 1, 16, 9, 4, 1, 8, 2, 21, 0, 0, 3, 6, 25, 0, 0, 6, 13, 12, 0, 2, 4, 6, 1, 0, 0, 6, 7, 12, 0, 0, 8, 19, 8, 0, 2, 8, 9, 8, 2048, 0, 12, 6, 8, 2048, 0, 6, 5, 12, 0, 2, 8, 17, 8, 0, 0, 19, 16, 1, 0, 1, 10, 12, 9, 4, 0, 17, 3, 9, 4, 0, 6, 9, 8, 2048, 0, 9, 4, 9, 0, 1, 10, 8, 9, 4, 0, 13, 4, 9, 4, 0, 11, 9, 9, 4, 1, 10, 13, 9, 4, 0, 16, 4, 10, 0, 1, 10, 18, 9, 4, 0, 21, 4, 10, 0, 1, 10, 7, 9, 4, 0, 10, 4, 10, 0, 3, 8, 11, 8, 0, 0, 13, 2, 1, 0, 0, 3, 14, 1, 0, 0, 15, 5, 1, 0, 0, 7, 14, 1, 0, 0, 16, 6, 1, 0, 1, 8, 6, 8, 2048, 0, 6, 8, 12, 0, 0, 36, 17, 8, 0, 2, 10, 11, 8, 0, 1, 6, 4, 10, 0, 0, 6, 2, 1, 0, 0, 2, 3, 10, 0, 0, 4, 2, 8, 0, 1, 8, 6, 1, 0, 0, 6, 7, 12, 0, 0, 8, 2, 8, 0, 0, 2, 13, 9, 4, 1, 8, 6, 1, 0, 0, 6, 8, 12, 0, 0, 9, 2, 8, 0, 0, 2, 1, 9, 0, 0, 1, 12, 9, 4, 0, 12, 7, 9, 0, 1, 8, 6, 1, 0, 0, 6, 9, 12, 0, 0, 13, 2, 8, 0, 0, 2, 1, 9, 0, 0, 1, 5, 9, 4, 0, 5, 7, 9, 0, 2, 12, 11, 8, 0, 0, 14, 2, 8, 0, 0, 2, 6, 9, 4, 0, 7, 2, 21, 0, 0, 3, 3, 25, 0, 0, 3, 8, 12, 0, 0, 14, 1, 8, 0, 0, 5, 1, 8, 0, 0, 1, 6, 9, 4, 0, 7, 1, 21, 0, 1, 14, 11, 8, 0, 1, 10, 4, 10, 0, 0, 8, 8, 12, 0, 1, 10, 4, 10, 0, 0, 5, 4, 8, 0, 1, 12, 6, 1, 0, 0, 6, 9, 12, 0, 0, 10, 4, 8, 0, 0, 4, 6, 9, 4, 1, 12, 6, 1, 0, 0, 6, 8, 12, 0, 0, 9, 4, 8, 0, 0, 4, 1, 9, 0, 0, 1, 6, 9, 4, 0, 6, 7, 9, 0, 1, 12, 6, 1, 0, 0, 6, 8, 12, 0, 0, 9, 4, 8, 0, 0, 4, 1, 9, 0, 0, 1, 13, 9, 4, 0, 13, 7, 9, 0, 1, 12, 6, 1, 0, 0, 6, 8, 12, 0, 0, 9, 4, 8, 0, 0, 4, 1, 9, 0, 0, 1, 4, 9, 4, 0, 4, 7, 9, 0, 1, 12, 6, 1, 0, 0, 6, 9, 12, 0, 0, 10, 4, 8, 0, 0, 4, 11, 9, 4, 0, 13, 4, 10, 0, 1, 12, 6, 1, 0, 0, 6, 9, 12, 0, 0, 10, 4, 10, 0, 0, 5, 18, 3, 0, 0, 18, 7, 3, 0, 0, 9, 4, 8, 0, 0, 4, 5, 9, 4, 1, 12, 6, 1, 0, 0, 6, 9, 12, 0, 0, 10, 4, 10, 0, 0, 16, 4, 8, 0, 0, 4, 9, 9, 4, 1, 12, 6, 1, 0, 0, 6, 9, 12, 0, 0, 10, 4, 10, 0, 0, 16, 4, 8, 0, 0, 4, 11, 9, 4, 1, 12, 6, 1, 0, 0, 6, 9, 12, 0, 0, 10, 4, 10, 0, 0, 6, 4, 8, 0, 0, 4, 17, 9, 4, 1, 12, 6, 1, 0, 0, 6, 8, 12, 0, 0, 9, 4, 8, 0, 0, 4, 1, 9, 0, 0, 1, 16, 9, 4, 0, 16, 7, 9, 0, 1, 12, 6, 1, 0, 0, 6, 8, 12, 0, 0, 9, 4, 8, 0, 0, 4, 1, 9, 0, 0, 1, 8, 9, 4, 0, 8, 7, 9, 0, 1, 12, 6, 1, 0, 0, 6, 7, 12, 0, 0, 8, 4, 8, 0, 0, 4, 1, 9, 0, 0, 1, 4, 9, 4, 0, 4, 7, 9, 0, 2, 16, 12, 8, 0, 0, 14, 14, 1, 0, 0, 17, 6, 8, 2048, 0, 6, 8, 12, 0, 0, 35, 4, 8, 0, 2, 12, 6, 1, 0, 0, 6, 9, 12, 0, 0, 10, 12, 8, 0, 0, 12, 7, 9, 4, 0, 9, 4, 10, 0, 1, 12, 6, 1, 0, 0, 6, 8, 12, 0, 0, 9, 12, 8, 0, 0, 12, 1, 9, 0, 0, 1, 13, 9, 4, 0, 13, 7, 9, 0, 2, 12, 15, 8, 0, 0, 18, 2, 8, 0, 0, 2, 6, 9, 4, 0, 7, 2, 21, 0, 0, 3, 3, 25, 0, 0, 3, 8, 12, 0, 0, 14, 1, 8, 0, 0, 5, 1, 8, 0, 0, 1, 6, 9, 4, 0, 7, 1, 21, 0, 1, 14, 15, 8, 0, 1, 10, 4, 10, 0, 0, 8, 8, 12, 0, 1, 10, 4, 10, 0, 0, 5, 4, 8, 0, 1, 12, 6, 1, 0, 0, 6, 9, 12, 0, 0, 10, 4, 8, 0, 0, 4, 6, 9, 4, 1, 12, 6, 1, 0, 0, 6, 8, 12, 0, 0, 9, 4, 8, 0, 0, 4, 1, 9, 0, 0, 1, 6, 9, 4, 0, 6, 7, 9, 0, 1, 12, 6, 1, 0, 0, 6, 8, 12, 0, 0, 9, 4, 8, 0, 0, 4, 1, 9, 0, 0, 1, 13, 9, 4, 0, 13, 7, 9, 0, 1, 12, 6, 1, 0, 0, 6, 8, 12, 0, 0, 9, 4, 8, 0, 0, 4, 1, 9, 0, 0, 1, 4, 9, 4, 0, 4, 7, 9, 0, 1, 12, 6, 1, 0, 0, 6, 9, 12, 0, 0, 10, 4, 8, 0, 0, 4, 11, 9, 4, 0, 13, 4, 10, 0, 1, 12, 6, 1, 0, 0, 6, 9, 12, 0, 0, 10, 4, 10, 0, 0, 5, 18, 3, 0, 0, 18, 7, 3, 0, 0, 9, 4, 8, 0, 0, 4, 5, 9, 4, 1, 12, 6, 1, 0, 0, 6, 9, 12, 0, 0, 10, 4, 10, 0, 0, 20, 4, 8, 0, 0, 4, 9, 9, 4, 1, 12, 6, 1, 0, 0, 6, 9, 12, 0, 0, 10, 4, 10, 0, 0, 20, 4, 8, 0, 0, 4, 11, 9, 4, 1, 12, 6, 1, 0, 0, 6, 9, 12, 0, 0, 10, 4, 10, 0, 0, 6, 4, 8, 0, 0, 4, 17, 9, 4, 1, 12, 6, 1, 0, 0, 6, 8, 12, 0, 0, 9, 4, 8, 0, 0, 4, 1, 9, 0, 0, 1, 16, 9, 4, 0, 16, 7, 9, 0, 1, 12, 6, 1, 0, 0, 6, 8, 12, 0, 0, 9, 4, 8, 0, 0, 4, 1, 9, 0, 0, 1, 8, 9, 4, 0, 8, 7, 9, 0, 1, 12, 6, 1, 0, 0, 6, 7, 12, 0, 0, 8, 4, 8, 0, 0, 4, 1, 9, 0, 0, 1, 4, 9, 4, 0, 4, 7, 9, 0, 2, 16, 12, 8, 0, 0, 14, 14, 1, 0, 0, 17, 6, 8, 2048, 0, 6, 8, 12, 0, 0, 35, 4, 8, 0, 2, 12, 6, 1, 0, 0, 6, 9, 12, 0, 0, 10, 12, 8, 0, 0, 12, 7, 9, 4, 0, 9, 4, 10, 0, 1, 12, 6, 1, 0, 0, 6, 7, 12, 0, 0, 8, 12, 8, 0, 0, 12, 1, 9, 0, 0, 1, 13, 9, 4, 0, 13, 7, 9, 0, 2, 18, 12, 8, 0, 0, 12, 14, 9, 4, 1, 14, 4, 10, 0, 0, 6, 2, 1, 0, 0, 2, 3, 10, 0, 0, 4, 6, 8, 0, 1, 16, 6, 1, 0, 0, 6, 9, 12, 0, 0, 10, 10, 3, 0, 0, 10, 10, 3, 0, 0, 12, 6, 8, 0, 0, 6, 5, 9, 4, 1, 16, 6, 1, 0, 0, 6, 9, 12, 0, 0, 50, 6, 8, 0, 0, 6, 6, 9, 4, 1, 19, 8, 12, 0, 3, 11, 8, 12, 0, 3, 2, 8, 1, 0, 1, 4, 38, 12, 0, 1, 8, 6, 8, 2048, 0, 9, 17, 12, 0, 0, 18, 20, 8, 0, 2, 4, 6, 8, 2048, 0, 6, 28, 12, 0, 2, 8, 9, 8, 2048, 0, 12, 6, 8, 2048, 0, 6, 5, 12, 0, 2, 8, 17, 8, 0, 0, 19, 16, 1, 0, 1, 10, 12, 9, 4, 0, 17, 3, 9, 4, 0, 6, 9, 8, 2048, 0, 9, 4, 9, 0, 1, 10, 8, 9, 4, 0, 13, 4, 9, 4, 0, 11, 9, 9, 4, 1, 10, 13, 9, 4, 0, 16, 4, 10, 0, 1, 10, 18, 9, 4, 0, 21, 4, 10, 0, 1, 10, 7, 9, 4, 0, 10, 4, 10, 0, 3, 8, 11, 8, 0, 0, 13, 2, 1, 0, 0, 3, 14, 1, 0, 0, 15, 5, 1, 0, 0, 7, 14, 1, 0, 0, 16, 6, 1, 0, 1, 8, 6, 8, 2048, 0, 6, 8, 12, 0, 0, 36, 17, 8, 0, 2, 10, 11, 8, 0, 1, 6, 4, 10, 0, 0, 6, 2, 1, 0, 0, 2, 3, 10, 0, 0, 4, 2, 8, 0, 1, 8, 6, 1, 0, 0, 6, 9, 12, 0, 0, 13, 2, 8, 0, 0, 2, 1, 9, 0, 0, 1, 5, 9, 4, 0, 5, 7, 9, 0, 2, 12, 11, 8, 0, 0, 14, 2, 8, 0, 0, 2, 6, 9, 4, 0, 7, 2, 21, 0, 0, 3, 3, 25, 0, 0, 3, 8, 12, 0, 0, 14, 1, 8, 0, 0, 5, 1, 8, 0, 0, 1, 6, 9, 4, 0, 7, 1, 21, 0, 1, 14, 11, 8, 0, 1, 10, 4, 10, 0, 0, 8, 8, 12, 0, 1, 10, 4, 10, 0, 0, 5, 4, 8, 0, 1, 12, 6, 1, 0, 0, 6, 9, 12, 0, 0, 21, 4, 8, 0, 0, 4, 6, 9, 4, 1, 12, 6, 1, 0, 0, 6, 8, 12, 0, 0, 9, 4, 8, 0, 0, 4, 1, 9, 0, 0, 1, 6, 9, 4, 0, 6, 7, 9, 0, 1, 12, 6, 1, 0, 0, 6, 8, 12, 0, 0, 9, 4, 8, 0, 0, 4, 1, 9, 0, 0, 1, 13, 9, 4, 0, 13, 7, 9, 0, 1, 12, 6, 1, 0, 0, 6, 9, 12, 0, 0, 10, 4, 10, 0, 0, 5, 18, 3, 0, 0, 18, 7, 3, 0, 0, 9, 4, 8, 0, 0, 4, 5, 9, 4, 2, 16, 12, 8, 0, 0, 14, 14, 1, 0, 0, 17, 6, 8, 2048, 0, 6, 8, 12, 0, 0, 35, 4, 8, 0, 2, 12, 6, 1, 0, 0, 6, 9, 12, 0, 0, 10, 12, 8, 0, 0, 12, 7, 9, 4, 0, 9, 4, 10, 0, 1, 12, 6, 1, 0, 0, 6, 8, 12, 0, 0, 9, 12, 8, 0, 0, 12, 1, 9, 0, 0, 1, 13, 9, 4, 0, 13, 7, 9, 0, 2, 11, 8, 12, 0 } } }

Checklist

• I have looked through existing issues to make sure that this bug has not been reported before
• I have provided a descriptive title for this issue
• I have made sure that that this bug is reproducible on the latest version of the package
• I have provided all the information needed to reproduce this bug as efficiently as possible
• I or my company would be willing to contribute this fix

[onion108]

I actually found this bug existing when I'm fixing the same bug in csharp-language-server (see how I fixed it in razzmatazz/csharp-language-server#261)

And also similarly, the problematic code is the same part:

FsAutoComplete/src/FsAutoComplete/LspHelpers.fs

Line 1192 in f644fd5

let tokenLen = uint32 (range.End.Character - range.Start.Character)

Though by playing around in the codebase for a time, I realized it might not be easy to fix since in FsAutoComplete ranges seems to be stores as a start location and an end location, and currently I can't find a way to obtain the true length (unlike the case in csharp-language-server I can just obtain the Length almost without any effort), so maybe some bigger edits need to be made...

[TheAngryByrd]

Unfortunately the Range comes from FSharp Compiler Service (FCS) itself:

FsAutoComplete/src/FsAutoComplete/LspServers/AdaptiveFSharpLspServer.fs

Line 112 in f644fd5

let r = tyRes.GetCheckResults.GetSemanticClassification(range)

There may be some issues with our scrubRanges:

FsAutoComplete/src/FsAutoComplete.Core/Commands.fs

Line 1111 in f644fd5

let scrubRanges (highlights: SemanticClassificationItem array) : SemanticClassificationItem array =

Or possibly we need to operate on FCS ranges and not preconvert them to LSP Ranges fcsRangeToLsp

FsAutoComplete/src/FsAutoComplete/LspServers/AdaptiveFSharpLspServer.fs

Line 119 in f644fd5

struct (fcsRangeToLsp item.Range, ty, mods))

Lastly, possibly we need a guard like other places in computeLine

FsAutoComplete/src/FsAutoComplete/LspHelpers.fs

Lines 1180 to 1190 in f644fd5

	let lineDelta =
	if prev.Start.Line = range.Start.Line then
	0u
	else
	uint32 (range.Start.Line - prev.Start.Line)
	let charDelta =
	if lineDelta = 0u then
	uint32 (range.Start.Character - prev.Start.Character)
	else
	uint32 range.Start.Character

[onion108]

Unfortunately the Range comes from FSharp Compiler Service (FCS) itself:

FsAutoComplete/src/FsAutoComplete/LspServers/AdaptiveFSharpLspServer.fs

Line 112 in f644fd5
let r = tyRes.GetCheckResults.GetSemanticClassification(range)

There may be some issues with our scrubRanges:

FsAutoComplete/src/FsAutoComplete.Core/Commands.fs

Line 1111 in f644fd5
let scrubRanges (highlights: SemanticClassificationItem array) : SemanticClassificationItem array =

Or possibly we need to operate on FCS ranges and not preconvert them to LSP Ranges fcsRangeToLsp

FsAutoComplete/src/FsAutoComplete/LspServers/AdaptiveFSharpLspServer.fs

Line 119 in f644fd5
struct (fcsRangeToLsp item.Range, ty, mods))

Lastly, possibly we need a guard like other places in computeLine

FsAutoComplete/src/FsAutoComplete/LspHelpers.fs

Lines 1180 to 1190 in f644fd5
let lineDelta =
if prev.Start.Line = range.Start.Line then
0u
else
uint32 (range.Start.Line - prev.Start.Line)

let charDelta =
if lineDelta = 0u then
uint32 (range.Start.Character - prev.Start.Character)
else
uint32 range.Start.Character

Checked documentation of FSharp.Compiler.Text.Range and unfortunately it doesn't have a proper Length property or anything like that, so avoid preconverting to LSP ranges won't help. We may have to obtain length of each line in order to get a proper length...