Robustness: make the provider-invocation output cap UTF-8-byte-accurate and surrogate-safe (capOutput)

[qwen-code-dev-bot]

sourceType

source:self-discovery — undirected maintainer review pass. Same defect class as the output-cap hardening series (#830/#832/#834/#836/#838), in a site that series never touched.

sourceLinkOrEvidence

Concrete evidence on main @ 28c180d. The provider-response output cap in src/provider-invocation.ts:241-244 treats UTF-16 code units as bytes and slices by code units:

function capOutput(text: string, maxBytes: number): { text: string; capped: boolean } {
  if (text.length <= maxBytes) return { text, capped: false };
  return { text: text.slice(0, maxBytes), capped: true };
}

It is applied to the raw provider response content at src/provider-invocation.ts:163 (const text = choice?.message?.content ?? ""; const capped = capOutput(text, opts.maxOutputBytes);). Provider responses routinely contain multi-byte text (CJK, emoji), so both defects are reachable:

1. Byte under-count — text.length counts UTF-16 code units, not UTF-8 bytes. For multi-byte text the real byte count can exceed maxBytes by up to ~3×, so the advertised byte cap does not bound bytes.
2. Surrogate split — text.slice(0, maxBytes) cuts at a code-unit offset and can orphan half of a surrogate pair, leaving a broken trailing character in the capped report.

Minimal reproduction (mirrors capOutput exactly):

capOutput("a".repeat(999) + "🚀", 1000) → capped=true, result ends in a lone high surrogate
capOutput("你".repeat(1000), 1500)      → capped=false, but the text is 3000 UTF-8 bytes (> the 1500-byte cap)

Observed output: surrogate-split: capped= true last code unit is lone high surrogate= true len= 1000 and byte-undercount: capped= false … actual bytes= 3000. The codebase already accounts bytes correctly elsewhere (Buffer.byteLength in src/headless-protocol.ts:143, src/desktop/service.ts:1184, src/tui-shell.ts:3216) and exposes a surrogate-safe byte-budgeted cut (safeByteCutEnd in src/text-cut.ts), so the fix is idiomatic.

problemStatement

The provider-invocation output cap is named, documented, and reported as a byte cap (maxOutputBytes, "…-byte output cap"), but capOutput compares String.length (UTF-16 code units) to the byte budget and slices by code units. For a multi-byte provider response the byte budget is silently exceeded (up to ~3×), and when the cap boundary lands inside a surrogate pair the capped text ends in an unpaired surrogate. The capped response is rendered into the invocation report, so the user can be shown a broken trailing character, and the cap does not provide the memory bound it advertises.

userValue

Capped provider responses keep non-ASCII text intact (no broken trailing glyph) and the byte cap actually bounds UTF-8 bytes, consistent with the hook/tool/MCP/shell output caps already hardened in #830/#832/#834/#836/#838. Removes a user-visible corruption and restores the advertised memory bound on the provider-invocation path.

scope

• Make capOutput (src/provider-invocation.ts) UTF-8-byte-accurate and surrogate-safe: decide "fits" via Buffer.byteLength(text, "utf8") <= maxBytes and cut via safeByteCutEnd from src/text-cut.ts (which honors the byte budget and never orphans a surrogate).
• Preserve the existing { text, capped } contract and the output-capped outcome/reason wording; in-range ASCII output must be unchanged.
• Add regression tests covering a surrogate straddling the cap, a multi-byte text whose bytes exceed the cap while its code units do not, and an in-range ASCII case.

nonGoals

• Not touching the hook/tool/MCP/shell output caps (already covered by Robustness: make the hook output-size cap surrogate-safe (spawnHookRunner) #830/Robustness: make the tool-invocation output-size cap surrogate-safe (spawnCommandRunner) #832/Robustness: account subprocess output caps in UTF-8 bytes, not UTF-16 code units (hook/tool/MCP) #834/Robustness: reassemble multi-byte UTF-8 split across chunks in the hook/tool/MCP output caps #836/Robustness: reassemble multi-byte UTF-8 in the shell tool's capped output (runShellCommand) #838).
• Not changing the cap default (DEFAULT_MAX_OUTPUT_BYTES), the output-capped exit code, or the report format beyond the capped text itself.
• Not introducing streaming/decoder machinery (the provider response is a single buffered string here, so a one-shot byte-safe cut is sufficient).
• Desktop, Dynamic Workflow, or governance surfaces.

acceptanceCriteria

• A provider response whose cap boundary falls inside a surrogate pair is capped without leaving an unpaired surrogate (no [\uD800-\uDBFF] at the end of the capped text).
• A multi-byte response whose UTF-8 byte count exceeds maxBytes is reported capped=true even when its UTF-16 length is <= maxBytes, and the capped text's Buffer.byteLength(…, "utf8") is <= maxBytes.
• In-range ASCII output is byte-for-byte unchanged (capped=false, text identical).
• Regression tests cover the surrogate-straddle, byte-overrun, and in-range ASCII cases; existing tests pass.

testPlan

Behavior-sensitive unit tests in tests/unit/provider-invocation.test.ts alongside the existing ASCII "caps oversized output" case: drive the runner (or the capped path) with a multi-byte response whose surrogate pair straddles the cap and assert no unpaired surrogate; with a CJK response whose bytes exceed the cap and assert capped=true and byteLength <= maxBytes; and assert the ASCII in-range case is unchanged. Deterministic string/byte assertions.

dogfoodPlan

Run the capped path with a multi-byte (emoji + CJK) provider response over the byte cap and confirm the report shows intact characters (no U+FFFD/lone-surrogate glyph) and the cap is honored in bytes; run the provider-invocation unit suite to confirm the regression cases hold.

riskAndSecurityNotes

Low risk: pure string-capping helper; the only behavior change is that multi-byte responses are capped at a true byte boundary without splitting characters. In-range ASCII output is unchanged. No state mutation, no privilege change. The credential is never part of the capped body (already redacted downstream when the report is built).

duplicateSearchEvidence

• gh search issues for capOutput, provider output cap surrogate, provider-invocation UTF-8 → no results. output-capped matches only Robustness: reassemble multi-byte UTF-8 in the shell tool's capped output (runShellCommand) #838/fix(cli): reassemble multi-byte UTF-8 in the shell tool's capped output (Issue #838) #839 (shell tool) and Robustness: make the hook output-size cap surrogate-safe (spawnHookRunner) #830/Robustness: make the tool-invocation output-size cap surrogate-safe (spawnCommandRunner) #832 (hook/tool caps).
• The output-cap series scoped to other sites: Robustness: account subprocess output caps in UTF-8 bytes, not UTF-16 code units (hook/tool/MCP) #834 (hook/tool/MCP subprocess byte accounting), Robustness: reassemble multi-byte UTF-8 split across chunks in the hook/tool/MCP output caps #836 (hook-contract.ts, tool-invocation.ts, mcp-invocation.ts), Robustness: reassemble multi-byte UTF-8 in the shell tool's capped output (runShellCommand) #838 (runShellCommand in src/tools.ts), Robustness: make the hook output-size cap surrogate-safe (spawnHookRunner) #830/Robustness: make the tool-invocation output-size cap surrogate-safe (spawnCommandRunner) #832 (hook/tool spawn caps). None touched src/provider-invocation.ts.
• Verified src/provider-invocation.ts imports neither text-cut nor any safeByteCutEnd/safeCutEnd, and its git history shows no encoding fix.
• The existing tests/unit/provider-invocation.test.ts "caps oversized output" case uses ASCII flood text only; no multi-byte/surrogate coverage.
  Conclusion: distinct, unfixed, untested site — not a duplicate of the Robustness: make the hook output-size cap surrogate-safe (spawnHookRunner) #830–Robustness: reassemble multi-byte UTF-8 in the shell tool's capped output (runShellCommand) #838 series.

parentChildRelationship

Standalone source:self-discovery robustness Issue (not a child of any roadmap parent). Complements the closed output-cap hardening series by covering the provider-invocation cap it did not.

dependencyOrder

No blocking dependencies; executable immediately (capOutput and safeByteCutEnd both already exist on main @ 28c180d). Single vertical slice: make one function byte-accurate and surrogate-safe, plus focused regression tests.

[github-actions]

Automated intake triage

• Type: bug
• Priority: priority:p2
• Confidence: 0.95

The provider-invocation output cap should measure and truncate UTF-8 bytes safely so multi-byte text is bounded correctly and surrogate pairs are not split.

Reported evidence (unverified)

• capOutput compares String.length to maxBytes and slices by UTF-16 code units
• multi-byte provider responses can exceed the advertised byte cap by up to about three times
• cutting by code-unit offset can leave an unpaired surrogate at the end of capped output

Repository verification

Pending. Repository verification is still required before this Issue can become agent-ready.

This bounded pass classifies intake only. It cannot execute code, close the Issue, or grant governance authority.

[qwen-code-dev-bot]

🤖 Leased by qwen-code-dev-bot (autonomous loop, priority 4 — highest-priority dependency-ready agent-ready; the Desktop/Dynamic-Workflow subtasks at equal or higher priority are blocked by open governance blockers #74/#106/#133/#142). Branch: issue/848-capoutput-utf8. Starting the train: make capOutput in provider-invocation.ts UTF-8-byte-accurate and surrogate-safe via safeByteCutEnd, red→green regression tests, gates, PR, merge, dogfood.

[qwen-code-dev-bot]

Evidence (append-only) — implementation complete, PR #849 open

• capOutput (src/provider-invocation.ts:245-251) now decides "fits" via Buffer.byteLength(text,"utf8") <= maxBytes and cuts via safeByteCutEnd(text, maxBytes) — UTF-8-byte-accurate and surrogate-safe, matching the byte-cap convention and the text-cut helpers.
• Red→green: with the src reverted to the buggy form, the two new tests failed (surrogate-straddle on the lone-surrogate/byte assertions; CJK with expected 'called' to be 'output-capped'); with the fix both pass. Full provider-invocation suite 24/24.
• Gate: typecheck ✓, build ✓ (desktop bundle), unit 271 files/4113 passed (+2), integration 158 files/956 passed (clean re-run; an initial failure was a stale-dist race from building concurrently with the suite), smoke 54 passed, npm audit --omit=dev 0 vulns. Desktop E2E leg runs only in CI (no xvfb locally).
• Regression tests drive the real openaiProviderRunner over the HTTP mock server (new flood-emoji / flood-cjk models).

PR: #849

[qwen-code-dev-bot]

Evidence (append-only) — merged & dogfooded; releasing lease

Merged to main via PR #849 (commit 7039855); CI verify (incl. desktop E2E leg) + protected-paths both green.

Post-merge dogfood against the rebuilt artifact (dist safeByteCutEnd, replicating the merged capOutput):

• emoji-straddle ("a".repeat(999)+"🚀"+…, cap 1000) → capped=true, 999 bytes, no lone surrogate
• cjk-overrun ("你".repeat(600), cap 1000) → capped=true, 999 bytes (byte budget honored)
• in-range ASCII (500 chars, cap 1000) → capped=false, unchanged

Issue closed by the merge (Closes #848). Releasing the agent-active lease; deleting the feature branch.