fix(rest): buffer split UTF-8 codepoints across PTY frames

[kishore280]

Summary

A multi-byte UTF-8 character split across two WS stdout/stderr frames was decoded per-frame, turning the incomplete half into U+FFFD. Buffer the incomplete tail per channel, flush it on terminal return, and preserve valid bytes after a genuinely malformed sequence.

Call graph

Before

attach_ws_pump          (rest client · src/boxlite/src/rest/litebox.rs:706)
  └─ 0x01/0x02 branch   ← BUG: from_utf8_lossy(payload) decodes each frame in isolation
       └─ stdout_tx/stderr_tx.send(text) — sends U+FFFD when a codepoint straddles two frames

After

attach_ws_pump                  (rest client · src/boxlite/src/rest/litebox.rs:706)
  ├─ 0x01/0x02 branch
  │    └─ decode_utf8_streaming(&mut stdout_pending/stderr_pending, payload)  (litebox.rs:653)
  │         └─ stdout_tx/stderr_tx.send(text) — only sent once a full codepoint is buffered; malformed bytes are skipped without dropping valid bytes after them
  └─ ControlFrame::Exit / ProbeResult::Terminal terminal returns  (litebox.rs:869, 907)
       └─ flush_pending(&mut stdout_pending/stderr_pending)  (litebox.rs:687)
            └─ stdout_tx/stderr_tx.send(text) — flushes a still-incomplete trailing sequence instead of dropping it silently

Fixes #1155

Changes

• Added decode_utf8_streaming, buffering an incomplete trailing UTF-8 sequence instead of lossily replacing it; on genuinely malformed bytes, only the bad sequence becomes U+FFFD and decoding continues on the rest of the buffer.
• Added flush_pending, called on both terminal-return paths so a partial sequence still in the buffer when the exec ends isn't silently lost.
• Two independent carry buffers (stdout_pending, stderr_pending) persist across reconnects and never mix bytes between channels.
• Tests: 3-byte (€) and 4-byte (😀, all split points) on stdout and stderr, box-drawing (─) and braille spinner (⠋), channel isolation, flush-on-exit, and valid-bytes-survive-malformed-sequence.

How to verify

cargo test -p boxlite --no-default-features --features rest,test-support --lib rest::litebox::tests::ws_std

[boxlite-agent]

📦 BoxLite review — couldn't complete

claude exited 1

stdout:
{"is_error":true,"duration_api_ms":0,"num_turns":1,"stop_reason":"stop_sequence","session_id":"5189b6e1-b8aa-40a8-b144-21e3b469e153","total_cost_usd":0,"usage":{"input_tokens":0,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":0,"server_tool_use":{"web_search_requests":0,"web_fetch_requests":0},"service_tier":"standard","cache_creation":{"ephemeral_1h_input_tokens":0,"ephemeral_5m_input_tokens":0},"inference_geo":"","iterations":[],"speed":"standard"},"modelUsage":{},"permission_denials":[],"terminal_reason":"api_error","fast_mode_state":"off","fast_mode_disabled_reason":"sdk_opt_in_required","subtype":"success","api_error_status":403,"result":"Your organization has disabled Claude subscription access for Claude Code · Use an Anthropic API key instead, or ask your admin to enable access","type":"result","duration_ms":282,"uuid":"9fe09d41-eb05-421b-9baa-188190d3b1ae"}

stderr:
<empty>

_{powered by BoxLite}

[cla-assistant]

All committers have signed the CLA.

[coderabbitai]

📝 Walkthrough

Walkthrough

The REST WebSocket output path now preserves incomplete UTF-8 sequences across binary frames for stdout and stderr. It flushes pending bytes at termination and handles malformed sequences with replacement characters. Tests cover multibyte, TUI, isolated-channel, and terminal-flush cases.

Changes

Streaming UTF-8 output

Layer / File(s)	Summary
Buffered stdout and stderr decoding src/boxlite/src/rest/litebox.rs	The WebSocket pump keeps separate pending-byte buffers for stdout and stderr. It decodes complete UTF-8 fragments across frames and preserves valid text around malformed bytes.
Terminal output flushing src/boxlite/src/rest/litebox.rs	The output path flushes incomplete stdout and stderr sequences before returning normal exit results or status-probe results after disconnects.
Split-character and malformed-sequence coverage src/boxlite/src/rest/litebox.rs	Integration tests cover split 3-byte, 4-byte, and TUI characters, channel isolation, terminal replacement, and malformed UTF-8 handling.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant WebSocket
  participant WebSocketPump
  participant UTF8Buffers
  participant ExecResult
  WebSocket->>WebSocketPump: receive stdout or stderr bytes
  WebSocketPump->>UTF8Buffers: append channel bytes
  UTF8Buffers-->>WebSocketPump: emit decoded fragments
  WebSocketPump->>ExecResult: flush pending bytes at termination

Loading

Possibly related PRs

• boxlite-ai/boxlite#665: Applies a similar streaming UTF-8 preservation fix to attach output in exec.rs.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes address issue #1155 by buffering incomplete UTF-8 sequences independently for stdout and stderr.
Out of Scope Changes check	✅ Passed	The implementation and tests remain focused on preserving UTF-8 output across PTY WebSocket frames.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Title check	✅ Passed	The title clearly and concisely describes buffering split UTF-8 codepoints across PTY frames, which is the primary change.
Description check	✅ Passed	The description includes the required summary, call graph, issue reference, changes, verification command, and relevant implementation details.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/boxlite/src/rest/litebox.rs (1)

849-855: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Flush pending bytes before terminal returns.

An Exit frame returns without decoding stdout_pending or stderr_pending. For example, a final stdout frame containing only 0xE2 is silently lost. The ProbeResult::Terminal return path has the same omission.

Before every terminal return, flush each pending buffer with lossy decoding so incomplete final bytes remain observable as U+FFFD instead of disappearing.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/boxlite/src/rest/litebox.rs` around lines 849 - 855, Update the terminal
return paths for ControlFrame::Exit and ProbeResult::Terminal to flush
stdout_pending and stderr_pending using lossy decoding before sending or
returning the final result. Preserve any already-decoded output and ensure
incomplete trailing bytes are emitted as U+FFFD rather than discarded.

🧹 Nitpick comments (1)

src/boxlite/src/rest/litebox.rs (1)

1620-1795: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add stderr and channel-isolation coverage.

These tests only send channel 0x01 and only read stdout_rx. Add a test that splits UTF-8 on channel 0x02 and reads stderr_rx. Also interleave incomplete stdout and stderr sequences to prove that the two pending buffers never combine bytes across channels.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/boxlite/src/rest/litebox.rs` around lines 1620 - 1795, Add tests
alongside ws_stdout_utf8_split_across_frames and its related cases that send
UTF-8 fragments with channel prefix 0x02 and assert the reassembled text arrives
through stderr_rx. Add an interleaved case with incomplete stdout and stderr
byte sequences, completing each on separate frames, and assert stdout_rx and
stderr_rx each contain only their own decoded character without cross-channel
byte combination.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/boxlite/src/rest/litebox.rs`:
- Around line 653-670: Update decode_utf8_streaming to handle malformed UTF-8 by
consuming the valid prefix and malformed sequence, appending U+FFFD, then
continuing to decode subsequent bytes in the same payload. Preserve valid bytes
after the malformed sequence and retain only an incomplete trailing sequence in
pending; do not clear the entire buffer when error_len() is Some.

---

Outside diff comments:
In `@src/boxlite/src/rest/litebox.rs`:
- Around line 849-855: Update the terminal return paths for ControlFrame::Exit
and ProbeResult::Terminal to flush stdout_pending and stderr_pending using lossy
decoding before sending or returning the final result. Preserve any
already-decoded output and ensure incomplete trailing bytes are emitted as
U+FFFD rather than discarded.

---

Nitpick comments:
In `@src/boxlite/src/rest/litebox.rs`:
- Around line 1620-1795: Add tests alongside ws_stdout_utf8_split_across_frames
and its related cases that send UTF-8 fragments with channel prefix 0x02 and
assert the reassembled text arrives through stderr_rx. Add an interleaved case
with incomplete stdout and stderr byte sequences, completing each on separate
frames, and assert stdout_rx and stderr_rx each contain only their own decoded
character without cross-channel byte combination.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: de40bcd1-d6e7-4593-9b93-c38f232fb618

📥 Commits

Reviewing files that changed from the base of the PR and between 2109bdb and 43afa62.

📒 Files selected for processing (1)

• src/boxlite/src/rest/litebox.rs