Skip to content

docs(research): re-measure pi/codex planner streaming — aic not the regressor - #137

Merged
CaicoLeung merged 1 commit into
mainfrom
fix-cli-agents-planner-streaming
Aug 14, 2026
Merged

docs(research): re-measure pi/codex planner streaming — aic not the regressor#137
CaicoLeung merged 1 commit into
mainfrom
fix-cli-agents-planner-streaming

Conversation

@CaicoLeung

Copy link
Copy Markdown
Owner

Why

Reported symptom: "CLI agents like Codex and PI have broken streaming output during the Planner phase." Suspects included the v0.5.1–0.5.3 reasoning-feed refactors (#130, #134, #135).

What this does

Re-ran the streaming attribution research from docs/research-cli-agent-streaming.md with today's CLI versions and a deterministic harness, and records the results there (+18 lines; no code change).

Findings

  1. aic's relay/render path is green. A pty harness drove real aic 0.5.3 (HEAD build and installed Homebrew binary) against a scratch repo with a fake pi emitting 20 tagged thinking_delta events at a fixed 100 ms cadence. Both binaries repainted the reasoning window per delta — 20/20 markers at exact 100 ms spacing, 1.97 s spread. A render regression would show all markers in one late burst.
  2. codex-cli 0.147 is silent by design. Raw probe: thread.started + turn.started instantly, then zero stdout until item.completed. No reasoning events exist in exec --json; the planner window correctly shows only the loading frame.
  3. pi 0.84.2 regressed upstream since the 2026-08 measurement:
    • ~6–8 s of complete stdout silence precedes the first delta on every run (server-side thinking latency).
    • Short thinking phases no longer stream — the whole thinking lands as one ~10 ms burst of deltas (also pre-filled in message_start) right before the text phase, so the window jumps in one chunky dump, exactly like Claude Code's documented batching.
    • Long thinking phases still stream live (618 deltas over 13.3 s on a thinking-heavy prompt). The old "pi is always smooth" claim was prompt-dependent.

Not done (deliberate)

Flipping pi's Encoding::streams_reasoning_live() to false would trade the chunky dump for a plain loading frame, but would also hide pi's still-working live streaming on thinking-heavy runs — left as-is.

…egressor

Planner-phase reasoning chunkiness on codex/pi was reported as an aic
regression. Raw stdout probes plus a deterministic pty harness (fake pi
emitting tagged deltas at fixed 100ms cadence) attribute it upstream:

- aic 0.5.3 repaints the reasoning window per delta (20/20 markers at
  native cadence) — relay/render path green.
- codex 0.147 `exec --json` is silent for the whole reasoning phase by
  design; only the loading frame can show.
- pi 0.84.2 now batches short thinking phases into one ~10ms delta burst
  after 6-8s of silence; long thinking still streams live (618 deltas /
  13.3s). The prior "pi is always smooth" measurement was prompt-dependent.
@CaicoLeung
CaicoLeung merged commit e531fb0 into main Aug 14, 2026
9 checks passed
@CaicoLeung
CaicoLeung deleted the fix-cli-agents-planner-streaming branch August 16, 2026 02:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant