docs(research): re-measure pi/codex planner streaming — aic not the regressor - #137
Merged
Merged
Conversation
…egressor Planner-phase reasoning chunkiness on codex/pi was reported as an aic regression. Raw stdout probes plus a deterministic pty harness (fake pi emitting tagged deltas at fixed 100ms cadence) attribute it upstream: - aic 0.5.3 repaints the reasoning window per delta (20/20 markers at native cadence) — relay/render path green. - codex 0.147 `exec --json` is silent for the whole reasoning phase by design; only the loading frame can show. - pi 0.84.2 now batches short thinking phases into one ~10ms delta burst after 6-8s of silence; long thinking still streams live (618 deltas / 13.3s). The prior "pi is always smooth" measurement was prompt-dependent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Reported symptom: "CLI agents like Codex and PI have broken streaming output during the Planner phase." Suspects included the v0.5.1–0.5.3 reasoning-feed refactors (#130, #134, #135).
What this does
Re-ran the streaming attribution research from
docs/research-cli-agent-streaming.mdwith today's CLI versions and a deterministic harness, and records the results there (+18 lines; no code change).Findings
thinking_deltaevents at a fixed 100 ms cadence. Both binaries repainted the reasoning window per delta — 20/20 markers at exact 100 ms spacing, 1.97 s spread. A render regression would show all markers in one late burst.thread.started+turn.startedinstantly, then zero stdout untilitem.completed. No reasoning events exist inexec --json; the planner window correctly shows only the loading frame.message_start) right before the text phase, so the window jumps in one chunky dump, exactly like Claude Code's documented batching.Not done (deliberate)
Flipping pi's
Encoding::streams_reasoning_live()tofalsewould trade the chunky dump for a plain loading frame, but would also hide pi's still-working live streaming on thinking-heavy runs — left as-is.