Skip to content

feat(sse): distinguish silent vs stalled upstreams on readiness timeout - #18586

Open
jjxyxsjr wants to merge 2 commits into
cheaper-inference:release/v3.8.53from
jjxyxsjr:feat/zero-byte-readiness-signal
Open

jjxyxsjr wants to merge 2 commits into
cheaper-inference:release/v3.8.53from
jjxyxsjr:feat/zero-byte-readiness-signal

Conversation

@jjxyxsjr

@jjxyxsjr jjxyxsjr commented Oct 30, 2026 •

Copy link
Copy Markdown

Problem and behavior

The stream-readiness watchdog reports one undifferentiated timeout: "no non-ping SSE event within the window". But two very different failure modes hide behind that message — a connection that never delivered any byte (dead path / silent upstream) versus first bytes (e.g. pings) flowing while content stalls before the first non-ping event (slow-but-alive upstream). When triaging TTFT incidents, operators currently cannot tell a network-side death from a provider-side stall without packet capture.

Changes

open-sse/utils/streamReadiness.ts — pure telemetry, no behavioral change:

  • The readiness result now carries anyByteAt: a timestamp when any byte (pings included) was observed, or an explicit null when the window expired with zero bytes ever — confirmed silence, not "unknown".
  • A response-header carrier exposes the same signal to the SSE layer; the timeout log lines distinguish the two shapes ("zero bytes ever" vs "stalled after first byte").
  • All state is per-call scoped (no module-level timestamps), so concurrent streams cannot contaminate each other's signal.

Validation

tests/unit/stream-readiness-zero-byte-signal.test.ts (new, 5 tests) pins: null on confirmed zero bytes; a timestamp once any byte arrives; the header carrier round-trip; per-call isolation between concurrent streams; and the default-path zero-change property. Adjacent stream-readiness suites (55 tests) green · tsc exit 0 · repo npm run lint exit 0 · changelog fragment 18586-zero-byte-readiness-signal.md included.

Limitation: telemetry only — no retry/timeout decision consumes the signal in this PR; wiring it into policy (e.g. zero-byte = terminate-and-never-resend vs stall = rotate) is deliberately left for a follow-up once the observability lands.

Notes

  • Default path is byte-identical: the fields are additive on the result object and the log wording change is the only visible delta.
  • No new env, no dependencies, no schema, no UI.
  • Base: release/v3.8.53 (active release branch); aware of the 3.8.55 v4-feature retarget policy.

A STREAM_READINESS_TIMEOUT currently looks identical whether the upstream
went fully silent (zero raw bytes) or stayed transport-alive (keepalive
pings, no content event). Surface the distinction as pure telemetry:
anyByteAt on the timeout result (null vs epoch-ms), the same value in an
X-OmniRoute-Any-Byte-At response header for external tooling, and a
grep-able log-line suffix. No behavior reads the signal.

The stamp is strictly per-call — a module-level version let one stream's
chunks mark concurrent streams as alive (observed in production), which
the dedicated per-call tests pin.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant