feat(providers): ride the SSE wire for LiteLLM generate()

[pdogra1299]

Summary

A non-streaming completion sends no bytes until it is fully done, so a proxy/tunnel idle limit (e.g. Cloudflare's ~100s 524 on tunneled LiteLLM gateways) kills any slow model — this is what fails the Yama review on qwen3.8-27b (see PR #1514). Callers like Yama cannot switch to stream() because their schema-bound contract needs the complete GenerateResult (structuredData, toolExecutions, stopReason, usage, JSON damage flags).

Fix: doGenerate in the OpenAI-compat base can now opt into stream: true on the wire (new useStreamingWireForGenerate() hook, default false; LiteLLM opts in) and aggregates the SSE via the existing parseSSEStream into the exact JSON-wire response shape — everything downstream (content parts, finish-reason mapping, usage clamping, schema coercion) is shared verbatim between the two wires, so callers see zero behavioral change beyond surviving slow completions.

• Request timeout now re-arms on every chunk (idle semantics) instead of capping total duration — createTimeoutController gained reset().
• A backend whose 400 names stream/stream_options gets one retry on the plain JSON wire.
• Escape hatch: NEUROLINK_LITELLM_SSE_GENERATE=false restores the plain JSON wire.

Test plan

• pnpm run check + check:tools-tests — clean
• pnpm run build — clean
• pnpm run test:providers-mocked — 58/58, including 4 new contract tests: stream flags on the wire body, multi-chunk aggregation + usage, schema-bound structuredData from streamed JSON, stream-rejection fallback, and the env escape hatch
• Non-litellm providers unaffected (hook defaults to false; existing mocked contract tests still pass on the JSON wire)

Summary by CodeRabbit

• New Features

  • LiteLLM generation now uses streaming responses by default.
  • Streamed results preserve response details, usage, reasoning, tool calls, and structured output.
  • Streaming automatically retries with a standard request when unsupported.
  • Set NEUROLINK_LITELLM_SSE_GENERATE=false to restore JSON requests.

• Bug Fixes

  • Improved timeout handling during long-running streamed responses.
  • Preserved response IDs and model information from streamed results.

• Tests

  • Added coverage for streaming, usage aggregation, structured output, fallback behavior, and the configuration override.

[github-actions]

✅ Single Commit Policy - COMPLIANT

Status: Policy requirements met • 1 commit • Valid format • Ready for merge

📊 View validation details

📝 Commit Details

• Hash: 4cb712593ddbc96698913c94e1e1d89e74dd135c
• Message: feat(providers): ride the SSE wire for LiteLLM generate()
• Author: Parth Dogra

✅ Validation Results

• Single commit requirement met
• No merge commits in branch
• Semantic commit message format verified
• Ready for squash merge to release branch

---

🤖 Automated validation by NeuroLink Single Commit Enforcement

[coderabbitai]

Warning

Review limit reached

Next included review available in 52 minutes.

View limit details

Limit details: You’ve used all 2 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 8788a2f0-415e-4c68-ad34-a58098054bcd

📥 Commits

Reviewing files that changed from the base of the PR and between d5ecd27 and 4cb7125.

📒 Files selected for processing (6)

• docs/api/type-aliases/OpenAICompatBuildBodyArgs.md
• docs/api/type-aliases/OpenAICompatSSEResult.md
• docs/api/type-aliases/OpenAICompatStreamChunk.md
• docs/api/type-aliases/OpenAICompatStreamLifecycleListeners.md
• docs/api/type-aliases/StreamLoopArgs.md
• docs/api/type-aliases/ToolExecutionSummaryInternal.md

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 65d0e326-bf1c-4647-9d65-fe4759b03b3f

📥 Commits

Reviewing files that changed from the base of the PR and between 1832587 and d5ecd27.

📒 Files selected for processing (6)

• src/lib/providers/litellm/client.ts
• src/lib/providers/openaiChatCompletionsBase.ts
• src/lib/providers/openaiChatCompletionsClient.ts
• src/lib/types/openaiCompatible.ts
• src/lib/utils/timeout.ts
• test/continuous-test-suite-providers-mocked.ts

Included review availability: Your plan provides up to 2 included reviews per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

LiteLLM generation now uses SSE by default. The base provider aggregates streamed responses into the existing result shape, resets idle timeouts per chunk, preserves stream metadata, and retries with JSON when streaming is rejected.

Changes

LiteLLM SSE generation

Layer / File(s)	Summary
Generation wire selection src/lib/providers/litellm/client.ts, src/lib/providers/openaiChatCompletionsBase.ts	LiteLLM opts into SSE generation by default. The environment variable NEUROLINK_LITELLM_SSE_GENERATE=false selects JSON transport.
SSE aggregation and fallback src/lib/types/openaiCompatible.ts, src/lib/utils/timeout.ts, src/lib/providers/openaiChatCompletionsClient.ts, src/lib/providers/openaiChatCompletionsBase.ts	SSE chunks are aggregated into the existing response shape. The parser preserves the first response ID and model. Idle timeouts reset per chunk. Unsupported streaming requests retry once with JSON.
SSE contract validation test/continuous-test-suite-providers-mocked.ts	Mocked tests cover streamed content, usage, structured output, JSON fallback, and the environment-controlled JSON mode.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: ⚪ Minimal · up to d5ecd

The change adds an opt-in SSE transport for LiteLLM while preserving the existing response contract and leaving other providers on the JSON wire; no actionable merge-blocking risk remains after normal checks and review.

Suggested reviewers: murdore

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: enabling SSE transport for LiteLLM generate().
Docstring Coverage	✅ Passed	Docstring check was indeterminate for this PR — some files could not be analyzed in time. Not blocking.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/openai-compat-sse-generate

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🤖 AI Review & Build Compliance ✅

Status: AI analysis complete • Build rules validated • Ready for review

📊 View detailed analysis results

🛡️ Analysis Complete

• ✅ Security scan (vulnerabilities, API keys)
• ✅ TypeScript safety & code quality
• ✅ Error handling & best practices
• ✅ Build rule enforcement validated
• ✅ Commit format & compliance checks

📋 Ready for Merge When

• All CI checks passing
• Manual review approved
• Any AI-flagged issues resolved

---

🤖 AI analysis complete - check individual code comments for specific feedback

[Tara-ag]

Starting review of PR #1519. Analyzing changes file by file following Yama methodology.

[github-actions]

Documentation Validation Results

🚀 Documentation validation passed!

Check	Status	Result
Frontmatter Validation	✅	Passed
TypeScript Check	✅	Passed
Build	✅	Passed
Link Validation	✅	Passed

---

📦 Build artifact uploaded successfully. Ready for deployment preview.

_{Commit: 354d462e32bce19e2489d7e2cf1dfe93a412a231 | Workflow: View logs}

[github-actions]

🤖 AI Review & Build Compliance ✅

Status: AI analysis complete • Build rules validated • Ready for review

📊 View detailed analysis results

🛡️ Analysis Complete

• ✅ Security scan (vulnerabilities, API keys)
• ✅ TypeScript safety & code quality
• ✅ Error handling & best practices
• ✅ Build rule enforcement validated
• ✅ Commit format & compliance checks

📋 Ready for Merge When

• All CI checks passing
• Manual review approved
• Any AI-flagged issues resolved

---

🤖 AI analysis complete - check individual code comments for specific feedback

[Tara-ag]

✅ Approved — The PR implements SSE wire support for LiteLLM's non-streaming generate() method, solving the critical problem of tunnel idle timeouts killing slow completions.

Summary of Changes

This PR adds streaming-capable wire transport for LiteLLM's generate() operation while maintaining full backward compatibility. The key insight: a non-streaming completion sends nothing until complete, so proxies with idle limits (e.g., Cloudflare's ~100s 524) will kill it. By using SSE wire with aggregation, bytes flow continuously while the SDK aggregates into the same complete result callers expect.

What Changed

1. Base Provider (openaiChatCompletionsBase.ts): Added useStreamingWireForGenerate() hook (default false), conditional SSE aggregation in doGenerate(), and 400 fallback retry logic
2. LiteLLM Provider: Overrides hook to true by default, with env var escape hatch
3. Timeout Controller: Added reset() for idle-timeout semantics
4. SSE Client: Enhanced parseSSEStream to capture id/model from first chunk
5. Types: Added optional id/model fields to OpenAICompatSSEResult
6. Tests: 4 new contract tests cover all scenarios

Verification Performed

• ✅ No breaking changes to public API
• ✅ SSE aggregation matches JSON response structure verbatim
• ✅ Idle timeout re-arms on each chunk (verified in code)
• ✅ 400 fallback only retries once when error mentions "stream"
• ✅ Escape hatch works via env var
• ✅ Structured output coercion tested end-to-end
• ✅ Tool call handling preserved (uses existing buffered mechanism)

Impact Analysis

• Affected files: 12 (6 docs auto-generated, 5 source, 1 test)
• Changed APIs: None breaking; internal implementation detail
• Downstream consumers: Zero behavioral change beyond fixing slow completions
• Test coverage: 4 new E2E tests + existing suite still passes

Review Scope

Reviewed all 12 changed files systematically:

• Type additions are additive and documented
• Implementation follows existing SSE/streaming patterns
• Error handling mirrors streaming path
• Tests comprehensively cover happy path, edge cases, and failures

No issues found requiring changes.

[github-actions]

🎉 This PR is included in version 11.23.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀