feat(server): OpenAI Responses API layer, /v1/metrics, cached-token accounting

[Iwctwbh]

Summary

• POST /v1/responses: stateless compatibility layer over the existing chat dispatch path. Supports text/message input items, custom function tools and tool_choice, non-streaming JSON, and typed SSE streaming (response.created / output_item.added / output_text.delta / function_call_arguments.* / response.completed). OpenAI-hosted built-in tools (web search, file search, computer use) and store=true are rejected with explicit 400 errors.
• Delta::ToolCallStart: emitted once the function name is decoded — dozens of tokens before the arguments finish — so streaming TTFT lands on the real decode boundary instead of attributing all tool-call tokens to one terminal frame. The Chat Completions wire format is unchanged.
• GenStats.n_cached: splits KV-prefix reuse from freshly prefilled tokens. usage.prompt_tokens now reports the full prompt (OpenAI semantics), with the cached share in prompt_tokens_details.cached_tokens (chat) / input_tokens_details.cached_tokens (responses). Interval prefill-TPS accounting excludes cached tokens.
• GET /v1/metrics (+ /metrics, auth-gated): request totals, last-request TTFT / prefill / decode TPS, slot occupancy, and GPU/host pager hits, misses, evictions, and hit rates.

Notes

• Both endpoints reuse the unified dispatch_chat path; Responses SSE sequence numbers continue across error frames.
• Verified: cargo check --workspace --all-targets, cargo test -p infr-server -p infr-chat (92 + 59 tests pass), cargo fmt --check clean.

[Headmaster218]

hanks for the PR. The cached-token accounting fix for Issue #5 looks correct, and all tests/checks pass. However, I don’t think the full PR is ready to merge as-is.
Main issues:

• previous_response_id and conversation are silently ignored, causing multi-turn requests to lose context. Unsupported stateful fields should return 400.
• Unsupported content such as input_image and input_file is silently dropped instead of rejected.
• truncation="auto" is accepted but not implemented.
• /metrics calculates cached-request prefill TPS using total prompt tokens, producing inflated results.
• The independent relaxed atomics for last_request can produce mixed snapshots under concurrency.
• Responses output hardcodes request metadata such as tools, tool_choice, temperature, and top_p.
  The PR also bundles cached-token accounting, Responses API, metrics, and ToolCallStart into one large change. Please split these concerns, with the Issue OpenAI API reports only uncached prompt tokens, causing incorrect token usage in Hermes and other harnesses #5 cached-token fix submitted separately first. That part should be straightforward to merge.

[Iwctwbh]

Thanks for the detailed review. I have extracted the cached-token accounting into a focused PR: #7.

One clarification: I developed this accounting independently as part of the Responses API and metrics work in #6. I did not pick up or claim Issue #5, and the original PR was not created in response to that issue. I understand that the resulting behavior overlaps with what #5 describes, but there was no intended issue linkage.

I agree with the remaining review points and will keep the follow-up changes separated by concern:

• ToolCallStart as an independent streaming-boundary change;
• Responses API with explicit 400 responses for unsupported stateful fields and content, no acceptance of unimplemented truncation="auto", and request metadata reflected accurately in the response;
• metrics with cached tokens excluded from prefill throughput and a coherent last_request snapshot under concurrency.

I’ll keep #6 as the original discussion context while preparing those focused replacements.

[Headmaster218]

Sounds good. Please continue with the focused follow-up PRs. We’ll leave #6 open as the original discussion context.

[Iwctwbh]

The focused Responses API follow-up is ready: #32.

It was rebuilt on the current main branch rather than split mechanically from the old #6 commit. The review findings from #6 are addressed: unsupported stateful fields and non-text content return 400, truncation="auto" is rejected, and response metadata reflects the effective request. The SSE text/function-call lifecycle was also checked against the official OpenAI Responses schema and the local DeepSeek Harness fixture.

Metrics and ToolCallStart remain separate and are not included in #32.