Latency when streaming responses

[manibatra]

When using the span of kind LLM and streaming responses I would want the latency to be time for first token. Currently latency is recorded when I end the span. However if I end the span at first token I can't capture the entire output. Ideally I don't want two spans (for e.g a chain to capture time to first token and a child LLM span to capture output). Any solution for this?

[dosubot[bot]]

Hello @manibatra! I'm here to help you with any bugs, questions, or contributions while you wait for a human maintainer. Let's work together to solve the issue you're facing.

To measure the latency for the first token in a streaming response using a single span of kind LLM without ending the span prematurely, you can use the _Span class's process_event method along with the _finalize_streaming_span function from the OpenInference library.

Here's how you can achieve this:

1. Start the Span: Create a span of kind LLM when the streaming process begins.

2. Capture the First Token Timestamp: Use the process_event method to listen for STREAMING_IN_PROGRESS_EVENTS. When such an event is detected, record the timestamp of the first token by checking if _first_token_timestamp is None and then setting it with the current time in nanoseconds.

3. Continue Streaming: Keep the span active while streaming events are in progress to ensure the span is not ended prematurely.

4. Aggregate Output: Use the _finalize_streaming_span function to process the stream asynchronously. This function aggregates the output messages and sets span attributes for the output value and messages.

5. End the Span: Once a STREAMING_FINISHED_EVENTS event is detected, end the span to capture the entire output.

This approach allows you to measure the latency for the first token while keeping the span open to capture the entire streaming output ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[axiomofjoy]

Thanks @manibatra. It makes sense to me to have time to first token as a first-class metric in instrumentation and in the UI. I would argue that latency refers to the total duration of a span and makes sense in its current form.