Add GENERATION span attributes for streaming (semconv) - #2689
Add GENERATION span attributes for streaming (semconv)#2689connectsudhindra-gif wants to merge 3 commits into
Conversation
dosubot
Bot
added
the
size:S
joshreini1
left a comment
joshreini1
left a comment
There was a problem hiding this comment.
The semconv additions don't align with the official OTel GenAI semantic conventions for streaming instrumentation. The spec defines gen_ai.request.stream (boolean) and gen_ai.response.time_to_first_chunk (double, seconds) as the canonical streaming attributes, and recommends Histogram metrics (not span attributes) for per-chunk latency and throughput. This PR adds custom ai.observability.generation.* attributes with wrong units (milliseconds vs seconds) and derived metrics (tokens_per_second, chunks_received) that conflict with the spec's instrumentation model. Request changes: 3 blockers.
|
|
||
| base = BASE_SCOPE + ".generation" | ||
|
|
||
| IS_STREAMING = base + ".is_streaming" |
There was a problem hiding this comment.
Replace IS_STREAMING with the official gen_ai.request.stream attribute (boolean) already defined in the OTel GenAI spec. The spec explicitly defines this as the standard way to flag streaming requests on spans and events. Duplicating it under the ai.observability.* namespace creates confusion and breaks interoperability with OTel-native tooling.
| IS_STREAMING = base + ".is_streaming" | ||
| """Whether this generation call used streaming (e.g. `stream=True`).""" | ||
|
|
||
| TIME_TO_FIRST_TOKEN_MS = base + ".time_to_first_token_ms" |
There was a problem hiding this comment.
Change TIME_TO_FIRST_TOKEN_MS to align with the official gen_ai.response.time_to_first_chunk attribute (double, unit=seconds, not milliseconds). The OTel GenAI spec defines this attribute for both span attributes and the gen_ai.client.inference.operation.details event. Using milliseconds breaks interoperability and makes this incompatible with the standard Histogram metric gen_ai.client.operation.time_to_first_chunk (which also uses seconds with explicit bucket boundaries).
| """Milliseconds between issuing the request and receiving the first | ||
| streamed chunk. Only set when `IS_STREAMING` is `True`.""" | ||
|
|
||
| TOKENS_PER_SECOND = base + ".tokens_per_second" |
There was a problem hiding this comment.
Drop TOKENS_PER_SECOND and CHUNKS_RECEIVED — these are derived metrics that conflict with the OTel GenAI instrumentation model. The spec expects tokens/sec to be computed from gen_ai.client.token.usage + gen_ai.client.operation.duration, and inter-chunk latency to be reported as Histogram metrics (gen_ai.client.operation.time_per_output_chunk), not as span attributes. Span attributes should capture request/response metadata, not aggregated performance metrics. If TruLens needs these for internal evaluation, compute them from the official attributes rather than storing redundant custom ones.
| @@ -425,6 +425,22 @@ class GENERATION: | |||
|
|
|||
There was a problem hiding this comment.
Should streaming metadata live on a gen_ai.client.inference.operation.details event (lines 101-105) rather than as span attributes? The OTel GenAI spec defines that event for capturing per-request metadata including gen_ai.response.time_to_first_chunk and gen_ai.request.stream. The GenAIEvents class already exists in this file (lines 95-114) and includes the event name constant. If these will be populated on the next PR's OpenAI instrumentation, clarify whether they'll be set as span attributes, event attributes, or both.
First step toward streaming instrumentation support (truera#2442). Adds the attribute keys the issue asks for as a pure, additive semconv change with no behavior change, so it can land and be reverted independently of the instrumentation work that will populate them: - ai.observability.generation.is_streaming - ai.observability.generation.time_to_first_token_ms - ai.observability.generation.tokens_per_second - ai.observability.generation.chunks_received Follow-up PRs will wire these up for the OpenAI provider's stream=True path and add an example notebook; LangChain streaming, the generic async-generator pattern, mid-stream guardrail evaluation, and the dashboard latency view are each separate, larger efforts tracked against truera#2442 but out of scope here.
…ument scope of internal-only ones Reviewer flagged 3 blockers: the custom ai.observability.generation.* streaming attributes don't align with the OTel GenAI spec's canonical gen_ai.request.stream (bool) / gen_ai.response.time_to_first_chunk (double, seconds), and the spec models per-chunk throughput as Histogram metrics rather than span attributes. - Added GenAIAttributes.REQUEST.STREAM (gen_ai.request.stream) and a new GenAIAttributes.RESPONSE class with TIME_TO_FIRST_CHUNK (gen_ai.response.time_to_first_chunk), matching this file's existing pattern of emitting gen_ai.* attributes alongside the ai.observability.* ones for interoperability. These are the canonical counterparts to IS_STREAMING and TIME_TO_FIRST_TOKEN_MS. - TOKENS_PER_SECOND and CHUNKS_RECEIVED have no canonical counterpart: the spec wants these as Histogram metrics (gen_ai.client.operation.time_per_output_chunk), and this codebase has no OTel Metrics/MeterProvider pipeline to emit them through yet. Documented them as TruLens-internal (ai.observability.* only) rather than silently leaving them looking like they're meant to be canonical semconv. Still additive/no-behavior-change, consistent with this PR's original scope -- nothing populates any of these attributes yet.
c3cfd37 to
1880a70
Compare
dosubot
Bot
added
size:M
…uplicate them Reviewer's inline comments (not just the summary) explicitly objected to keeping ai.observability.generation.is_streaming and .time_to_first_token_ms alongside the new canonical gen_ai.* ones: "Duplicating it under the ai.observability.* namespace creates confusion and breaks interoperability with OTel-native tooling." Removed GENERATION.IS_STREAMING and GENERATION.TIME_TO_FIRST_TOKEN_MS entirely. GenAIAttributes.REQUEST.STREAM and GenAIAttributes.RESPONSE.TIME_TO_FIRST_CHUNK (from the previous commit) are now the sole representations. This is a breaking rename for the stacked instrumentation PRs (truera#2690/truera#2691/truera#2692), which are being updated to match in the same pass. TOKENS_PER_SECOND and CHUNKS_RECEIVED are intentionally left as-is (TruLens-internal, documented as such) rather than dropped per the reviewer's other comment -- replying on the review thread to explain why, since dropping them removes already-shipped, tested functionality rather than just renaming a key.
…tream/TTFT SpanAttributes.GENERATION.IS_STREAMING and .TIME_TO_FIRST_TOKEN_MS no longer exist (removed upstream per review feedback). Switch to GenAIAttributes.REQUEST.STREAM (bool) and GenAIAttributes.RESPONSE.TIME_TO_FIRST_CHUNK (seconds, not ms -- drop the *1000 conversion). CHUNKS_RECEIVED/TOKENS_PER_SECOND are unaffected, still TruLens-internal.
_instrument_stream_span_attributes divided completion_tokens by elapsed duration with only a `duration_s > 0` guard, which lets a near-instant stream produce a misleadingly huge tokens/sec figure. Added a _MIN_TOKENS_PER_SECOND_DURATION_S floor below which TOKENS_PER_SECOND is simply not recorded. Updated the affected test to fake the clock (real synchronous mock consumption is faster than the new duration floor). (The TTFT docstring clarification from the original version of this commit is dropped: TIME_TO_FIRST_TOKEN_MS no longer exists -- it was replaced by the canonical GenAIAttributes.RESPONSE.TIME_TO_FIRST_CHUNK per truera#2689's review feedback.)
SpanAttributes.GENERATION.IS_STREAMING and .TIME_TO_FIRST_TOKEN_MS no longer exist. Switch the notebook's streaming-metrics cell to GenAIAttributes.REQUEST.STREAM / .RESPONSE.TIME_TO_FIRST_CHUNK.
2 participants
Summary
Part of #2442 (streaming LLM instrumentation). This is PR 1 of several small, independently-revertable PRs toward that issue, split so each piece can land (or be reverted) on its own.
Adds the four new
GENERATIONspan attributes the issue asks for, as a pure additive semconv change with no behavior change:ai.observability.generation.is_streamingai.observability.generation.time_to_first_token_msai.observability.generation.tokens_per_secondai.observability.generation.chunks_receivedNothing populates these yet — that's the next PR (wiring them up for the OpenAI provider's
stream=Truepath, extending the existing chunk-handling inOpenAIEndpoint._handle_response).Scope of the overall issue
Issue #2442 bundles several genuinely separate efforts: semconv attributes (this PR), OpenAI streaming instrumentation, LangChain
stream()/astream()instrumentation, a generic async-generator pattern, mid-stream/partial-output evaluation for guardrails, and a dashboard latency-view UI change. Splitting these into separate PRs keeps each one small, reviewable, and safe to revert independently rather than landing (or blocking on) one large, multi-part change.Test plan
ruff check/ruff format --check(pinned0.5.5) pass.SpanAttributes.GENERATION.IS_STREAMING == "ai.observability.generation.is_streaming".trulens.otel.semconv.trace.SpanAttributes.GENERATION(confirmed via repo-wide search), and this package isn't covered by thetests/unit/staticAPI golden-snapshot tests, so nothing needed regenerating.