Skip to content

fix(hermes): preserve provider response IDs#247

Draft
sipercai wants to merge 3 commits into
mainfrom
fix/hermes-provider-response-id
Draft

fix(hermes): preserve provider response IDs#247
sipercai wants to merge 3 commits into
mainfrom
fix/hermes-provider-response-id

Conversation

@sipercai

@sipercai sipercai commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR makes the Hermes instrumentation prefer the model provider's operation/request identifier for gen_ai.response.id instead of the synthetic response identifier produced by Hermes.

It adds a shared util-genai helper for provider-first response-ID extraction with framework fallback, then uses that helper in Hermes as the first consumer. Hermes observes request-local OpenAI-compatible chat.completions.create responses and streaming chunks, isolates retry attempts across worker threads, propagates the selected identifier to LLM and aggregate AGENT spans, and falls back to the Hermes response identifier when the provider does not expose one.

For providers such as DashScope, request_id intentionally has higher priority than the OpenAI-compatible completion id. A later streaming usage trailer carrying request_id can therefore replace an earlier chunk id. Transport-only metadata such as _request_id and HTTP headers is not read implicitly.

The Hermes package is also aligned with the OpenTelemetry release set used by LoongSuite 0.7.0: compatible-release floors of API/SDK 1.39.1 and instrumentation/semantic-conventions 0.60b1, with the exact set pinned in tests.

Fixes # (N/A)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

  • tox -e precommit
  • pyright util/opentelemetry-util-genai/src/opentelemetry/util/genai/response_id.py
  • pytest -q util/opentelemetry-util-genai/tests/test_response_id.py instrumentation-loongsuite/loongsuite-instrumentation-hermes-agent/tests/test_telemetry_spec.py
  • Built both changed wheels and installed them together in a fresh Python 3.11 environment using standard PyPI dependency resolution.

Does This PR Require a Core Repo Change?

  • Yes. - Link to PR:
  • No.

Checklist:

See contributing.md for styleguide, changelog guidelines, and more.

  • Followed the style guidelines of this project
  • Changelogs have been updated
  • Unit tests have been added
  • Documentation has been updated

Validation Evidence

Spec and Scope

  • Approved behavior: prefer a provider-returned request_id, id, or response_id; fall back to the Hermes response ID only when no provider value is available.
  • Shared abstraction: normalize, extract, and resolve response IDs while keeping field priority caller-configured.
  • First consumer: Hermes OpenAI-compatible chat completions, including synchronous, streaming, retry, tool-loop, and worker-thread paths.
  • Non-goals: bulk migration of other instrumentations, Responses API, Anthropic Messages API, and transport-header extraction.

Local Checks

Check Result Evidence
Static readiness pass LoongSuite PR readiness checker reported no blocking findings.
Precommit pass Ruff, Ruff format, uv-lock, and rstcheck passed.
Pyright pass The changed response_id.py reports 0 errors, 0 warnings, and 0 information messages.
Focused tests pass 57 response-ID and Hermes telemetry tests passed.
Dependency resolution pass Fresh wheel installation resolved API/SDK 1.39.1 and instrumentation/threading/semantic-conventions 0.60b1; package compatibility check passed.
Independent Qoder review pass Two review rounds completed with no blocking defects; the streaming trailer priority edge identified in review was fixed and covered by a focused test.
Privacy scan pass No credentials, private trace IDs, local user paths, or generated telemetry artifacts are included in the diff.

Real E2E Matrix

Scenario Status Evidence
non-streaming pass An isolated Hermes 0.19.0 run through DashScope exported the provider identifier on both LLM and AGENT spans.
streaming pass Streaming exported a provider identifier with TTFT and no Hermes stream-* fallback.
concurrency pass Two simultaneous calls produced distinct provider identifiers without cross-call contamination.
agent/tool/ReAct pass A real tool loop produced two LLM calls around one TOOL call; AGENT retained the final provider identifier.
tool-heavy N/A Response-ID capture is provider-call scoped; the two-LLM tool loop exercises aggregation, while multiple tool definitions do not change the capture path.
error path pass A bounded connection-error run emitted ERROR LLM/STEP spans without inventing a provider response ID.

Telemetry and Weaver

Check Status Evidence
Span tree / span kinds pass The isolated run captured 45 spans locally and the backend readback returned the same 45 spans across ENTRY, AGENT, STEP, LLM, and TOOL kinds.
Content capture modes pass Response-ID selection remains independent of message-content capture; no-content behavior remains covered by focused tests.
Concurrency isolation pass Deterministic threaded tests and the real two-call run both kept response IDs invocation-local.
Weaver live-check pass Eight real-provider LLM samples were filtered to remove content and trace identifiers, then passed the LoongSuite GenAI advice profile with no violations.

CI

  • The previous remote head passed all checks except typecheck, which reported the two Pyright Unknown-type errors fixed by this update.
  • The PR remains a draft while the fresh full-matrix checks run for this commit.
  • The full local tox typecheck environment was not used as final evidence because it bootstraps large unrelated SDK packages; the changed file passed Pyright directly, and GitHub CI remains the authoritative full typecheck gate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants