feat(genai): align util telemetry with upstream semantic conventions

[sipercai]

Description

This PR aligns LoongSuite's existing opentelemetry-util-genai 0.3 lifecycle and LoongSuite adapters with the current official OpenTelemetry GenAI semantic conventions.

It:

• aligns LLM stream, response ID, reasoning, prompt, compacted-conversation, Agent, Retrieval, and Memory attributes;
• records gen_ai.response.time_to_first_chunk as double seconds independently from the LoongSuite gen_ai.response.time_to_first_token integer nanoseconds;
• uses integer top_k, CLIENT Retrieval spans, and upstream Memory operation names and attributes;
• treats Retrieval query/documents and Memory query/records as opt-in content attributes, omitting them under NO_CONTENT;
• retains LoongSuite ENTRY, STEP, MEMORY, skill, baggage, multimodal, and compatibility extensions;
• keeps the existing invocation lifecycle and metrics implementation; the upstream 1.0 self-managed invocation lifecycle and new metrics are not included;
• leaves inherited instrumentation-genai provider packages unchanged.

The upstream opentelemetry-python-genai 1.0b0/current implementation was used as an implementation reference. The current standalone semantic-conventions-genai registry is treated as the authority for newer attribute names and types.

Fixes # (N/A)

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

• PRE_COMMIT_HOME="$(mktemp -d)" tox -e precommit
• tox -e generate-workflows
• tox -c tox-loongsuite.ini -e py311-test-detect-loongsuite-changes
• tox run-parallel -c tox-loongsuite.ini -p 3 -e py313-test-util-genai,py313-test-loongsuite-instrumentation-langchain-latest,py313-test-loongsuite-instrumentation-mem0-latest -- -q
• Representative oldest/latest LoongSuite adapter matrix covering AgentScope, Agno, AutoGen, DashScope, Google ADK, Hermes Agent, LangChain, LiteLLM, Mem0, and Qwen-Agent
• python -m build util/opentelemetry-util-genai

Does This PR Require a Core Repo Change?

• Yes. - Link to PR:
• No.

Checklist:

• Followed the style guidelines of this project
• Changelogs have been updated
• Unit tests have been added
• Documentation has been updated

Validation Evidence

Spec and Scope

• Linked spec: current official open-telemetry/semantic-conventions-genai
• Approved spec/comment: N/A
• Changed surface: opentelemetry-util-genai and existing instrumentation-loongsuite adapters only; no inherited instrumentation-genai provider changes

Local Checks

Check	Result	Notes
Static readiness	pass	No errors or warnings
Fresh-cache precommit	pass	No files modified
Focused tests	pass	util, LangChain latest, and Mem0 latest
Compatibility matrix	pass	11 representative oldest/latest environments
Independent review	pass	Full review and focused follow-up found no blocking issues
Privacy scan	N/A	No cassette or fixture changes

Real E2E Matrix

Scenario	Status	Evidence
non-streaming	pass	DashScope and Mem0 VCR-backed paths
streaming	pass	DashScope and Hermes Agent streaming paths
concurrency	pass	AgentScope, Agno, Google ADK, Hermes Agent, and LiteLLM isolation tests
agent/tool/ReAct	pass	AgentScope and Hermes Agent suites
tool-heavy	pass	DashScope and Hermes Agent tool-call cases
error path	pass	Mem0 error cassettes and adapter error-path tests

Telemetry and Weaver

Check	Status	Notes
Span tree / span kinds	pass	LLM, Retrieval, Agent, and Memory samples validated
Content capture modes	pass	NO_CONTENT omission and enabled capture verified
Concurrency isolation	pass	No cross-invocation state or span leakage
Weaver live-check	pass	Weaver 0.24.2 checked 5 spans and 34 attributes with 0 violations

The Weaver replay used the current official GenAI attribute definitions together with the official error.type definition. The full standalone registry currently has unrelated unresolved OpenAI and Bedrock references, so the attribute layer was checked independently.

CI

• GitHub checks: pending PR creation
• Known unrelated failures: none
• Follow-up needed: run the complete GitHub Actions matrix after publication

[Copilot]

Pull request overview

This PR updates LoongSuite’s GenAI utility telemetry and selected LoongSuite adapters to match the latest upstream OpenTelemetry GenAI semantic conventions, including updated attribute names/types, span kinds, and opt-in content capture behavior.

Changes:

• Align LLM attributes (stream flag, prompt fields/variables, reasoning level/usage) and add gen_ai.response.time_to_first_chunk (seconds) alongside existing time_to_first_token (ns).
• Update Retrieval and Memory telemetry to upstream conventions (CLIENT retrieval spans, integer top_k, upstream Memory operation names/fields, and stricter opt-in content attributes).
• Propagate these convention changes through multiple LoongSuite instrumentations and refresh related tests/changelogs.

Reviewed changes

Copilot reviewed 52 out of 52 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
util/opentelemetry-util-genai/tests/test_utils.py	Adds coverage for new LLM semconv fields and separate first-chunk/first-token times.
util/opentelemetry-util-genai/tests/test_extended_memory.py	Rewrites Memory tests to validate upstream operation names/attributes and opt-in content capture.
util/opentelemetry-util-genai/tests/test_extended_handler.py	Updates agent/retrieval tests for new attributes (agent version, CLIENT retrieval spans, top_k, opt-in docs).
util/opentelemetry-util-genai/src/opentelemetry/util/genai/types.py	Extends LLMInvocation with prompt/stream/reasoning fields and first-chunk timestamp; makes top_k integer.
util/opentelemetry-util-genai/src/opentelemetry/util/genai/span_utils.py	Emits new LLM attributes (prompt vars, stream, reasoning, first-chunk time) and reasoning usage tokens.
util/opentelemetry-util-genai/src/opentelemetry/util/genai/extended_types.py	Adds agent version and changes retrieval top_k type to int.
util/opentelemetry-util-genai/src/opentelemetry/util/genai/extended_span_utils.py	Aligns agent version emission; makes retrieval docs opt-in; switches retrieval top_k attribute to upstream key.
util/opentelemetry-util-genai/src/opentelemetry/util/genai/extended_semconv/gen_ai_memory_attributes.py	Adds upstream Memory attribute constants and upstream operation-name enum; retains legacy constants for transition.
util/opentelemetry-util-genai/src/opentelemetry/util/genai/extended_semconv/gen_ai_extended_attributes.py	Adds missing upstream attributes (prompt, stream, first-chunk, reasoning usage, compacted, agent version, retrieval top_k) and updates Moonshot provider value.
util/opentelemetry-util-genai/src/opentelemetry/util/genai/extended_memory/memory_utils.py	Reworks Memory span attributes to upstream convention and removes legacy event emission.
util/opentelemetry-util-genai/src/opentelemetry/util/genai/extended_memory/memory_types.py	Updates MemoryInvocation shape to upstream identifiers/content fields and adds configurable span_kind.
util/opentelemetry-util-genai/src/opentelemetry/util/genai/extended_memory/init.py	Stops exporting the removed legacy memory event helper.
util/opentelemetry-util-genai/src/opentelemetry/util/genai/extended_handler.py	Switches retrieval spans to CLIENT; makes Memory span name/kind driven by upstream operation and invocation kind; removes legacy event emission.
util/opentelemetry-util-genai/CHANGELOG.md	Documents upstream-alignment behavior changes (LLM, Retrieval, Memory) and first-chunk timing.
tox-loongsuite.ini	Installs util-genai as editable in tox envs.
instrumentation-loongsuite/loongsuite-instrumentation-qwen-agent/src/opentelemetry/instrumentation/qwen_agent/utils.py	Marks streaming requests in LLM invocation creation.
instrumentation-loongsuite/loongsuite-instrumentation-qwen-agent/src/opentelemetry/instrumentation/qwen_agent/patch.py	Tracks first-chunk vs first-token timing and detects “token-bearing” chunks.
instrumentation-loongsuite/loongsuite-instrumentation-qwen-agent/CHANGELOG.md	Notes new stream/response-id/first-chunk timing capture.
instrumentation-loongsuite/loongsuite-instrumentation-mem0/tests/test_memory_operations_vcr.py	Updates VCR tests to assert upstream Memory operation naming/attributes.
instrumentation-loongsuite/loongsuite-instrumentation-mem0/tests/test_memory_operations_record_vcr.py	Refactors record-flow assertions for upstream Memory spans and attributes.
instrumentation-loongsuite/loongsuite-instrumentation-mem0/tests/test_memory_client_vcr.py	Updates client-flow assertions for upstream Memory spans and opt-in record payloads.
instrumentation-loongsuite/loongsuite-instrumentation-mem0/tests/test_hooks.py	Updates wrapper/hook tests for new operation mapping and span kind behavior.
instrumentation-loongsuite/loongsuite-instrumentation-mem0/src/opentelemetry/instrumentation/mem0/internal/_wrapper.py	Maps Mem0 operations to upstream Memory operations and normalizes records/query/count for new util Memory telemetry.
instrumentation-loongsuite/loongsuite-instrumentation-mem0/pyproject.toml	Removes util-genai version pin to allow alignment with current util package.
instrumentation-loongsuite/loongsuite-instrumentation-mem0/CHANGELOG.md	Documents Mem0 → upstream Memory operation/attribute mapping and legacy-field/event removal.
instrumentation-loongsuite/loongsuite-instrumentation-litellm/tests/test_genai_util_wrapper.py	Adds assertions for streaming flag and first-chunk timing attribute types.
instrumentation-loongsuite/loongsuite-instrumentation-litellm/src/opentelemetry/instrumentation/litellm/_utils.py	Captures stream flag, reasoning level, reasoning token usage, and fixes provider parsing edge case.
instrumentation-loongsuite/loongsuite-instrumentation-litellm/src/opentelemetry/instrumentation/litellm/_stream_wrapper.py	Records stream state and first-chunk time on incoming chunks.
instrumentation-loongsuite/loongsuite-instrumentation-litellm/CHANGELOG.md	Notes new stream/reasoning/first-chunk timing capture.
instrumentation-loongsuite/loongsuite-instrumentation-langchain/tests/test_retriever_spans.py	Updates NO_CONTENT expectations to omit opt-in retrieval query/documents.
instrumentation-loongsuite/loongsuite-instrumentation-langchain/src/opentelemetry/instrumentation/langchain/internal/_tracer.py	Marks streaming state on token callbacks.
instrumentation-loongsuite/loongsuite-instrumentation-langchain/CHANGELOG.md	Notes streaming flag and retrieval opt-in omission behavior.
instrumentation-loongsuite/loongsuite-instrumentation-hermes-agent/src/opentelemetry/instrumentation/hermes_agent/wrappers.py	Marks streaming and records first-chunk timing on stream deltas.
instrumentation-loongsuite/loongsuite-instrumentation-hermes-agent/CHANGELOG.md	Notes upstream stream/first-chunk fields.
instrumentation-loongsuite/loongsuite-instrumentation-google-adk/src/opentelemetry/instrumentation/google_adk/internal/_plugin.py	Captures stream flag, first-chunk/first-token timing, response IDs, and reasoning token usage; enforces int top_k.
instrumentation-loongsuite/loongsuite-instrumentation-google-adk/CHANGELOG.md	Notes new upstream-aligned telemetry capture.
instrumentation-loongsuite/loongsuite-instrumentation-dashscope/src/opentelemetry/instrumentation/dashscope/patch/multimodal_conversation.py	Adds token-bearing chunk detection and records stream + first-chunk/first-token times.
instrumentation-loongsuite/loongsuite-instrumentation-dashscope/src/opentelemetry/instrumentation/dashscope/patch/generation.py	Adds token-bearing chunk detection and records stream + first-chunk/first-token times (sync/async).
instrumentation-loongsuite/loongsuite-instrumentation-dashscope/CHANGELOG.md	Notes new upstream-aligned telemetry capture.
instrumentation-loongsuite/loongsuite-instrumentation-claude-agent-sdk/tests/test_unit.py	Updates Moonshot provider expectation to upstream value.
instrumentation-loongsuite/loongsuite-instrumentation-claude-agent-sdk/CHANGELOG.md	Notes upstream Moonshot provider name fix.
instrumentation-loongsuite/loongsuite-instrumentation-autogen/tests/test_patch.py	Adds test coverage for marking streaming calls.
instrumentation-loongsuite/loongsuite-instrumentation-autogen/src/opentelemetry/instrumentation/autogen/patch.py	Records stream state and first-chunk time; tightens first-token detection.
instrumentation-loongsuite/loongsuite-instrumentation-autogen/CHANGELOG.md	Notes first-chunk vs first-token and conditional stream indicator behavior.
instrumentation-loongsuite/loongsuite-instrumentation-agno/src/opentelemetry/instrumentation/agno/_wrapper.py	Centralizes streaming chunk marking and improves token-bearing detection.
instrumentation-loongsuite/loongsuite-instrumentation-agno/CHANGELOG.md	Notes upstream-aligned streaming timing changes.
instrumentation-loongsuite/loongsuite-instrumentation-agentscope/tests/test_utils.py	Adds tests for upstream Moonshot provider naming.
instrumentation-loongsuite/loongsuite-instrumentation-agentscope/src/opentelemetry/instrumentation/agentscope/utils.py	Updates provider mapping and adds Moonshot base-url mapping.
instrumentation-loongsuite/loongsuite-instrumentation-agentscope/src/opentelemetry/instrumentation/agentscope/message_converter.py	Updates Moonshot provider branch to upstream value.
instrumentation-loongsuite/loongsuite-instrumentation-agentscope/src/opentelemetry/instrumentation/agentscope/_wrapper.py	Improves streaming chunk detection and records first-chunk/first-token times.
instrumentation-loongsuite/loongsuite-instrumentation-agentscope/src/opentelemetry/instrumentation/agentscope/_v2_middleware.py	Marks streaming calls and refines first-chunk/first-token detection for v2 middleware.
instrumentation-loongsuite/loongsuite-instrumentation-agentscope/CHANGELOG.md	Notes upstream-aligned provider/streaming/agent-version behavior changes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ralf0131]

@/tmp/pr246_review_body.md