gen-ai: add judgment boundary attributes to evaluation result #3336
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds attributes to the gen_ai.evaluation.result event to support traceability of decision boundaries where multiple alternative outcomes were evaluated. Implements discussion from open-telemetry#3244
…t intent Adds two concrete JSON examples demonstrating judgment boundary attribute usage: - Content safety pre-execution check with automatic decision - Cost boundary evaluation with human escalation Clarifies that judgment boundary attributes are intended for event-level auditability and post-hoc inspection rather than high-cardinality metric aggregation.
Author
|
Note: This PR reintroduces the same changes as #3297, which was closed during repository cleanup. |
lmolkova
moved this from Untriaged
to Awaiting codeowners approval
in Semantic Conventions Triage
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a small set of attributes to the existing
gen_ai.evaluation.resultevent to improve traceability of decision boundaries where multiple alternative outcomes were evaluated.The design follows the direction discussed in #3244 and does not introduce new events.
Changes
Added four new attributes to
model/gen-ai/registry.yaml:gen_ai.evaluation.judgment.phasegen_ai.evaluation.judgment.selected_pathgen_ai.evaluation.judgment.alternatives_evaluatedgen_ai.evaluation.judgment.human_in_loopThese attributes are connected to the
gen_ai.evaluation.resultevent inmodel/gen-ai/events.yaml.Motivation
Current GenAI traces capture execution outcomes but do not provide an explicit signal that alternative paths (e.g. allow vs block) were evaluated.
These attributes allow systems to demonstrate that such evaluations occurred, without exposing internal reasoning or policy logic.
Design Rationale
gen_ai.evaluation.resultrather than creating a new eventNote
Documentation files under
docs/gen-ai/are autogenerated from the registry and event schemas and are not edited directly in this PR.Related Issue