Proposal: GenAI semantic event for pre-execution judgment and negative proof

[slovingu8688-beep]

Area(s)

area:gen-ai

Propose new conventions

• Current GenAI semantic conventions cover execution well (invocations, tool calls, responses).
• In post-incident analysis and regulatory audits, teams often need to prove that a decision was evaluated but intentionally not executed.
• There is currently no standard event to represent this “judgment boundary”.
• This leads to audit blind spots (e.g., EU AI Act Article 12 traceability requirements).
• AJT is a minimal, zero-impact event schema designed to fill exactly this gap.
• This proposal does not introduce enforcement, control flow, or new trace types — only a lightweight semantic event.

• Reference implementation: https://github.com/Nick-heo-eg/spec

This is intended to be modeled as an event attached to existing GenAI spans, not as a new span type or signal.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[Nick-heo-eg]

One clarification from the proposal author, to make the scope more precise.

In AJT, a “judgment” is not defined by the outcome itself (allow / block / escalate),
but by whether alternative outcomes were explicitly evaluated.

We use the following minimal definition:

An event is considered a judgment if and only if:

1. Multiple outcome paths existed (≥2),
2. At least one non-selected path was explicitly evaluated,
3. The selected outcome is recorded as a choice among those evaluated paths.

Under this definition:

• An automatic pass-through with no evaluated alternatives is NOT a judgment.
• A deterministic rule application with no override or escalation path evaluated is NOT a judgment.
• “allow” becomes a judgment only when deny/escalate paths were evaluated and not selected.

The intent is not to explain why a decision was correct,
but to make it unambiguous that a decision boundary existed and was evaluated.

In short: AJT records not outcomes, but the existence of evaluated alternatives and a selected path.

[lmolkova]

/cc @singankit in case he has thoughts.

I believe this should be considered in the context of evaluation result event we defined https://github.com/open-telemetry/semantic-conventions/blob/main/docs/gen-ai/gen-ai-events.md#event-gen_aievaluationresult and can probably be expressed as a set of attributes.

[singankit]

/cc @singankit in case he has thoughts.

I believe this should be considered in the context of evaluation result event we defined https://github.com/open-telemetry/semantic-conventions/blob/main/docs/gen-ai/gen-ai-events.md#event-gen_aievaluationresult and can probably be expressed as a set of attributes.

Agree getting to an outcome based on evaluation result is a good approach.

Is the idea here to capture the entire judgement process or selected path + alternatives that were evaluated ?

[Nick-heo-eg]

Thanks for good question

The intent is explicitly "not" to capture the entire judgment process.

AJT does not record internal reasoning, full decision trees, or step-by-step evaluation.
Instead, it records decision "evidence" at the boundary:

• that multiple alternatives were evaluated,
• that a specific path was selected (or execution was intentionally stopped),
• and that this happened at a defined decision phase (e.g. pre-execution).

So the signal is closer to:
“an evaluation occurred and led to this decision outcome” rather than “here is how the system reasoned internally.”

This is why we see it as compatible with gen_ai.evaluation.result,
but with attributes that express decision phase, selected path,
and the existence (not content) of evaluated alternatives.

[Nick-heo-eg]

@lmolkova sharing a concrete attribute sketch here, based on the direction you mentioned (expressed as attributes
within gen_ai.evaluation.result). To make this more concrete, here is a minimal attribute set that could express AJT
within the existing gen_ai.evaluation.result event. The intent is to capture the existence of a decision boundary,
not internal reasoning or full decision trees.

Proposed attributes:

• gen_ai.evaluation.judgment.phase (string)
  e.g. pre_execution, post_execution, runtime

• gen_ai.evaluation.judgment.selected_path (string)
  e.g. allow, block, escalate, stop

• gen_ai.evaluation.judgment.alternatives_evaluated (boolean)
  true if ≥2 possible outcome paths were explicitly evaluated

• gen_ai.evaluation.judgment.alternatives_count (int, optional)
  number of evaluated paths (content intentionally not recorded)

• gen_ai.evaluation.judgment.human_in_loop (boolean)
  whether a human review participated in the evaluation

This keeps the signal lightweight and aligned with existing evaluation.result semantics,
while making it unambiguous that an evaluation led to a decision boundary.