Problem
An executor can report success even when the intended effect never becomes observable. Examples include a push to a branch that no longer affects the default branch, or a deploy that exits successfully while production still serves the previous revision.
VOLY should distinguish action completion from outcome effectiveness in its multi-agent traces and episodes.
Source discussion: microsoft/autogen#7265 (comment)
Proposed direction
- Add an explicit postcondition/effectiveness result to
AgentTrace / MultiAgentEpisode.
- Keep execution success and observed effect as separate semantics.
- Run the postcondition at the earliest point where the expected effect can be observed.
- Allow the verifier/observer to use a channel independent from the actor's execution channel.
- Persist verifier identity, observation timestamp, evidence/artifacts, verdict, and failure reason in the episode.
- Make the result available to Agentic Judge and role metrics.
Possible initial model:
PostconditionResult
- status: passed | failed | inconclusive | not_run
- observer_role
- observed_at
- assertion
- evidence_refs[]
- failure_reason
- channel
Acceptance criteria
Non-goals
- Generic workflow engine.
- Self-play or training from postcondition data in the first iteration.
- Claiming an effect when the verifier itself fails; that state must remain
inconclusive.
Problem
An executor can report
successeven when the intended effect never becomes observable. Examples include a push to a branch that no longer affects the default branch, or a deploy that exits successfully while production still serves the previous revision.VOLY should distinguish action completion from outcome effectiveness in its multi-agent traces and episodes.
Source discussion: microsoft/autogen#7265 (comment)
Proposed direction
AgentTrace/MultiAgentEpisode.Possible initial model:
Acceptance criteria
success + failed effect,success + passed effect, and verifier-channel failure.Non-goals
inconclusive.