Skip to content

Commit 3fd3ad3

Browse files
spboyerCopilot
spboyer
and
Copilot
authored
feat: port Azure ML tool_call evaluation rubrics (microsoft#106) (microsoft#160)
Closes microsoft#106 Adapts Azure ML's tool_call evaluator rubrics (tool_call_accuracy, tool_selection, tool_input_accuracy, tool_output_utilization) as waza-compatible YAML configs for the prompt grader. ## What's included | File | Evaluates | Scale | |------|-----------|-------| | `tool_call_accuracy.yaml` | Overall tool call effectiveness | 1–5 ordinal → 0.0–1.0 | | `tool_selection.yaml` | Right tools chosen, none missed | Binary → 0.0/1.0 | | `tool_input_accuracy.yaml` | Parameter correctness | Binary → 0.0/1.0 | | `tool_output_utilization.yaml` | Correct use of tool results | Binary → 0.0/1.0 | | `README.md` | Usage guide and rubric structure docs | — | ## Dependencies These are config artifacts (YAML + docs). They become usable once the `prompt` grader (microsoft#104) merges. ## Source Adapted from [Azure ML built-in evaluators](https://github.com/Azure/azureml-assets/tree/main/assets/evaluators/builtin) (MIT License). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 6ec1c27 commit 3fd3ad3

5 files changed

Lines changed: 521 additions & 0 deletions

File tree

‎examples/rubrics/README.md‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
# Tool Call Evaluation Rubrics
2+
3+
Pre-built rubric configurations for evaluating how well an AI agent uses tools.
4+
These are adapted from [Azure ML's built-in evaluators](https://github.com/Azure/azureml-assets/tree/main/assets/evaluators/builtin)
5+
for use with waza's `prompt` grader.
6+
7+
> **Note:** These rubrics require the `prompt` grader ([#104](https://github.com/spboyer/waza/issues/104)),
8+
> which is not yet merged. The YAML files are ready to use once that grader lands.
9+
10+
## Rubrics
11+
12+
| File | What it evaluates | Scale |
13+
|------|-------------------|-------|
14+
| `tool_call_accuracy.yaml` | Overall tool call effectiveness (relevance, parameters, completeness, efficiency, execution) | 1–5 ordinal → 0.0–1.0 |
15+
| `tool_selection.yaml` | Whether the right tools were chosen and no essential tools were missed | Binary pass/fail → 0.0/1.0 |
16+
| `tool_input_accuracy.yaml` | Whether all parameters passed to tools are correct (grounded, typed, formatted) | Binary pass/fail → 0.0/1.0 |
17+
| `tool_output_utilization.yaml` | Whether the agent correctly uses data returned by tools in its response | Binary pass/fail → 0.0/1.0 |
18+
19+
## How they work together
20+
21+
These four rubrics decompose tool-use quality into orthogonal dimensions:
22+
23+
```
24+
tool_call_accuracy (umbrella)
25+
├── tool_selection — Did the agent pick the right tools?
26+
├── tool_input_accuracy — Did it pass the right parameters?
27+
└── tool_output_utilization — Did it use the results correctly?
28+
```
29+
30+
`tool_call_accuracy` is the composite evaluator — it covers all three concerns
31+
in a single 1–5 score. The other three are focused evaluators that give you
32+
granular pass/fail signals on each dimension.
33+
34+
## YAML structure
35+
36+
Each rubric file includes:
37+
38+
| Field | Purpose |
39+
|-------|---------|
40+
| `name` | Identifier for the rubric |
41+
| `description` | What it evaluates |
42+
| `version` | Semantic version of the rubric definition |
43+
| `evaluation_criteria` | Detailed instructions for the LLM judge |
44+
| `rating_levels` | Scoring scale with descriptions and examples |
45+
| `score_normalization` | How raw scores map to the 0.0–1.0 range waza expects |
46+
| `input_mapping` | How waza's `graders.Context` fields feed into the rubric |
47+
| `chain_of_thought` | Step-by-step reasoning instructions for the LLM judge |
48+
| `output_schema` | Expected JSON structure from the LLM judge |
49+
50+
## Usage with the prompt grader
51+
52+
Once the `prompt` grader is available, reference a rubric in your eval spec:
53+
54+
```yaml
55+
# eval.yaml
56+
tasks:
57+
- name: "tool-use-quality"
58+
prompt: "Book a flight from SEA to JFK for next Friday"
59+
graders:
60+
- type: prompt
61+
rubric: rubrics/tool_call_accuracy.yaml
62+
63+
# Or use the focused evaluators for granular scoring:
64+
- type: prompt
65+
rubric: rubrics/tool_selection.yaml
66+
- type: prompt
67+
rubric: rubrics/tool_input_accuracy.yaml
68+
- type: prompt
69+
rubric: rubrics/tool_output_utilization.yaml
70+
```
71+
72+
## Input mapping
73+
74+
The rubrics expect these fields from waza's grading context:
75+
76+
| Rubric field | Waza context field | Description |
77+
|-------------|-------------------|-------------|
78+
| `query` / `session_transcript` | `session_transcript` | The full conversation history |
79+
| `tool_calls` | `tool_calls` | Tool calls made by the agent |
80+
| `tool_definitions` | `tool_definitions` | Available tool schemas |
81+
| `response` / `agent_response` | `agent_response` | The agent's latest message (tool_output_utilization only) |
82+
83+
## Score normalization
84+
85+
All rubrics normalize to waza's standard 0.0–1.0 range:
86+
87+
- **Binary rubrics** (tool_selection, tool_input_accuracy, tool_output_utilization): `0 → 0.0`, `1 → 1.0`
88+
- **Ordinal rubric** (tool_call_accuracy): `(raw - 1) / 4` maps `1 → 0.0`, `5 → 1.0`
89+
90+
## Attribution
91+
92+
These rubrics are adapted from the Azure ML evaluator prompty templates, licensed
93+
under the MIT License. See the [source repository](https://github.com/Azure/azureml-assets)
94+
for the original implementations.

‎examples/rubrics/tool_call_accuracy.yaml‎

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
# Tool Call Accuracy Rubric
2+
# Adapted from Azure ML's tool_call_accuracy evaluator
3+
# Source: https://github.com/Azure/azureml-assets/tree/main/assets/evaluators/builtin/tool_call_accuracy
4+
# For use with waza's `prompt` grader (see #104)
5+
6+
name: tool_call_accuracy
7+
description: >
8+
Evaluates the overall effectiveness of ALL tool calls made by an agent in response
9+
to a user's query. Considers collective relevance, parameter correctness, completeness,
10+
efficiency, and execution success.
11+
version: "1.0.0"
12+
13+
# Ordinal scale: 1–5 mapped to 0.0–1.0
14+
score_normalization:
15+
type: ordinal
16+
raw_min: 1
17+
raw_max: 5
18+
normalized_min: 0.0
19+
normalized_max: 1.0
20+
formula: "(raw - 1) / 4"
21+
22+
input_mapping:
23+
query: "session_transcript"
24+
tool_calls: "tool_calls"
25+
tool_definitions: "tool_definitions"
26+
27+
evaluation_criteria: |
28+
Evaluate based on these factors:
29+
30+
1. **Collective Relevance**: Do the tool calls, taken together, appropriately
31+
address the user's query?
32+
33+
2. **Parameter Correctness**: Are all parameter values extracted from or reasonably
34+
inferred from the conversation?
35+
- Fabricated parameters automatically result in Level 2.
36+
37+
3. **Completeness**: Did the agent make all necessary tool calls available in the
38+
tool definitions?
39+
- Failed calls don't count as missing.
40+
41+
4. **Efficiency**: Did the agent avoid unnecessary duplicate tool calls with
42+
identical parameters?
43+
- Don't penalize single tools returning multiple results (like file_search).
44+
45+
5. **Execution Success**: Were tool calls executed successfully or recovered from
46+
errors appropriately?
47+
48+
6. **Scope Limitation**: ONLY evaluate tool calls in the agent's response to the
49+
user's LAST query. Prior tool calls are context only.
50+
51+
**Success Criteria**: Tools should retrieve relevant data to help answer the query.
52+
Complete final answers are not required from individual tools.
53+
54+
**Tool Assessment**: Focus solely on appropriate use of available tools, not on
55+
capabilities beyond what tools can provide.
56+
57+
rating_levels:
58+
1:
59+
label: "Irrelevant"
60+
description: >
61+
Tool calls were not relevant to the user's query, resulting in irrelevant
62+
or unhelpful output.
63+
example: >
64+
User asks for distance between two cities → Agent calls a weather function
65+
to get the weather in the two cities.
66+
67+
2:
68+
label: "Partially Relevant — Wrong Execution"
69+
description: >
70+
Tool calls were somewhat related to the user's query, but the agent could not
71+
reach information that helps address the query due to one or more of:
72+
incorrect parameters, missing tool calls (available in definitions), or
73+
tool errors with no successful retrial.
74+
examples:
75+
- >
76+
User asks for coordinates of Chicago. Agent calls the correct tool but
77+
passes 'New York' instead of 'Chicago' as parameter.
78+
- >
79+
User asks a question needing 3 tool calls. Agent calls only 1 of the 3.
80+
81+
3:
82+
label: "Relevant but Inefficient"
83+
description: >
84+
Tool calls were relevant with correct and grounded parameters leading to
85+
correct output. However, multiple excessive, unnecessary tool calls were made.
86+
Do NOT penalize built-in tools like file_search that naturally return multiple
87+
results in a single call.
88+
example: >
89+
User asks for popular hotels in a place. Agent calls the same tool with the
90+
same parameters multiple times, even though a single call is sufficient.
91+
92+
4:
93+
label: "Correct with Retrials"
94+
description: >
95+
Tool calls were fully relevant and efficient. Correct tools were called with
96+
correct grounded parameters. A tool returned an error, but the agent retried
97+
and successfully got output.
98+
example: >
99+
User asks for weather forecast. Agent calls the correct tool, gets an error,
100+
retries, and gets the correct output.
101+
102+
5:
103+
label: "Optimal Solution"
104+
description: >
105+
Tool calls were fully relevant and efficient. Correct tools called with correct
106+
grounded parameters. No unnecessary or excessive calls. No errors occurred.
107+
The tool calls helped the agent address the user's query without issues.
108+
examples:
109+
- >
110+
User asks for distance between two places. Agent correctly calls coordinate
111+
retrieval tools then the distance calculator, passing correct arguments
112+
without excess calls.
113+
- >
114+
User asks to summarize a SharePoint file. Agent calls sharepoint_grounding
115+
to retrieve the file, enabling summarization.
116+
117+
chain_of_thought: |
118+
Structure your reasoning as follows:
119+
1. **Start with the user's last query**: Understand what the last user message asks.
120+
2. **Identify relevant available tools**: Analyze which tools in TOOL DEFINITIONS
121+
could help answer the query.
122+
3. **Analyze the actual tool calls made**: Compare what was done vs. what should
123+
have been done.
124+
4. **Check parameter grounding**: Ensure all parameters are grounded from the
125+
conversation and not hallucinated.
126+
5. **Determine the appropriate level**: Follow the level definitions precisely.
127+
128+
output_schema:
129+
chain_of_thought: "string — step-by-step reasoning"
130+
tool_calls_success_level: "integer 1–5"
131+
details:
132+
tool_calls_made_by_agent: "integer"
133+
correct_tool_calls_made_by_agent: "integer"
134+
per_tool_call_details: "list of {tool_name, total_calls_required, correct_calls_made_by_agent, correct_tool_percentage, tool_call_errors, tool_success_result}"
135+
excess_tool_calls: "{total, details: [{tool_name, excess_count}]}"
136+
missing_tool_calls: "{total, details: [{tool_name, missing_count}]}"

‎examples/rubrics/tool_input_accuracy.yaml‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# Tool Input Accuracy Rubric
2+
# Adapted from Azure ML's tool_input_accuracy evaluator
3+
# Source: https://github.com/Azure/azureml-assets/tree/main/assets/evaluators/builtin/tool_input_accuracy
4+
# For use with waza's `prompt` grader (see #104)
5+
6+
name: tool_input_accuracy
7+
description: >
8+
Performs a strict binary evaluation (PASS/FAIL) of whether ALL parameters passed
9+
to tool calls are correct. Checks groundedness, type compliance, format compliance,
10+
required parameters, unexpected parameters, and value appropriateness.
11+
version: "1.0.0"
12+
13+
# Binary pass/fail: 0 or 1 → 0.0 or 1.0
14+
score_normalization:
15+
type: binary
16+
values: [0, 1]
17+
normalized_map:
18+
0: 0.0
19+
1: 1.0
20+
21+
input_mapping:
22+
query: "session_transcript"
23+
tool_calls: "tool_calls"
24+
tool_definitions: "tool_definitions"
25+
26+
evaluation_criteria: |
27+
The evaluation must check ALL of the following criteria. If ANY criterion fails,
28+
the overall result is FAIL:
29+
30+
1. **Parameter Groundedness**: ALL parameters must be derived from or supported by
31+
information in the conversation history/query. NO fabricated or unsupported values.
32+
33+
2. **Type Compliance**: ALL parameters must match the exact type specified in the
34+
tool definitions (string, number, boolean, array, object, etc.).
35+
36+
3. **Format Compliance**: ALL parameters must follow the exact format, structure,
37+
and constraints specified in the tool definitions.
38+
39+
4. **Required Parameters**: ALL required parameters must be provided. Missing any
40+
required parameter results in FAIL.
41+
42+
5. **Unexpected Parameters**: NO parameters should be provided that are not defined
43+
in the tool definition. Any extra/unexpected parameters result in FAIL.
44+
45+
6. **Value Appropriateness**: ALL parameter values must be contextually appropriate
46+
and meaningful for the tool's purpose.
47+
48+
rating_levels:
49+
0:
50+
label: "Fail"
51+
description: >
52+
Any of the evaluation criteria fails, including: any parameter lacks grounding
53+
in conversation history, any parameter has wrong type, any parameter has wrong
54+
format/structure, any required parameter is missing, any unexpected parameter
55+
is present, or any parameter value is inappropriate for the context.
56+
57+
1:
58+
label: "Pass"
59+
description: >
60+
ALL criteria are satisfied perfectly. Every single parameter is properly
61+
grounded in conversation history/query, correct type per tool definition,
62+
proper format and structure, all required parameters present, no unexpected
63+
parameters, and contextually appropriate values.
64+
65+
chain_of_thought: |
66+
Structure your reasoning as follows:
67+
1. **Identify all tool calls**: List every tool call made by the agent.
68+
2. **For each tool call, check each parameter against all criteria**:
69+
- Is the parameter grounded in conversation history?
70+
- Does it match the expected type from the tool definition?
71+
- Does it follow the required format and constraints?
72+
- Are all required parameters present?
73+
- Are there any unexpected/undefined parameters?
74+
- Is the value contextually appropriate?
75+
3. **Aggregate results**: If ANY parameter fails ANY criterion, the overall
76+
result is FAIL. Only PASS if every parameter passes every criterion.
77+
78+
output_schema:
79+
chain_of_thought: "string — step-by-step analysis of all parameters"
80+
details:
81+
total_parameters_passed: "integer"
82+
correct_parameters_passed: "integer"
83+
incorrect_parameters: "list of strings describing incorrect parameters with reasons"
84+
result: "integer 0 or 1"

0 commit comments

Comments
 (0)