Skip to content

feat(mcp_readability): group style findings per tool - #592

Draft
akangsha7 wants to merge 3 commits into
GoogleCloudPlatform:mainfrom
akangsha7:evalbench/mcp-readability-per-tool-findings
Draft

feat(mcp_readability): group style findings per tool#592
akangsha7 wants to merge 3 commits into
GoogleCloudPlatform:mainfrom
akangsha7:evalbench/mcp-readability-per-tool-findings

Conversation

@akangsha7

@akangsha7 akangsha7 commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Style-judge feedback was organized by severity, with a single finding carrying a comma-joined tool list ("tool": "get_cluster, list_clusters"). A reviewer fixing one tool had to scan every severity bucket and work out which findings applied to it.

The judge now returns its findings already grouped by tool, so the scorer parses that grouping rather than reconstructing it:

"findings_by_tool": [
  {"tool": "general", "findings": [{"severity": "P0", "rule_id": "Tool Count Limits", ...}]},
  {"tool": "get_cluster", "findings": [{"severity": "P0", ...}, {"severity": "P2", ...}]}
]
  • Prompt: one entry per tool, findings ordered P0 -> P1 -> P2 within an entry, and a violation affecting several tools reported under each with wording specific to that tool (its own parameters, description). A general entry covers issues tied to no single tool — the server exceeding the 40-tool limit, a missing LRO polling tool, a parameter named inconsistently across tools.
  • Parsing: _clean_findings_by_tool only drops entries that can't be rendered (no tool name, findings not a list). Entry and finding order are the judge's.
  • Feedback HTML: one findings list per tool, headed by that tool's severity tally (get_cluster — 1 P0, 1 P2).
  • Counts: p0/p1/p2_issues count distinct rules violated rather than finding occurrences. Without this, one finding per tool would score a 40-tool surface far worse than a 4-tool one for the same design mistake, and would swing with how many tools the judge attributed a rule to.

Test plan

  • cd evalbench && uv run pytest test -q -k mcp — 109 passed
  • One live judge run against a real endpoint to confirm the model returns the nested shape; unit tests cover parsing and rendering but not the model's behavior

Akangsha Goel added 3 commits September 4, 2026 12:02
The style judge reported a rule once with a comma-joined tool list, so
reviewers had to read a severity-ordered list and work out which tool each
finding applied to. Ask the judge for one finding per affected tool with
tool-specific wording, render the feedback as one list per tool, and add a
findings_by_tool view to the persisted JSON.

P0/P1/P2 stay comparable across implementations: they now count distinct
rules violated rather than finding occurrences, so a large tool surface is
not penalized for repeating the same rule.
…neral issues

"All tools" claimed a finding applied to every tool. The bucket actually
holds issues that belong to no individual tool -- a server over the 40-tool
limit, a missing LRO polling or discovery tool, a parameter named
inconsistently across tools.
Ask for findings already nested under the tool they affect, with a
"general" entry for issues tied to no single tool, so the scorer parses the
grouping instead of reconstructing it. Drops findings_by_tool() and the
sentinel bucket it needed; only shape validation and the distinct-rule
counts remain on our side.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant