feat(mcp_readability): group style findings per tool - #592
Draft
akangsha7 wants to merge 3 commits into
Draft
Conversation
added 3 commits
September 4, 2026 12:02
The style judge reported a rule once with a comma-joined tool list, so reviewers had to read a severity-ordered list and work out which tool each finding applied to. Ask the judge for one finding per affected tool with tool-specific wording, render the feedback as one list per tool, and add a findings_by_tool view to the persisted JSON. P0/P1/P2 stay comparable across implementations: they now count distinct rules violated rather than finding occurrences, so a large tool surface is not penalized for repeating the same rule.
…neral issues "All tools" claimed a finding applied to every tool. The bucket actually holds issues that belong to no individual tool -- a server over the 40-tool limit, a missing LRO polling or discovery tool, a parameter named inconsistently across tools.
Ask for findings already nested under the tool they affect, with a "general" entry for issues tied to no single tool, so the scorer parses the grouping instead of reconstructing it. Drops findings_by_tool() and the sentinel bucket it needed; only shape validation and the distinct-rule counts remain on our side.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Style-judge feedback was organized by severity, with a single finding carrying a comma-joined tool list (
"tool": "get_cluster, list_clusters"). A reviewer fixing one tool had to scan every severity bucket and work out which findings applied to it.The judge now returns its findings already grouped by tool, so the scorer parses that grouping rather than reconstructing it:
generalentry covers issues tied to no single tool — the server exceeding the 40-tool limit, a missing LRO polling tool, a parameter named inconsistently across tools._clean_findings_by_toolonly drops entries that can't be rendered (no tool name, findings not a list). Entry and finding order are the judge's.get_cluster — 1 P0, 1 P2).p0/p1/p2_issuescount distinct rules violated rather than finding occurrences. Without this, one finding per tool would score a 40-tool surface far worse than a 4-tool one for the same design mistake, and would swing with how many tools the judge attributed a rule to.Test plan
cd evalbench && uv run pytest test -q -k mcp— 109 passed