feat(llm-obs): add agent-insights and model-pricing commands - #687
Merged
platinummonkey merged 1 commit intoJul 28, 2026
Merged
Conversation
Closes the Agent Insights and model-pricing gaps between the dd-source LLM Obs MCP server and pup. These were 5 of the 7 MCP tools with no pup equivalent; the rest of the 41-tool surface was already at parity. - `pup llm-obs agent-insights list|get|update-status|submit-feedback` — proxies the four `/agent-insights/*` endpoints (src/commands/llm_obs.rs) - `pup llm-obs model-pricing` — proxies `/pricing/model` for cost math - Lifecycle statuses and usefulness verdicts are validated client-side against the server's enums, so a typo fails with the valid values instead of a 400 - `--feedback` uses `target_key=usefulness[=reasoning]`: real target keys embed colons (`suggested_evaluator:<eval_name>`), so `=` is the only separator that round-trips them - `is_write_command_name` now treats `submit-*` as a write, so `submit-feedback` is blocked under `--read-only` (verified end-to-end) and reports `read_only: false` in the agent command catalog (src/main.rs) Read paths verified against the live API: list (with every filter), get, model-pricing by exact model, by partial query across providers, and cursor pagination. Write paths are asserted body-for-body against mocks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gsvigruha
marked this pull request as draft
July 28, 2026 01:29
gsvigruha
marked this pull request as ready for review
July 28, 2026 14:27
platinummonkey
approved these changes
Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the Agent Insights and model-pricing gaps between the dd-source LLM Obs MCP server (
domains/ml-observability/shared/libs/mcp/tools/) and pup. Of that server's 41 LLM Obs tools, 34 already had pup equivalents at full parameter parity; this PR adds 5 of the remaining 7.Still unimplemented after this PR:
launch_llmobs_experimentandget_llmobs_bits_session.Changes
pup llm-obs agent-insights list|get|update-status|submit-feedback— proxies the four/agent-insights/*endpoints (src/commands/llm_obs.rs:1097)pup llm-obs model-pricing— proxies/pricing/modelfor deterministic cost projection (src/commands/llm_obs.rs:1247)--feedbacktakestarget_key=usefulness[=reasoning]. Real target keys embed colons —suggested_evaluator:<eval_name>— so=is the only separator that round-trips them; a colon-separated entry is rejected rather than silently mis-parsed (src/commands/llm_obs.rs:1190)is_write_command_namenow treatssubmit-*as a write (src/main.rs:10945), sosubmit-feedbackis blocked under--read-onlyand reportsread_only: falsein the agent command catalogstatus=for_review,limit=25) still applyTesting
18 new tests, all mock-based via the existing
test_supporthelpers.Positive: every filter populated and asserted body-for-body against the MCP tool definitions; omitted-argument bodies asserted to be empty; feedback parsing with and without reasoning; colon-bearing target keys and
=-bearing reasoning preserved; exactly 25 feedback items accepted.Negative: unknown status (both list and update-status), unknown usefulness, missing usefulness, empty target key, colon instead of
=, 26 feedback items,model-pricingwith neither--providernor--model, plus 404/403/400/500 and no-auth paths.Read paths were also verified against the live API:
listwith every filter,get(including the 404 path),model-pricingby exact model and by partial query across all providers, and cursor pagination returning a distinct second page.--read-onlyblocking both writes was verified end-to-end. The two write endpoints themselves were exercised against mocks only, to avoid mutating real insight state.cargo fmt --check,cargo clippy --all-targets -- -D warnings, andcargo testare clean for this change. Five unrelated tests (cases::test_cases_timeline,dbm,monitors::test_monitors_diff_detects_changes,security::test_security_iocs_get,traces::test_spans_metrics_list) fail intermittently onmaintoo — they race on the sharedPUP_MOCK_SERVERenv var and fall through to a DNS lookup ofunused.local; each passes when run alone.🤖 Generated with Claude Code