Skip to content

feat(llm-obs): add agent-insights and model-pricing commands - #687

Merged
platinummonkey merged 1 commit into
DataDog:mainfrom
gsvigruha:gergely.svigruha/agent-obs-gaps-3
Jul 28, 2026
Merged

feat(llm-obs): add agent-insights and model-pricing commands#687
platinummonkey merged 1 commit into
DataDog:mainfrom
gsvigruha:gergely.svigruha/agent-obs-gaps-3

Conversation

@gsvigruha

Copy link
Copy Markdown
Contributor

Summary

Closes the Agent Insights and model-pricing gaps between the dd-source LLM Obs MCP server (domains/ml-observability/shared/libs/mcp/tools/) and pup. Of that server's 41 LLM Obs tools, 34 already had pup equivalents at full parameter parity; this PR adds 5 of the remaining 7.

Still unimplemented after this PR: launch_llmobs_experiment and get_llmobs_bits_session.

Changes

  • pup llm-obs agent-insights list|get|update-status|submit-feedback — proxies the four /agent-insights/* endpoints (src/commands/llm_obs.rs:1097)
  • pup llm-obs model-pricing — proxies /pricing/model for deterministic cost projection (src/commands/llm_obs.rs:1247)
  • Lifecycle statuses and usefulness verdicts are validated client-side against the server's declared enums, so a typo reports the valid values rather than a 400 (src/commands/llm_obs.rs:1123)
  • --feedback takes target_key=usefulness[=reasoning]. Real target keys embed colons — suggested_evaluator:<eval_name> — so = is the only separator that round-trips them; a colon-separated entry is rejected rather than silently mis-parsed (src/commands/llm_obs.rs:1190)
  • is_write_command_name now treats submit-* as a write (src/main.rs:10945), so submit-feedback is blocked under --read-only and reports read_only: false in the agent command catalog
  • Optional filters are omitted from request bodies rather than sent as defaults, so the server's own defaults (status=for_review, limit=25) still apply
  • docs/COMMANDS.md updated

Testing

18 new tests, all mock-based via the existing test_support helpers.

Positive: every filter populated and asserted body-for-body against the MCP tool definitions; omitted-argument bodies asserted to be empty; feedback parsing with and without reasoning; colon-bearing target keys and =-bearing reasoning preserved; exactly 25 feedback items accepted.

Negative: unknown status (both list and update-status), unknown usefulness, missing usefulness, empty target key, colon instead of =, 26 feedback items, model-pricing with neither --provider nor --model, plus 404/403/400/500 and no-auth paths.

Read paths were also verified against the live API: list with every filter, get (including the 404 path), model-pricing by exact model and by partial query across all providers, and cursor pagination returning a distinct second page. --read-only blocking both writes was verified end-to-end. The two write endpoints themselves were exercised against mocks only, to avoid mutating real insight state.

cargo fmt --check, cargo clippy --all-targets -- -D warnings, and cargo test are clean for this change. Five unrelated tests (cases::test_cases_timeline, dbm, monitors::test_monitors_diff_detects_changes, security::test_security_iocs_get, traces::test_spans_metrics_list) fail intermittently on main too — they race on the shared PUP_MOCK_SERVER env var and fall through to a DNS lookup of unused.local; each passes when run alone.


🤖 Generated with Claude Code

Closes the Agent Insights and model-pricing gaps between the dd-source LLM Obs
MCP server and pup. These were 5 of the 7 MCP tools with no pup equivalent; the
rest of the 41-tool surface was already at parity.

- `pup llm-obs agent-insights list|get|update-status|submit-feedback` — proxies
  the four `/agent-insights/*` endpoints (src/commands/llm_obs.rs)
- `pup llm-obs model-pricing` — proxies `/pricing/model` for cost math
- Lifecycle statuses and usefulness verdicts are validated client-side against
  the server's enums, so a typo fails with the valid values instead of a 400
- `--feedback` uses `target_key=usefulness[=reasoning]`: real target keys embed
  colons (`suggested_evaluator:<eval_name>`), so `=` is the only separator that
  round-trips them
- `is_write_command_name` now treats `submit-*` as a write, so `submit-feedback`
  is blocked under `--read-only` (verified end-to-end) and reports
  `read_only: false` in the agent command catalog (src/main.rs)

Read paths verified against the live API: list (with every filter), get,
model-pricing by exact model, by partial query across providers, and cursor
pagination. Write paths are asserted body-for-body against mocks.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gsvigruha
gsvigruha requested a review from a team as a code owner July 28, 2026 01:08
@gsvigruha
gsvigruha marked this pull request as draft July 28, 2026 01:29
@gsvigruha
gsvigruha marked this pull request as ready for review July 28, 2026 14:27
@platinummonkey
platinummonkey merged commit 949b269 into DataDog:main Jul 28, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants