Skip to content

feat(agent-obs): add Topic Discovery (Patterns) commands - #683

Merged
platinummonkey merged 1 commit into
DataDog:mainfrom
gsvigruha:gergely.svigruha/agent-obs-pup-round-2
Jul 27, 2026
Merged

feat(agent-obs): add Topic Discovery (Patterns) commands#683
platinummonkey merged 1 commit into
DataDog:mainfrom
gsvigruha:gergely.svigruha/agent-obs-pup-round-2

Conversation

@gsvigruha

Copy link
Copy Markdown
Contributor

Summary

Adds pup llm-obs patterns, closing the last fully-missing LLM Observability MCP domain in pup. All 7 Topics Discovery tools now have a CLI equivalent.

Topic Discovery clusters LLM Obs spans into a topic hierarchy: a config defines what to cluster (via evp_query), a run is one clustering of those spans at a point in time, and a completed run yields topics, each backed by clustering points (spans).

Follow-up to #681, which closed the project/dataset/span-search gaps.

Changes

Seven new commands, one per MCP tool — all read-only (src/commands/llm_obs.rs:974):

Command MCP tool Endpoint
patterns configs list list_llmobs_pattern_configs topic-discovery/configs/list
patterns configs get get_llmobs_pattern_config topic-discovery/config/get
patterns runs list --config-id list_llmobs_pattern_runs topic-discovery/runs/list
patterns runs status --config-id get_llmobs_pattern_run_status topic-discovery/run-status
patterns topics --config-id [--run-id] get_llmobs_patterns topic-discovery/topics
patterns topics-with-points --config-id [--run-id] [--include-metrics] get_llmobs_patterns_with_points topic-discovery/topics-with-points
patterns points --topic-id [--page-size] [--page-token] get_llmobs_pattern_points topic-discovery/clustered-points

The subcommand layout follows the natural read flow: configs listruns statustopicspoints.

Two design notes:

  • Optional args are omitted from the request body rather than sent as explicit defaults, so the server's own defaults apply — matching MCP behavior for run_id (latest completed run), include_metrics, and the page_size/page_token cursor. Two tests pin this by asserting the key is absent rather than false/null.
  • A shared patterns_post helper keeps the seven wrappers to one line of transport each, rather than repeating the path-build/POST/map_err/output chain seven times.

Arg schemas were taken from the live MCP server, not the dd-source checkout, which was several days stale. They matched, including the non-obvious detail that configs get takes no arguments — it returns the most-recently-modified config for the org, so configs list is how you resolve a specific config_id.

Testing

14 new tests:

  • Strict Matcher::Json body assertions per endpoint, verifying config_id / run_id / topic_id and the paging args reach the server exactly as expected
  • Both branches of every optional arg — including two tests asserting omitted args produce an absent key rather than an explicit default
  • Error paths: 404 on configs get (an org with no config yet), 404 on unknown topic, 500 on run-status, and a no-auth case

cargo fmt --check, cargo clippy --all-targets -- -D warnings, and cargo test llm_obs (116 passing) are clean.

I also verified all 7 leaves classify as read_only: true in the agent schema. Worth checking because patterns runs status yields the leaf name status, which sits close to the write-verb list in is_write_command_name.

Smoke-tested end to end against the live API on a real config (Eval: product-analytics-vs-rum-tool-selection): listed configs, resolved the latest run as completed, read the topic hierarchy (Incorrect Datadog tool or action choice, 3 points), pulled a page of 2 points, then followed the returned next_page_token to page 2 and got a different set of spans — confirming the cursor actually advances.

Still open

From the same audit, for follow-up: Agent Insights (4 tools, gated behind the llm-obs-insights feature flag), get_llmobs_model_pricing and get_llmobs_bits_session (both org-2 allowlisted in the MCP), and the 9 LLM-Obs workflow skills that pup skills can't reach (it targets /api/v2/onboarding/skills, a different backend).


🤖 Generated with Claude Code

Closes the last fully-missing LLM Observability MCP domain in pup: all 7
Topics Discovery tools now have a CLI equivalent under `pup llm-obs patterns`.
Topic Discovery clusters LLM Obs spans into a topic hierarchy — a config
defines what to cluster, a run is one clustering, and a completed run yields
topics backed by clustering points.

New commands, one per MCP tool (all read-only):

- patterns configs list        -> list_llmobs_pattern_configs
- patterns configs get         -> get_llmobs_pattern_config
- patterns runs list           -> list_llmobs_pattern_runs
- patterns runs status         -> get_llmobs_pattern_run_status
- patterns topics              -> get_llmobs_patterns
- patterns topics-with-points  -> get_llmobs_patterns_with_points
- patterns points              -> get_llmobs_pattern_points

Optional args are omitted from the request body rather than sent as defaults,
so the server's own defaults apply — matching MCP behavior for run_id
(latest run), include_metrics, and the points page_size/page_token cursor.

Shared patterns_post helper keeps the seven thin wrappers to one line of
transport each (src/commands/llm_obs.rs:974).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gsvigruha
gsvigruha requested a review from a team as a code owner July 27, 2026 17:53
@gsvigruha gsvigruha changed the title feat(llm-obs): add Topic Discovery (Patterns) commands feat(agent-obs): add Topic Discovery (Patterns) commands Jul 27, 2026
@platinummonkey
platinummonkey merged commit 6e3be36 into DataDog:main Jul 27, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants