feat(llm-obs): add dataset-records reads + experiment eval-metric submit - #673
Closed
tillwf wants to merge 1 commit into
Closed
feat(llm-obs): add dataset-records reads + experiment eval-metric submit#673tillwf wants to merge 1 commit into
tillwf wants to merge 1 commit into
Conversation
Add three llm-obs subcommands backed by the unstable llm-obs-mcp v1
endpoints, closing the gap vs the LLM Obs MCP toolset:
- experiments events submit -> POST /experiment/ingest-events
(submit_llmobs_experiment_events; metrics/tags via --file JSON,
experiment_id from the positional arg)
- datasets records -> POST /dataset/records
(get_llmobs_dataset_records; record-ids/tags/canonical-id/
dataset-version/limit/cursor/compute-schema flags)
- datasets records-full -> POST /dataset/records-full
(get_llmobs_full_dataset_records; 1-3 record ids)
These are the tools the agent-observability auto-experiment skill needs
that pup previously lacked. Docs (README coverage table, COMMANDS.md)
updated; unit tests added for success + error paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds three
llm-obssubcommands backed by the unstablellm-obs-mcpv1 endpoints, closing the gap between pup and the LLM Obs MCP toolset:llm-obs experiments events submit <exp-id> --filesubmit_llmobs_experiment_eventsPOST /api/unstable/llm-obs-mcp/v1/experiment/ingest-eventsllm-obs datasets recordsget_llmobs_dataset_recordsPOST /api/unstable/llm-obs-mcp/v1/dataset/recordsllm-obs datasets records-fullget_llmobs_full_dataset_recordsPOST /api/unstable/llm-obs-mcp/v1/dataset/records-fullWhy
These are the three LLM Obs MCP tools the
agent-observability-auto-experimentskill relies on that pup previously lacked (experiment eval-metric submission + both dataset-record reads). Adding them lets that workflow run through pup.Design
events submittakes the events body (metrics, optionaltags) via--fileJSON — matching the existing pup convention for writes (create/update/batch-update).experiment_idis the positional arg and overrides any value in the file.datasets recordsexposes the read filters as flags:--record-ids,--tags,--canonical-id,--dataset-version,--limit,--cursor,--compute-schema.datasets records-fulltakes--project-id,--dataset-id,--record-ids(1–3).experiments events/datasetsgroups.Tests / checks
cargo build✅cargo test llm_obs✅ — 6 new tests (success + error paths for each command)cargo clippy✅ no warningscargo fmtapplieddocs/COMMANDS.md🤖 Generated with Claude Code