feat(llm-obs): add datasets records-all to page past the preview cap - #678
Merged
platinummonkey merged 1 commit intoJul 27, 2026
Merged
Conversation
`pup llm-obs datasets records` posts to /api/unstable/llm-obs-mcp/v1/dataset/records, which trims
its response to a size budget and returns no cursor. On a dataset whose records carry sizeable
inputs that lands at 19 records with `truncated: true`, and because the endpoint emits no cursor
there is no way to reach the remainder — `--cursor` exists as a flag but nothing ever produces a
value for it. Confirmed against the API directly: --limit 25 and --limit 50 both return 19, and no
cursor key appears in the response at any limit. `records-full` is not a workaround either; it
caps at 3 ids per call ("record_ids has N entries; cap is 3") and you would still need the full id
list to use it.
The plain REST route has no such cap and pages properly:
GET /api/unstable/llm-obs/v1/datasets/{id}/records?page[limit]=N&page[cursor]=<meta.after>
This adds `datasets records-all`, which pages that route via meta.after and emits the aggregate
under the same records/returned keys plus pages_fetched. Verified against a 50-record dataset:
--limit 7 returns all 50 across 8 pages with 50 unique ids and no __nested_object__ placeholders,
where the existing command returns 19.
Added as a separate subcommand rather than changing `records` in place, because the two routes
return different shapes — `records` gives size-trimmed previews with __nested_object__ /
__nested_array__ placeholders, the REST route gives full records. Switching routes under the
existing flag would silently change its output shape and break callers that parse it. `records`
stays the cheap browse path; this is the explicit "give me everything" path. It takes only
--dataset-id, since the REST route does not require a project id.
Loop safety: terminates on an empty cursor or a cursor identical to the previous one (a server
returning a fixed cursor would otherwise spin), with a 200-page hard stop that warns on stdout
before returning what it has.
Tests cover the happy path (asserts BOTH pages were requested, proving the cursor is followed),
the repeated-cursor guard, and a 500 response.
platinummonkey
approved these changes
Jul 27, 2026
tillwf
added a commit
to tillwf/agent-skills
that referenced
this pull request
Jul 27, 2026
The pup backend was reading the eval corpus with `datasets records`, which posts to an MCP-token-budget endpoint that trims to a response-size budget (~19 records on a dataset with sizeable inputs), reports truncated: true, and returns no cursor. A run built that way silently measures a different corpus than an mcp run of the same dataset_id — different split, different class balance, nothing comparable. That is exactly what happened on a 50-record dataset: the pup run scored 19 records while two earlier mcp runs scored 50. DataDog/pup#678 (merged) adds `datasets records-all`, which pages the REST route internally and returns every record in one call. The table now makes that the corpus path, demotes `records` to browsing with its cap marked, and Step 1's dataset_id branch names the per-backend command explicitly: page next_cursor on mcp, records-all on pup. Two guards, both learned the hard way: * Absence of records-all is a STOP under datadog_backend: pup, like a missing binary — continuing on the capped path yields a corpus that is a truncation artifact. Detect it by running the subcommand and checking the exit code, NOT by --help, which exits 0 for unknown subcommands on some builds and reports a feature present when it is absent. * After loading, assert the materialized count equals the dataset's true size before splitting. That is the cheap check that would have caught the 19-of-50 truncation immediately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
pup llm-obs datasets recordsposts to/api/unstable/llm-obs-mcp/v1/dataset/records, which trims its response to a size budget and returns no cursor. On a dataset whose records carry sizeable inputs that lands at 19 records withtruncated: true— and since the endpoint emits no cursor, there's no way to reach the rest.--cursorexists as a flag but nothing ever produces a value for it.Confirmed against the API directly, not inferred from the CLI:
--limit 25and--limit 50both return 19; no cursor key appears in the response at any limit.records-fullisn't a workaround either — it caps at 3 ids per call (record_ids has N entries; cap is 3), and you'd need the full id list to use it, which is exactly what you can't get.So a dataset larger than ~19 records currently cannot be enumerated through pup.
Fix
The plain REST route has no such cap and pages properly:
Adds
pup llm-obs datasets records-all --dataset-id D [--limit N], which pages that route viameta.afterand emits the aggregate under the samerecords/returnedkeys, pluspages_fetched.Verified against a real 50-record dataset:
datasets records --limit 50(existing)truncated: truedatasets records-all --limit 7(new)datasets records-all(default limit 100)Why a new subcommand rather than fixing
recordsThe two routes return different shapes.
recordsgives size-trimmed previews with__nested_object__/__nested_array__placeholders; the REST route gives full records. Switching routes under the existing flag would silently change its output shape and break anything parsing it.recordsstays the cheap browse path; this is the explicit "give me everything" path.It takes only
--dataset-id— the REST route doesn't require a project id.Loop safety
Terminates on an empty cursor or a cursor identical to the previous one (a server returning a fixed cursor would otherwise spin forever), with a 200-page hard stop that warns before returning what it has.
Checks
cargo fmt --checkclean,cargo clippy --bin pup -- -D warningscleanCargo.tomlchange, so the dependency set is untouched and no new advisories are possibledocs/COMMANDS.mdcommand list updated+174/−0 in
src/, 2 lines in docs. Opened from a fork — I don't have write access to DataDog/pup.🤖 Generated with Claude Code