feat(llm-obs): add datasets records-all to page past the preview cap

[tillwf]

Problem

pup llm-obs datasets records posts to /api/unstable/llm-obs-mcp/v1/dataset/records, which trims its response to a size budget and returns no cursor. On a dataset whose records carry sizeable inputs that lands at 19 records with truncated: true — and since the endpoint emits no cursor, there's no way to reach the rest. --cursor exists as a flag but nothing ever produces a value for it.

Confirmed against the API directly, not inferred from the CLI:

POST /api/unstable/llm-obs-mcp/v1/dataset/records   limit=50
  -> returned: 19, truncated: true, next_cursor: None

--limit 25 and --limit 50 both return 19; no cursor key appears in the response at any limit. records-full isn't a workaround either — it caps at 3 ids per call (record_ids has N entries; cap is 3), and you'd need the full id list to use it, which is exactly what you can't get.

So a dataset larger than ~19 records currently cannot be enumerated through pup.

Fix

The plain REST route has no such cap and pages properly:

GET /api/unstable/llm-obs/v1/datasets/{id}/records?page[limit]=N&page[cursor]=<meta.after>

Adds pup llm-obs datasets records-all --dataset-id D [--limit N], which pages that route via meta.after and emits the aggregate under the same records/returned keys, plus pages_fetched.

Verified against a real 50-record dataset:

command	result
datasets records --limit 50 (existing)	19 records, truncated: true
datasets records-all --limit 7 (new)	50 records, 8 pages, 50 unique ids, no placeholders
datasets records-all (default limit 100)	50 records, 1 page

Why a new subcommand rather than fixing records

The two routes return different shapes. records gives size-trimmed previews with __nested_object__ / __nested_array__ placeholders; the REST route gives full records. Switching routes under the existing flag would silently change its output shape and break anything parsing it. records stays the cheap browse path; this is the explicit "give me everything" path.

It takes only --dataset-id — the REST route doesn't require a project id.

Loop safety

Terminates on an empty cursor or a cursor identical to the previous one (a server returning a fixed cursor would otherwise spin forever), with a 200-page hard stop that warns before returning what it has.

Checks

• 3 tests: happy path (asserts both pages were requested, proving the cursor is followed), repeated-cursor guard, 500 response — all pass
• cargo fmt --check clean, cargo clippy --bin pup -- -D warnings clean
• No Cargo.toml change, so the dependency set is untouched and no new advisories are possible
• docs/COMMANDS.md command list updated

+174/−0 in src/, 2 lines in docs. Opened from a fork — I don't have write access to DataDog/pup.

🤖 Generated with Claude Code