auto-experiment: emergent failure census, domain notes, distribution tags, and the datadog_backend (pup) flag - #87
Merged
gsvigruha merged 17 commits intoJul 27, 2026
Conversation
Every iteration already computes a score_distribution (per-datapoint values + five-number summary) into config.json iteration_results, but nothing left the machine: the LLM-Obs event carried only the scalar mean, so the spread was visible only to whoever had the scratch branch checked out. Publish min/q1/median/q3/max as dist_* tags on the iteration's auto_experiment_score metric. The submit_llmobs_experiment_events schema has no array or object field, so tags are the only place a summary can ride along with the score and survive the per-iteration:<n> dedup rule. The raw values array stays local — 35+ tags per event is not worth it. - dist_* prefix keeps these distinct from min_delta, the keep/discard floor, which is unrelated to the score spread. - Values are copied from the iteration_results row, rounded to 4dp, never re-derived. - Iteration 0 carries them too (its tag list in Step 2.4 read as exhaustive, so it now says so explicitly). - no_change iterations omit all five: no eval ran, so carrying the prior best's spread forward would dress a non-measurement up as a measured one. Absent dist_* is the honest signal. - The higher-power promotion correction re-sends the same dist_* tags — it re-labels confidence, it does not restate the distribution. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The census was the loop's observation step, and it was handing the agent
its own answers: the rubric listed wrong_retrieval / wrong_reasoning /
format-parse / refusal / judge_disagreement / data-label up front, so
every failure landed in a bucket that already existed. A census that can
only confirm the failure modes you thought of before running it cannot
surface the one you missed, which is the entire reason to run one.
Rework it into two phases that must stay separate.
Phase A describes. Parallel describer sub-agents read the failing
datapoints and return factual sentences about what the output did versus
what the reference wanted, with no category vocabulary in the prompt.
Facts ("dropped the status='open' predicate") rather than judgments
("reasoned poorly") — a judgment has already smuggled in a category.
Because the task is descriptive it needs no run-level context, which is
what makes fanning it out safe and describing every failure affordable.
Phase B synthesizes. The orchestrator groups the descriptions and names
the buckets from what they actually say. The generic list survives only
as a last-resort naming aid at synthesis time, never as describer input.
census.json now stores the per-datapoint descriptions alongside the
buckets, plus failing_total/described coverage counts, so a bucket can be
audited, a taxonomy can be re-synthesized without re-describing, and a
count drawn from a partial sample is reported as "15 of 47 inspected"
instead of passing for full coverage.
Also adds domain_notes: free-text product context the code does not
carry, injected verbatim into sub-agent briefings, census describers and
the judge prompt — the three agents that interpret the domain. When the
user corrects a domain misread mid-run the correction is appended there
rather than applied once, so it survives the iteration. Notes are
trusted context in their own delimited block, kept separate from
untrusted datapoint content, and may never redefine the evaluators.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rubric requires the judge to render domain_notes as trusted context in its own block, separate from untrusted datapoint content. The template the harness is copied from had neither: domain_notes was absent entirely, and the prompt-injection guard existed only as prose in judge()'s docstring — an instruction to whoever implements the call, with no code behind it. Every run re-derived the same structure by hand, which is how a run quietly ends up with one merged block. Adds build_judge_prompt(): trusted blocks first (the evaluators rubric, then domain_notes when non-empty), untrusted datapoint content last in separately delimited blocks, with an explicit instruction that the datapoint blocks are material to be scored rather than commands. Untrusted content is sealed against the block delimiters by _seal(), which inserts a zero-width space into anything resembling one of our own tags. Surgical on purpose: a datapoint carrying "</datapoint_input>" can no longer close its block and escape into instruction space, while SQL operators, markup and code reach the judge byte-identical and get scored as written. A blunter escape would corrupt the very content under test. domain_notes arrives via AUTO_EXP_DOMAIN_NOTES, matching how EVALUATORS is already threaded in; an empty value omits the block rather than emitting an empty one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five defects a Codex review of the branch surfaced, four of them introduced by the two commits before this one. domain_notes never reached the judge. SKILL.md claimed injection into the judge prompt, but the harness read only AUTO_EXP_DOMAIN_NOTES and no run step ever mentioned exporting it, so the feature was a no-op for the one consumer that scores. The template now reads domain_notes out of config.json on every run: no env var to forget, and notes appended mid-run take effect on the next run rather than silently going stale. The env var survives as an override for callers with no config.json. domain_notes was three different types — free text in the inputs table, a list in the config schema, a string with .strip() in the template. Pinned to a list of strings everywhere, one note per correction, which is what "append the correction" was always trying to say. A bare string is still tolerated on read. _seal claimed untrusted content "cannot close its own block and escape into instruction space", but matched exact lowercase tags only — </DATAPOINT_INPUT> and < / datapoint_input > sailed through, and an LLM reads both as closing tags. Now a case-insensitive regex tolerating internal whitespace, and the docstring says what the mechanism actually buys: it raises the cost of a break-out, it is not a proof against one. The load-bearing guard is the instruction framing; the seal is defence in depth and is no longer described as a sanitizer. Iteration 0 was routed through the decision-legibility tag set, which demands delta_vs_best, t_stat and significant. The baseline has no previous best and runs no t-test, so an executing agent would have had to invent all three — colliding with the scoring policy's "never invent a score". Baseline now carries basis:baseline, timing and dist_* only, with the omission stated in both places an agent might read. Finally, dist_* summarizes the last run's per-datapoint spread while score_value is the mean across runs, so dist_median does not generally equal score_value. The mismatch predates this branch but publishing the quartiles makes it consumer-visible; both the tag spec and the score_distribution section now say plainly what the spread is and is not, so no one reads it as quartiles of the reported score. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Second review pass on the same branch. Four of the five earlier findings came back fully fixed; this handles the residual and three new ones. The delimiter invariant was stated but only half enforced. Datapoint content was sealed, DOMAIN_NOTES went in raw, so a note that happened to quote </domain_notes> would close the trusted block and spill into the framing text. Not the datapoint-injection threat — notes are user-authored — but it breaks the prompt structure by accident, which is the more likely failure. Every interpolated block is now sealed. "Trusted" was doing two jobs and hiding the difference. The template called EVALUATORS and DOMAIN_NOTES alike "instruction-level trust" while SKILL.md said a note must never redefine the rubric — read together, an implementer could reasonably give notes rubric authority. Trust and authority are now split explicitly in both files: evaluators is trusted AND authoritative and alone sets the criteria; domain_notes is trusted but NOT authoritative, relied on to understand what the data means and powerless to widen what counts as correct; datapoint content is neither. _load_domain_notes validated nothing past a string check, so a dict silently rendered its keys into the prompt as plausible context and a list of ints crashed inside the join. Both now fail fast with a message naming the expected shape, and malformed JSON is caught rather than surfacing as a bare decode error at import. A broken config reaching the judge as plausible-looking context is a scoring bug that would not look like one. _seal's docstring still said the zero-width space "stops" the text parsing as a closing tag. A model is not a parser and <ZWSP/tag> is still tag-shaped to it; the docstring now says so and stops claiming more than the mechanism delivers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The _BLOCK_TAGS comment claimed every interpolated block was sealed, but EVALUATORS went in raw — so a rubric that quoted </evaluators> could close its own block, the same accident the notes sealing prevents. Seal it as well and the comment is true as written. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by inspecting really-ingested experiment events from a live run of this skill — none of
the three were catchable from the code, and three rounds of review missed them, because two are
properties of Datadog's tag pipeline and one only shows up on a discrete metric.
Datadog normalizes tag values, so a tag is not a byte-faithful channel:
* A leading "+" is rewritten to "_". A tag sent as delta_vs_best:+0.0447 landed as
delta_vs_best:_0.0447 — the sign, which is the entire point of a delta, destroyed. Worse, "-"
survives, so negatives land as -0.1180 while positives land as _0.1180, an asymmetric
encoding a consumer has to reverse-engineer. delta_vs_best now carries the absolute value and
delta_sign:<pos|neg|zero> carries the direction.
* Tag values are lowercased. time_start:2026-07-22T14:31:07Z landed as ...t14:31:07z, which
neither parses as ISO-8601 nor byte-matches the iteration_results row the spec requires it to
match. Now time_start_ms/time_end_ms as integer epoch millis, which pass through untouched.
The quartiles were also the wrong summary for a discrete metric:
* Interpolated quartiles invent values the metric cannot produce. A ground-truth F1 over set
overlap yields per-case values in {0.0, 0.667, 0.8, 1.0}; interpolation between the 9th and
10th sorted values reported q1=0.1667, a number no datapoint scored, dressed as a measurement.
Quartiles are now nearest-rank, so every number in the summary is a score some case got.
* Quartiles alone go blind when most cases are perfect. With 26 of 34 at exactly 1.0,
q1=median=q3=1.0 across iterations while the distribution moved hard (cases at 0.0 fell
10 -> 5). dist_n/dist_zero/dist_perfect are now required, not optional — on such a metric they
are the only part of the summary that carries the change.
Also generalizes the lesson into a standing warning next to the tag spec, so the next tag added
to this event does not repeat it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The schema gained n/zero/perfect but the sentence above it still described the row as scores plus a five-number summary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… pup
Adds a `datadog_backend: mcp|pup` intake field (default mcp) selecting the client for EVERY
Datadog call the run makes — dataset reads, span/trace reads, and the experiment update /
event-submit writes. One switch, not per-call, so a run's provenance is never mixed;
`backend_used` is recorded in config.json because two runs on different backends are not
strictly comparable.
pup has full parity for what this skill touches: datasets records / records-full, spans
search / get-trace / get-details / get-content / expand, experiments update / events submit.
A substitution table maps each purpose to both clients and is declared to apply file-wide, so
the steps below keep naming MCP tools without every one needing a pup twin.
Failure policy is deliberately asymmetric, per the requested behaviour:
* backend=pup and pup missing or unauthenticated -> STOP. No fallback to MCP: the user asked
for pup explicitly, so quietly using another client would make the recorded provenance
false. A PUP_BIN override is honoured first so a dev checkout works without installing.
* backend=mcp and an MCP call fails -> fall back to pup, loudly, recording backend_fallback.
Everything below was verified against pup 1.7.0 rather than assumed, and three findings are
documented because each would otherwise bite silently:
* Reads are wrapped in {status, data, metadata} in agent mode, where `data` is exactly the
MCP body. Verified record-for-record identical on the same dataset call.
* `experiments update` and `experiments events submit` take the experiment id POSITIONALLY,
and events submit's payload omits the experiment_id key entirely.
* A non-zero pup exit does NOT mean the write failed. `experiments create` and
`experiments update` fail while deserializing the API response (missing field `config`;
EOF on update's empty body) and exit non-zero AFTER the write has landed — both confirmed
applied by reading back. So pup writes must be verified by reading state back, never by
exit code, or a retry loop will double-write. The setup gate is amended accordingly.
`events submit` is well behaved and returns MCP's exact response shape.
Also corrects a claim made earlier in this file: the inability to read back submitted
summary-level experiment metrics is a PLATFORM limitation, not an MCP one. pup's
`experiments events list` and `experiments summary` both report zero events for an experiment
whose submission was accepted, so the fallback rescues failed calls and cannot rescue missing
reads.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Installed pup 1.8.0 from main and re-ran every call the skill makes. Three corrections to what the previous commit claimed, all found by testing rather than reading: pup CANNOT serve a trace-derived data source. The four `spans get-*` drill-downs return HTTP 404 from /api/unstable/llm-obs-mcp/v1/trace/* under API-key auth, while the MCP tools read the very same trace fine — verified side by side on one trace id, where MCP returned 36 spans and pup 404'd. So it is a pup gap, not a platform gap. `spans search` works; only the per-trace reads fail. Step 1 needs get-trace / get-details / get-content to pull the `messages` field, so pairing backend=pup with `trace_ids` or `ml_app` produces a run that dies in Step 1. That is now an intake-gate STOP: the user picks mcp or supplies a dataset_id / local_dataset_path. Silently falling back to MCP would falsify the recorded provenance, and starting the run anyway would waste it. backend=pup remains fine for local_dataset_path and dataset_id. The CLI is not stable across minor versions. `experiments events submit` took `--file <path>` in 1.7.0 and takes `--metrics '<json array>'` in 1.8.0 — the flag documented one commit ago is already wrong. The table now says to check `pup --version` and `pup agent schema` for the installed build instead of trusting it verbatim, and `backend_version` is recorded in config.json next to backend_used. `spans search --from` also wants `7d`, not `now-7d`. The exit-code warning holds on 1.8.0: `experiments update` still exits 1 with "EOF while parsing a value" while the write lands (confirmed by reading status and metadata back), and `experiments create` still exits 1 on `missing field config` after creating. `events submit` is still the well-behaved one — exit 0 and MCP's exact response shape. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…efault window The previous commit claimed pup's four per-trace commands (get-trace, get-details, get-content, expand) return HTTP 404 under API-key auth, called it a pup gap rather than a platform gap, and added an intake-gate STOP forbidding backend=pup with a trace_ids/ml_app data source. All of that was wrong, and the error was mine. Every pup `llm-obs spans *` command defaults to --from 1h. The trace I tested was older than an hour, so the API answered 404 with detail "no spans found for trace <id>" — I truncated stderr at 110 characters, never saw the detail, and read a route-level 404 into it. Worse, I had passed --from now-7d to the MCP tool for the same trace and nothing to pup, so the comparison that "proved" a pup gap was measuring two different time windows. With an explicit window all four succeed on 1.8.0 and get-trace returns the same 36-span structure as MCP on the same id. pup can serve every data source the skill supports, trace_ids and ml_app included, so the intake STOP is removed and the table rows are corrected. What replaces it is the finding that actually matters: the 1h default, that a stale window produces a 404 which is trivially mistaken for an unsupported route, and the instruction to read the whole error body before declaring a command unsupported. The MCP tools default to now-1d, so the same trace id can succeed on MCP and 404 on pup purely from the default — a window difference, not a capability difference. The other findings from that commit stand and were re-verified: the --file to --metrics signature change between 1.7.0 and 1.8.0, and `experiments update`/`create` exiting non-zero after the write lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The table had accumulated noise from the retraction: `--from 7d --to now` repeated verbatim on four rows, a parenthetical flag note on a fifth, and inline warning text in a sixth. Rewritten as four columns with a two-marker legend, the shared flags factored into the legend, and the `pup llm-obs` prefix stated once instead of on every row. Also states plainly that every row was run successfully against pup 1.8.0 and that there are no unsupported purposes — the previous shape still read as though some rows were doubtful. Two leftovers fixed while here: the substitution-rule paragraph still cited the 1.7.0 `events submit --file` flag, and the "pup rejects the MCP-style now-7d" gotcha had lived only in a row note that the rewrite removed, so it moved into the window section where it belongs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit's table rewrite replaced everything between the table header and the substitution paragraph — and both the 1-hour-window section and the version-sensitivity section lived in that span, so both were silently wiped. The window finding is the whole point of the preceding retraction (a stale window returns a 404 that reads like an unsupported route), and the version note is what stops a reader trusting a flag that changed between 1.7.0 and 1.8.0. Both restored verbatim after the marker legend, where the legend's "see below" pointers now resolve. Also fixes the two remaining `events submit --file` references (1.7.0's flag) in the substitution paragraph and the reporting section, and folds the "pup rejects the MCP-style now-7d" gotcha into the restored window section — it had lived only in a table row note that the rewrite dropped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rified Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pup backend was reading the eval corpus with `datasets records`, which posts to an MCP-token-budget endpoint that trims to a response-size budget (~19 records on a dataset with sizeable inputs), reports truncated: true, and returns no cursor. A run built that way silently measures a different corpus than an mcp run of the same dataset_id — different split, different class balance, nothing comparable. That is exactly what happened on a 50-record dataset: the pup run scored 19 records while two earlier mcp runs scored 50. DataDog/pup#678 (merged) adds `datasets records-all`, which pages the REST route internally and returns every record in one call. The table now makes that the corpus path, demotes `records` to browsing with its cap marked, and Step 1's dataset_id branch names the per-backend command explicitly: page next_cursor on mcp, records-all on pup. Two guards, both learned the hard way: * Absence of records-all is a STOP under datadog_backend: pup, like a missing binary — continuing on the capped path yields a corpus that is a truncation artifact. Detect it by running the subcommand and checking the exit code, NOT by --help, which exits 0 for unknown subcommands on some builds and reports a feature present when it is absent. * After loading, assert the materialized count equals the dataset's true size before splitting. That is the cheap check that would have caught the 19-of-50 truncation immediately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ge either
The previous commit said the mcp backend loads the full corpus by paging
get_llmobs_dataset_records until next_cursor is empty. That is wrong. Verified against the live
tool at limit=100: returned 19, truncated true, next_cursor None, with __nested_object__
placeholders. The tool's schema documents next_cursor but the server never populates it, because
the tool posts to the same response-budget endpoint that pup's capped `records` uses. The cap is a
property of the endpoint, not of either client.
Consequences now documented:
* No MCP tool can enumerate a dataset larger than ~19 records. get_llmobs_full_dataset_records
caps at 3 per call and needs the id list you cannot obtain, so it is not a workaround.
* On mcp such a corpus must come from a direct REST call to
GET /api/unstable/llm-obs/v1/datasets/{id}/records paging meta.after — the same route pup
wraps — and the data_note must say so, because it deviates from "every call went through the
backend".
* pup is therefore AHEAD of mcp here, not merely on par: records-all is the only first-class
command for a full dataset. For a large dataset and a single-client run, prefer pup.
Also relevant to earlier runs in this experiment: the two runs labelled "mcp" loaded their 50
records via a direct curl to that REST route, not through an MCP tool, so they were never pure-MCP
runs. The skill now names that deviation instead of leaving it implied.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…he pending fix
The exit-code warning said pup's experiment writes "fail while deserializing the API's response"
without saying why, which makes it impossible to tell whether an installed build is affected.
Both causes are now stated, each confirmed against the live API:
* `experiments update` — a successful PATCH answers HTTP 200 with a ZERO-BYTE body, which the
generated typed client hands to serde_json::from_str, failing with "EOF while parsing a value".
* `experiments create` — the 200 response omits `config`, a field the generated model requires,
giving "missing field config".
Neither is a request failure. In one run this fired four times and all four writes had applied.
Also records that DataDog/pup#682 fixes both, by routing these two writes through pup's raw client
(as every other llm-obs command already does) and making parse_response_json treat an empty
successful body as JSON null. With that build update exits 0 and prints
{"experiment_id": ..., "status": "updated"}, and create exits 0 returning the new id.
That PR is OPEN, NOT MERGED, and the skill says so rather than describing unreleased behaviour as
current. The detection advice is deliberately not "check the version": run the command and compare
the exit code against a read-back, the same discipline the rest of this file uses — a version
number would not have caught the --help probe that reported records-all present on a binary that
lacked it.
The setup gate keeps read-back unconditionally, since it is correct on both builds and avoids
branching on which one is installed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tillwf
changed the title
gsvigruha
approved these changes
Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Six changes to the
agent-observability-auto-experimentskill. Docs only, plus one Python template.1. The failure census no longer hands the agent its own answers
It used to list the buckets up front (
wrong_retrieval,format/parse, …), so every failure landed in a category that already existed. Now two phases: parallel sub-agents describe failures with no category vocabulary, then the orchestrator names the buckets from what the descriptions say.census.jsonnow stores the per-datapoint descriptions plusfailing_total/described, so buckets are auditable and partial coverage is reported as "15 of 47 inspected" instead of passing for complete.2.
domain_notes— product context the code doesn't carryNew optional list of strings, injected into sub-agent briefings, census describers, and the judge prompt. Grows mid-run: a corrected misread is appended, not applied once. Trusted as context, never authoritative — it can't redefine
evaluators.3. Score distribution reaches LLM-Obs
Each iteration already computed a
score_distribution; none of it left the machine. Now published asdist_*tags:dist_n/dist_zero/dist_perfectcounts plus a nearest-rank five-number summary.The counts matter more than the quartiles here — with 26 of 34 cases at exactly 1.0,
q1=median=q3=1.0looks frozen whiledist_zeromoved 11 → 5.4. Tag encoding fixes (found by inspecting ingested events)
+to_, sodelta_vs_best:+0.0447landed as_0.0447— the sign destroyed. Now unsigned magnitude + a separatedelta_signtag.time_start_ms/time_end_msepoch millis.q1=0.1667, a value no datapoint scored. Now nearest-rank.5.
datadog_backend: mcp | pupNew intake field (default
mcp) selecting the client for every Datadog call, so a run's provenance is never mixed.backend_used/backend_versionrecorded.Asymmetric on failure:
puprequested but unavailable → STOP (falling back would falsify provenance);mcpcall failing → fall back to pup, loudly.Corpus loading is the one real difference. Neither client's "records" tool can page past ~19 records — both hit the same response-budget endpoint with no cursor. pup#678 (merged) added
datasets records-all, so pup is ahead of MCP here, and that's the documented corpus path. The loaded count must be asserted against the dataset size before splitting.6. Judge prompt scaffold in the harness template
build_judge_prompt()assembles the prompt with trusted blocks first and untrusted datapoint content last, each in its own sealed delimiter. Replaces a prose instruction that every run re-implemented by hand.Verified: three runs of the same experiment — two on MCP, one entirely on pup — with identical corpus, split and scorer gave baselines of 0.6357 / 0.6337 / 0.6412, agreeing within 0.008. The backend changes transport, not measurement.
Two caveats for reviewers:
spans get-*commands return HTTP 404 and can't serve trace-derived sources. That was wrong — pup defaults to a 1-hour window and I misread the resulting 404. Retracted in a later commit; all four commands work with an explicit--from.experiments update/createon released pup exit non-zero after the write lands (HTTP 200 with an empty body; response missingconfig). pup#682 fixes both but is not merged, so the skill assumes the broken behaviour and verifies writes by read-back.Supersedes #88, whose commits were folded in here.