fix(llm-obs): use raw HTTP for annotation schema and annotation commands - #729
Merged
platinummonkey merged 1 commit intoAug 13, 2026
Conversation
Integration testing against the real API showed three of the four new commands failing on ordinary responses. The generated client declares `annotation_schema`, `annotations` and `annotation_ids` as non-`Option`, but the API returns `null` for each in common cases: - `schema get` on a queue with no schema yet answers `"annotation_schema": null` → "invalid type: null, expected a mapping". Most queues have no schema, so this was the common case, not an edge. - `annotations upsert` reports per-item failures with HTTP 200 and `"annotations": null` beside a populated `errors` array → "invalid type: null, expected a sequence", turning a readable partial-failure report into an opaque serde error. Switch all four to the raw client, matching the existing precedent in this file (see `experiments_update`). Request shapes and paths are unchanged. - Route the four commands through raw_client (src/commands/llm_obs.rs) - Revert the now-unneeded UNSTABLE_OPS entries (src/client.rs) - Add regression tests pinning the null-response shapes, captured from real responses rather than hand-written (src/commands/llm_obs.rs) Verified against the real API: all four commands succeed, including the schema-less queue and the 200 partial-failure report. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
platinummonkey
approved these changes
Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #728. That PR merged while I was still integration-testing, so the fix below never landed — three of the four commands it added are currently broken on
mainagainst real API responses.The generated client declares
annotation_schema,annotationsandannotation_idsas non-Option, but the API returnsnullfor each in ordinary cases:maintodayannotation-queues schema get"annotation_schema": null→invalid type: null, expected a mapping. Most queues have no schema, so this is the common path, not an edge case.annotation-queues annotations upsert"annotations": nullbeside a populatederrorsarray →invalid type: null, expected a sequence, turning a readable partial-failure report into an opaque serde error.annotation-queues annotations deleteannotation_idsanderrorsare non-OptionVecs.schema updatewas unaffected: it echoes back the schema just written, which is never null.Changes
raw_client(src/commands/llm_obs.rs), matching the existing precedent in this file — seeexperiments_update, which uses the raw path for the same class of generated-client mismatch. URL paths and request shapes are unchanged.UNSTABLE_OPSadditions (src/client.rs), since the typed SDK methods are no longer called. This file is now identical to its pre-feat(llm-obs): add annotation queue label schema and annotation commands #728 state.Testing
Verified against a live org, not only mocks — which is how these bugs were missed the first time.
mock_anymatches any path and any body, so mock-only tests could not catch a response shape I had not anticipated.schema geton a schema-less queue → returns"annotation_schema": nullcleanly (previously a serde error)schema geton a populated queue → both label schemas returnedschema update→ 2xx, and a follow-upschema getconfirms the schema persistedannotations upsertwith an unknown interaction → renders the 200 partial-failure report (previously a serde error)annotations deletewith an unknown id → renders the per-item error reportTest scaffolding was a throwaway queue, created and then deleted; deletion confirmed by a 404.
Tests 9 → 13.
cargo fmt --checkandcargo clippy --all-targets -- -D warningsclean.Pre-existing suite note: four tests in
dbm,monitors,securityandtracesfail under parallel execution onmainas well (verified on a clean tree: 1716 passed, same 4 failed). They pass individually — pre-existing env-var races, unrelated to this change.Out of scope
pup llm-obs annotation-queues listfails on real data with the same root cause (invalid type: null, expected a sequence). It predates #728 and is untouched here — worth its own issue.🤖 Generated with Claude Code