Skip to content

fix(llm-obs): use raw HTTP for annotation schema and annotation commands - #729

Merged
platinummonkey merged 1 commit into
DataDog:mainfrom
gsvigruha:gergely.svigruha/annotation-raw-http-fix
Aug 13, 2026
Merged

fix(llm-obs): use raw HTTP for annotation schema and annotation commands#729
platinummonkey merged 1 commit into
DataDog:mainfrom
gsvigruha:gergely.svigruha/annotation-raw-http-fix

Conversation

@gsvigruha

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #728. That PR merged while I was still integration-testing, so the fix below never landed — three of the four commands it added are currently broken on main against real API responses.

The generated client declares annotation_schema, annotations and annotation_ids as non-Option, but the API returns null for each in ordinary cases:

Command Failure on main today
annotation-queues schema get A queue with no schema yet returns "annotation_schema": nullinvalid type: null, expected a mapping. Most queues have no schema, so this is the common path, not an edge case.
annotation-queues annotations upsert Per-item failures come back as HTTP 200 with "annotations": null beside a populated errors array → invalid type: null, expected a sequence, turning a readable partial-failure report into an opaque serde error.
annotation-queues annotations delete Same exposure — both annotation_ids and errors are non-Option Vecs.

schema update was unaffected: it echoes back the schema just written, which is never null.

Changes

  • Route all four commands through raw_client (src/commands/llm_obs.rs), matching the existing precedent in this file — see experiments_update, which uses the raw path for the same class of generated-client mismatch. URL paths and request shapes are unchanged.
  • Revert the UNSTABLE_OPS additions (src/client.rs), since the typed SDK methods are no longer called. This file is now identical to its pre-feat(llm-obs): add annotation queue label schema and annotation commands #728 state.
  • Add regression tests pinning the null-response shapes, captured from real API responses rather than hand-written.

Testing

Verified against a live org, not only mocks — which is how these bugs were missed the first time. mock_any matches any path and any body, so mock-only tests could not catch a response shape I had not anticipated.

  • schema get on a schema-less queue → returns "annotation_schema": null cleanly (previously a serde error)
  • schema get on a populated queue → both label schemas returned
  • schema update → 2xx, and a follow-up schema get confirms the schema persisted
  • annotations upsert with an unknown interaction → renders the 200 partial-failure report (previously a serde error)
  • annotations delete with an unknown id → renders the per-item error report
  • All three writes against a nonexistent queue → distinct, endpoint-specific 404s, confirming correct URL routing

Test scaffolding was a throwaway queue, created and then deleted; deletion confirmed by a 404.

Tests 9 → 13. cargo fmt --check and cargo clippy --all-targets -- -D warnings clean.

Pre-existing suite note: four tests in dbm, monitors, security and traces fail under parallel execution on main as well (verified on a clean tree: 1716 passed, same 4 failed). They pass individually — pre-existing env-var races, unrelated to this change.

Out of scope

pup llm-obs annotation-queues list fails on real data with the same root cause (invalid type: null, expected a sequence). It predates #728 and is untouched here — worth its own issue.


🤖 Generated with Claude Code

Integration testing against the real API showed three of the four new
commands failing on ordinary responses. The generated client declares
`annotation_schema`, `annotations` and `annotation_ids` as non-`Option`,
but the API returns `null` for each in common cases:

- `schema get` on a queue with no schema yet answers
  `"annotation_schema": null` → "invalid type: null, expected a mapping".
  Most queues have no schema, so this was the common case, not an edge.
- `annotations upsert` reports per-item failures with HTTP 200 and
  `"annotations": null` beside a populated `errors` array → "invalid type:
  null, expected a sequence", turning a readable partial-failure report
  into an opaque serde error.

Switch all four to the raw client, matching the existing precedent in this
file (see `experiments_update`). Request shapes and paths are unchanged.

- Route the four commands through raw_client (src/commands/llm_obs.rs)
- Revert the now-unneeded UNSTABLE_OPS entries (src/client.rs)
- Add regression tests pinning the null-response shapes, captured from
  real responses rather than hand-written (src/commands/llm_obs.rs)

Verified against the real API: all four commands succeed, including the
schema-less queue and the 200 partial-failure report.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gsvigruha
gsvigruha requested a review from a team as a code owner August 13, 2026 20:50
@platinummonkey
platinummonkey merged commit 08a6438 into DataDog:main Aug 13, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants