Skip to content

fix(llmobs): capture openai cache write input tokens#18989

Open
Yun-Kim wants to merge 3 commits into
mainfrom
llmobs-openai-cache-write
Open

fix(llmobs): capture openai cache write input tokens#18989
Yun-Kim wants to merge 3 commits into
mainfrom
llmobs-openai-cache-write

Conversation

@Yun-Kim

@Yun-Kim Yun-Kim commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Description

Captures OpenAI prompt-cache write token details on LLM Observability spans for instrumented chat completion and responses API calls.

Newer OpenAI models (GPT-5.6+) report the number of prompt tokens written to the cache via cache_write_tokens on the usage details object — usage.prompt_tokens_details for the Chat Completions API and usage.input_tokens_details for the Responses API. The integration already extracted cached_tokens (cache reads) from that same object; this change extracts cache_write_tokens into the cache_write_input_tokens LLMObs metric, mirroring how Anthropic/Bedrock/LiteLLM already report cache writes.

Because it flows through _extract_llmobs_metrics_tags, the metric also propagates to the APM shadow tag (_dd.llmobs.cache_write_input_tokens) and to the streamed-response path (usage lives on the first chunk).

API reference: https://developers.openai.com/api/docs/guides/prompt-caching

Testing

  • Added test_chat_completion_prompt_caching_cache_write and test_responses_prompt_caching_cache_write, plus an assertion on the existing test_shadow_tags_chat_completion_with_cache_tokens.
  • New VCR cassettes were recorded against gpt-5.6-sol (write call: cache_write_tokens=1217, cached_tokens=0; read call: cached_tokens=1216, cache_write_tokens=0). Cache reads are per-replica and flaky back-to-back on the live API, so the chat read cassette's usage block was normalized to the verified cache-hit values.
  • scripts/run-tests --venv ec404a0 tests/contrib/openai/test_openai_llmobs.py -- -- -k "prompt_caching or cache_tokens" -> 7 passed.

Risks

Low. Additive metric extraction gated on is not None; models that don't report cache_write_tokens are unaffected (no key set). No public API change.

Additional Notes

Claude session: 9f8553c5-6d78-40bf-8a8e-dc740ed41a73
Resume: claude --resume 9f8553c5-6d78-40bf-8a8e-dc740ed41a73

🤖 Generated with Claude Code

OpenAI reports prompt-cache writes via `cache_write_tokens` on the usage
details of both the chat completions (`prompt_tokens_details`) and responses
(`input_tokens_details`) APIs for newer models (GPT-5.6+). Extract it into the
`cache_write_input_tokens` LLMObs metric, alongside the existing
`cache_read_input_tokens` (from `cached_tokens`).

Adds VCR cassettes recorded against gpt-5.6-sol and tests covering both APIs
plus the APM shadow-metric path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 10, 2026

Copy link
Copy Markdown

Codeowners resolved as

tests/contrib/openai/test_openai_llmobs.py                              @DataDog/ml-observability

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 10, 2026

Copy link
Copy Markdown

Circular import analysis

⚠️ Existing circular imports

There are 4974 circular imports that already exist on the base branch and have not been changed by this PR.

Show existing cycles (showing 5 of 4974 shortest)
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.kafka -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.botocore -> ddtrace.internal.datastreams
ddtrace.internal.core -> ddtrace._trace.span -> ddtrace.internal.core
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.google_cloud_pubsub -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.kombu -> ddtrace.internal.datastreams

To see all cycles, download the cycles-base.json and cycles-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/cycles.py compare cycles-base.json cycles-pr.json

Comment thread ddtrace/llmobs/_integrations/openai.py Outdated
@Yun-Kim Yun-Kim changed the title feat(openai): capture cache write input tokens for LLMObs spans fix(openai): capture cache write input tokens for LLMObs spans Jul 10, 2026
@Yun-Kim Yun-Kim changed the title fix(openai): capture cache write input tokens for LLMObs spans fix(llmobs): capture openai cache write input tokens Jul 10, 2026
Comment thread releasenotes/notes/openai-cache-write-tokens-1a880b4309652378.yaml Outdated
Comment thread releasenotes/notes/openai-cache-write-tokens-1a880b4309652378.yaml Outdated
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 2117bab | Docs | Datadog PR Page | Give us feedback!

Comment thread tests/contrib/openai/test_openai_llmobs.py Outdated
Yun-Kim and others added 2 commits July 10, 2026 15:29
- Remove explanatory comment in openai.py (self-evident code)
- Reframe release note as a fix
- Drop cache-read requests/assertions and their VCR cassettes from the
  new prompt-caching tests; keep the cache-write scenario only

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The kwarg is rejected by older openai SDK versions (e.g. 1.0.0, 1.30.1)
in the riot matrix, failing those shards with TypeError. It is not
needed for the test, and VCR matches on path only, so the cassette is
unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

This change is marked for backport to 4.11 and it does not conflict with that branch.
The command used to test backporting was

git fetch origin 4.11 && git checkout origin/4.11 && git checkout -b backport-18989-to-4.11 && git cherry-pick -x --mainline 1 b59361b6a3fe34f87d98416080ddbfe360cf30d2

@Yun-Kim Yun-Kim marked this pull request as ready for review July 10, 2026 20:56
@Yun-Kim Yun-Kim requested review from a team as code owners July 10, 2026 20:56
@Yun-Kim Yun-Kim requested review from avara1986 and wconti27 July 10, 2026 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants