fix(llmobs): capture openai cache write input tokens#18989
Open
Yun-Kim wants to merge 3 commits into
Open
Conversation
OpenAI reports prompt-cache writes via `cache_write_tokens` on the usage details of both the chat completions (`prompt_tokens_details`) and responses (`input_tokens_details`) APIs for newer models (GPT-5.6+). Extract it into the `cache_write_input_tokens` LLMObs metric, alongside the existing `cache_read_input_tokens` (from `cached_tokens`). Adds VCR cassettes recorded against gpt-5.6-sol and tests covering both APIs plus the APM shadow-metric path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codeowners resolved as |
Circular import analysis
|
Yun-Kim
commented
Jul 10, 2026
Yun-Kim
commented
Jul 10, 2026
Yun-Kim
commented
Jul 10, 2026
Contributor
🎉 All green!🧪 All tests passed 🔗 Commit SHA: 2117bab | Docs | Datadog PR Page | Give us feedback! |
Yun-Kim
commented
Jul 10, 2026
- Remove explanatory comment in openai.py (self-evident code) - Reframe release note as a fix - Drop cache-read requests/assertions and their VCR cassettes from the new prompt-caching tests; keep the cache-write scenario only Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The kwarg is rejected by older openai SDK versions (e.g. 1.0.0, 1.30.1) in the riot matrix, failing those shards with TypeError. It is not needed for the test, and VCR matches on path only, so the cassette is unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
This change is marked for backport to 4.11 and it does not conflict with that branch. |
ZStriker19
approved these changes
Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Captures OpenAI prompt-cache write token details on LLM Observability spans for instrumented chat completion and responses API calls.
Newer OpenAI models (GPT-5.6+) report the number of prompt tokens written to the cache via
cache_write_tokenson the usage details object —usage.prompt_tokens_detailsfor the Chat Completions API andusage.input_tokens_detailsfor the Responses API. The integration already extractedcached_tokens(cache reads) from that same object; this change extractscache_write_tokensinto thecache_write_input_tokensLLMObs metric, mirroring how Anthropic/Bedrock/LiteLLM already report cache writes.Because it flows through
_extract_llmobs_metrics_tags, the metric also propagates to the APM shadow tag (_dd.llmobs.cache_write_input_tokens) and to the streamed-response path (usage lives on the first chunk).API reference: https://developers.openai.com/api/docs/guides/prompt-caching
Testing
test_chat_completion_prompt_caching_cache_writeandtest_responses_prompt_caching_cache_write, plus an assertion on the existingtest_shadow_tags_chat_completion_with_cache_tokens.gpt-5.6-sol(write call:cache_write_tokens=1217,cached_tokens=0; read call:cached_tokens=1216,cache_write_tokens=0). Cache reads are per-replica and flaky back-to-back on the live API, so the chat read cassette's usage block was normalized to the verified cache-hit values.scripts/run-tests --venv ec404a0 tests/contrib/openai/test_openai_llmobs.py -- -- -k "prompt_caching or cache_tokens"-> 7 passed.Risks
Low. Additive metric extraction gated on
is not None; models that don't reportcache_write_tokensare unaffected (no key set). No public API change.Additional Notes
Claude session:
9f8553c5-6d78-40bf-8a8e-dc740ed41a73Resume:
claude --resume 9f8553c5-6d78-40bf-8a8e-dc740ed41a73🤖 Generated with Claude Code