fix: apply OpenAI prompt-cache discount to cache_read tokens (#2065) - #2070
Open
aranya-chatterjee wants to merge 1 commit into
Open
fix: apply OpenAI prompt-cache discount to cache_read tokens (#2065)#2070aranya-chatterjee wants to merge 1 commit into
aranya-chatterjee wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
_extract_usage_tokensingpt_researcher/utils/costs.pyonly read theflat
input_tokens/output_tokensfields off LangChain's standardizedusage_metadata, andcalculate_llm_cost's OpenAI branch priced thefull
input_tokenscount at the standard rate. Neither looked atinput_token_details.cache_read, the portion of input tokens that hitOpenAI's automatic prompt caching and are billed at a discount.
Any research run that benefits from provider-side caching gets its cost
overreported.
This is the same class of gap as #1986/#1989 (Anthropic cache tokens
priced at $0), but on the OpenAI/generic branch, in the opposite
direction: overcounted here instead of undercounted there.
Fix
_extract_usage_tokensnow also extractsinput_token_details.cache_readand returns it alongside input/output tokens.
calculate_llm_costsplitsinput_tokensinto its cached andnon-cached portions, pricing the cached portion at
OPENAI_CACHED_INPUT_DISCOUNT(50%, per OpenAI's pricing page) insteadof the full input rate.
fallback path.
Testing
Added
TestOpenAICachedInputPricingtotests/test_costs.py, covering:cache_read: 0(matches baseline)Verified against the reporter's exact repro from #2065:
0.0145regardless of cache hits0.0145(no cache) vs0.008875(4500/5000 tokens cached)Fixes #2065