Skip to content

fix: apply OpenAI prompt-cache discount to cache_read tokens (#2065) - #2070

Open
aranya-chatterjee wants to merge 1 commit into
assafelovic:mainfrom
aranya-chatterjee:fix/openai-cache-read-pricing
Open

fix: apply OpenAI prompt-cache discount to cache_read tokens (#2065)#2070
aranya-chatterjee wants to merge 1 commit into
assafelovic:mainfrom
aranya-chatterjee:fix/openai-cache-read-pricing

Conversation

@aranya-chatterjee

Copy link
Copy Markdown

Problem

_extract_usage_tokens in gpt_researcher/utils/costs.py only read the
flat input_tokens/output_tokens fields off LangChain's standardized
usage_metadata, and calculate_llm_cost's OpenAI branch priced the
full input_tokens count at the standard rate. Neither looked at
input_token_details.cache_read, the portion of input tokens that hit
OpenAI's automatic prompt caching and are billed at a discount.

Any research run that benefits from provider-side caching gets its cost
overreported.

This is the same class of gap as #1986/#1989 (Anthropic cache tokens
priced at $0), but on the OpenAI/generic branch, in the opposite
direction: overcounted here instead of undercounted there.

Fix

  • _extract_usage_tokens now also extracts input_token_details.cache_read
    and returns it alongside input/output tokens.
  • calculate_llm_cost splits input_tokens into its cached and
    non-cached portions, pricing the cached portion at
    OPENAI_CACHED_INPUT_DISCOUNT (50%, per OpenAI's pricing page) instead
    of the full input rate.
  • Applies to both the known-model pricing table and the flat-rate
    fallback path.

Testing

Added TestOpenAICachedInputPricing to tests/test_costs.py, covering:

Verified against the reporter's exact repro from #2065:

  • Before: both calls returned 0.0145 regardless of cache hits
  • After: 0.0145 (no cache) vs 0.008875 (4500/5000 tokens cached)

Fixes #2065

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OpenAI cost tracking never reads input_token_details.cache_read, so prompt-cache-hit tokens are billed at full price

1 participant