fix: 🔧 don't double-bill cached tokens in LiteLLM registry cost fallback - #113
Merged
Conversation
Code-review follow-up. OpenAI-style `prompt_tokens` INCLUDES cached tokens, but `catalog.estimate_cost_cents` (written for the Anthropic SDK) expects disjoint buckets where input excludes cache. Passing full `prompt_tokens` as `input_tokens` charged the cached portion at the full input rate AND again at the 0.1x cache-read rate (~5.7x over-count for a 90%-cached call). Only affects the registry cost estimate (unmapped models like GLM-5.2 when LiteLLM reports no real cost) — inflates logged/displayed cost, never pipeline control flow. Anthropic/claude-cli paths unaffected. - litellm_client.py: normalize input_tokens to non-cached (max(prompt_tokens - cache_read, 0)) so cost + UsageLog match the Anthropic-style disjoint accounting. - guard the prose-fallback .strip() against non-str content; correct the tool_choice comment. - add test_cached_tokens_not_double_billed (100k prompt / 90k cached on GLM-5.2 -> 1.767c, was ~10c). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Code-review follow-up to #111. Cached tokens were double-billed in the LiteLLM registry cost fallback: OpenAI-style
prompt_tokensincludes cached tokens, butcatalog.estimate_cost_cents(Anthropic-shaped) expects input to exclude cache — so the cached portion was charged at the full input rate and again at 0.1× (~5.7× over for a 90%-cached call).Bounded impact: only the registry cost estimate (unmapped models like GLM-5.2 when LiteLLM reports no real cost), which feeds logged/displayed cost — never pipeline control flow (rate-limiting gates on evaluation count). Anthropic/claude-cli paths unaffected.
litellm_client.py: normalizeinput_tokensto non-cached (max(prompt_tokens - cache_read, 0)) so cost andUsageLogmatch the Anthropic-style disjoint accounting..strip()against non-str content; fix a misleadingtool_choicecomment.test_cached_tokens_not_double_billed(100k prompt / 90k cached on GLM-5.2 → 1.767¢, was ~10¢).Verified: ruff + mypy clean; litellm test suite (12) passes locally.
🤖 Generated with Claude Code