Skip to content

fix(messages): fall back to stripped model name for cost calculation#4877

Open
JasonCZMeng wants to merge 1 commit intopydantic:mainfrom
JasonCZMeng:fix/issue-4839
Open

fix(messages): fall back to stripped model name for cost calculation#4877
JasonCZMeng wants to merge 1 commit intopydantic:mainfrom
JasonCZMeng:fix/issue-4839

Conversation

@JasonCZMeng
Copy link
Copy Markdown

Summary

When providers like OpenRouter return model names with a date suffix (e.g. openai/gpt-5.2-20251211), genai-prices doesn't recognize them and ModelResponse.cost() raises LookupError. This causes operation.cost to be missing from traces.

Fix

Added a fallback in ModelResponse.cost(): when the initial calc_price lookup fails, strip the trailing -YYYYMMDD date suffix from the model name and retry. The regex -\d{8}$ matches exactly 8-digit date suffixes at the end of the string.

The fallback chain is now:

  1. Try with provider_api_url (most specific)
  2. Try with provider_id
  3. Strip date suffix → try with provider_api_url
  4. Strip date suffix → try with provider_id
  5. Raise LookupError

Tests

  • test_strip_model_date_suffix: Unit test for the suffix stripping helper
  • test_model_response_cost_with_date_suffix: Integration test verifying ModelResponse.cost() works with date-suffixed model names

Fixes #4839

Some providers like OpenRouter return model names with a date suffix
(e.g. `openai/gpt-5.2-20251211`) that `genai-prices` does not
recognize. When the initial cost lookup fails, strip the trailing
`-YYYYMMDD` suffix and retry.

Fixes pydantic#4839
@github-actions github-actions bot added size: S Small PR (≤100 weighted lines) bug Report that something isn't working, or PR implementing a fix labels Mar 27, 2026
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

View 4 additional findings in Devin Review.

Open in Devin Review

Comment on lines +1785 to +1814
try:
return calc_price(
self.usage,
self.model_name,
provider_id=self.provider_name,
genai_request_timestamp=self.timestamp,
)
except LookupError:
pass
# Some providers (e.g. OpenRouter) return model names with a date suffix
# like "openai/gpt-5.2-20251211". Strip the suffix and retry.
normalized = _strip_model_date_suffix(self.model_name)
if normalized != self.model_name:
if self.provider_url:
try:
return calc_price(
self.usage,
normalized,
provider_api_url=self.provider_url,
genai_request_timestamp=self.timestamp,
)
except LookupError:
pass
return calc_price(
self.usage,
normalized,
provider_id=self.provider_name,
genai_request_timestamp=self.timestamp,
)
raise LookupError(f'Could not find pricing for model {self.model_name!r}')
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Duplicate try/except LookupError blocks violate rule:894 and cause inconsistent error messages

The cost() method repeats the same two-step fallback pattern (try provider_api_url, then try provider_id) twice — once for the original model name and once for the normalized (date-stripped) name. This violates rule:894 from pydantic_ai_slim/pydantic_ai/AGENTS.md ("Consolidate try/except blocks catching the same exception into one block") and rule:559 from agent_docs/index.md ("Consolidate duplicate logic across conditional branches").

Beyond the style violation, there's also an inconsistency: when the model name has a date suffix and all lookups fail, the final unwrapped calc_price on line 1808 raises a LookupError with calc_price's own message (referencing the normalized name). But when there's no date suffix, the explicit raise LookupError(...) on line 1814 references the original name. A loop-based consolidation would fix both issues.

Suggested change
try:
return calc_price(
self.usage,
self.model_name,
provider_id=self.provider_name,
genai_request_timestamp=self.timestamp,
)
except LookupError:
pass
# Some providers (e.g. OpenRouter) return model names with a date suffix
# like "openai/gpt-5.2-20251211". Strip the suffix and retry.
normalized = _strip_model_date_suffix(self.model_name)
if normalized != self.model_name:
if self.provider_url:
try:
return calc_price(
self.usage,
normalized,
provider_api_url=self.provider_url,
genai_request_timestamp=self.timestamp,
)
except LookupError:
pass
return calc_price(
self.usage,
normalized,
provider_id=self.provider_name,
genai_request_timestamp=self.timestamp,
)
raise LookupError(f'Could not find pricing for model {self.model_name!r}')
# Build candidate model names: original first, then date-stripped variant.
names = [self.model_name]
normalized = _strip_model_date_suffix(self.model_name)
if normalized != self.model_name:
names.append(normalized)
for name in names:
if self.provider_url:
try:
return calc_price(
self.usage,
name,
provider_api_url=self.provider_url,
genai_request_timestamp=self.timestamp,
)
except LookupError:
pass
try:
return calc_price(
self.usage,
name,
provider_id=self.provider_name,
genai_request_timestamp=self.timestamp,
)
except LookupError:
pass
raise LookupError(f'Could not find pricing for model {self.model_name!r}')
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +1392 to +1407
def test_strip_model_date_suffix():
"""_strip_model_date_suffix removes YYYYMMDD date suffixes from model names."""
from pydantic_ai.messages import _strip_model_date_suffix

# Date suffix should be stripped
assert _strip_model_date_suffix('openai/gpt-5.2-20251211') == 'openai/gpt-5.2'
assert _strip_model_date_suffix('gpt-4o-2024-08-06-20240806') == 'gpt-4o-2024-08-06'
assert _strip_model_date_suffix('claude-sonnet-4-5-20250514') == 'claude-sonnet-4-5'

# No date suffix — unchanged
assert _strip_model_date_suffix('gpt-5.2') == 'gpt-5.2'
assert _strip_model_date_suffix('openai/gpt-5.2') == 'openai/gpt-5.2'
assert _strip_model_date_suffix('claude-sonnet-4-5') == 'claude-sonnet-4-5'

# Short numbers should NOT be stripped (not 8-digit date)
assert _strip_model_date_suffix('gpt-4o-mini-123') == 'gpt-4o-mini-123'
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Test directly tests private _strip_model_date_suffix function instead of public API

test_strip_model_date_suffix imports and tests _strip_model_date_suffix (a private function prefixed with _) directly. This violates rule:177 from tests/AGENTS.md: "Test through public APIs, not private methods (prefixed with _) or helpers." The date-stripping behavior should be tested through the public ModelResponse.cost() method (which is what test_model_response_cost_with_date_suffix already does), making this test redundant.

Prompt for agents
Remove the test_strip_model_date_suffix function (lines 1392-1407 in tests/test_messages.py) entirely. The date-stripping logic it tests is an internal implementation detail of ModelResponse.cost(), and is already exercised through the public API by test_model_response_cost_with_date_suffix. If additional edge-case coverage is desired (e.g. short numeric suffixes not being stripped), add those as additional test cases within test_model_response_cost_with_date_suffix by constructing ModelResponse objects with those model names.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Report that something isn't working, or PR implementing a fix size: S Small PR (≤100 weighted lines)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

operation.cost missing for OpenRouter with OpenAIChatModel + LiteLLMProvider

1 participant