Skip to content

[BE]: register replicate and watsonx as canonical providers - #7894

Draft
Anuj7411 wants to merge 1 commit into
comet-ml:mainfrom
Anuj7411:Anuj7411/register-replicate-watsonx-providers
Draft

[BE]: register replicate and watsonx as canonical providers#7894
Anuj7411 wants to merge 1 commit into
comet-ml:mainfrom
Anuj7411:Anuj7411/register-replicate-watsonx-providers

Conversation

@Anuj7411

Copy link
Copy Markdown
Contributor

Details

CostService.PROVIDERS_MAPPING only registers a subset of the litellm_provider values in model_prices_and_context_window.json. Models whose provider is missing from that map have their price dropped at load time (buildModelPrice returns null), so their spans bill at DEFAULT_COST (zero) and cost tracking / the per-evaluation spend budget silently under-report.

This registers two more unmapped, per-token-priced providers so their bundled prices load: replicate (40 priced models) and watsonx (28 priced models, IBM watsonx.ai). Neither publishes cache rates today, so every request routes through the standard textGenerationCost path and no PROVIDERS_CACHE_COST_CALCULATOR entry is needed.

Part of the ongoing provider-coverage cleanup (#7697).

Testing

Added one @Test per provider asserting a representative model prices to a non-zero, exact expected cost (it returns zero before this change).

Documentation

No documentation changes needed

@github-actions github-actions Bot added java Pull requests that update Java code Backend tests Including test files, or tests related like configuration. labels Aug 18, 2026
Comment on lines +58 to +59
Map.entry("replicate", "replicate"),
Map.entry("watsonx", "watsonx")));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Watsonx transcription costs remain zero

Adding watsonx to PROVIDERS_MAPPING admits watsonx/whisper-large-v3-turbo, but its input_cost_per_second and output_cost_per_second fields have no representation in ModelCostData, so buildModelPrice creates zero rates; with mode: audio_transcription, resolveCalculator falls through to defaultCost, and public calculateCost(..., usage, null) reports zero for non-zero transcription usage. Should we add the per-second fields and a duration-keyed transcription calculator with a regression test, or exclude this model/provider shape from PROVIDERS_MAPPING?

Severity

Want Baz to fix this for you? Activate Fixer

Other fix methods

Fix in Cursor

Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`apps/opik-backend/src/main/java/com/comet/opik/domain/cost/CostService.java` around
lines 58-59, fix the `watsonx` provider registration because `whisper-large-v3-turbo`
uses `input_cost_per_second` and `output_cost_per_second`, which the current pricing
model and `audio_transcription` calculator ignore, resulting in zero cost. Prefer adding
the per-second fields, a duration-based transcription calculator, and regression
coverage for `calculateCost(..., usage, null)`; if that support is not intended, remove
`watsonx` from `PROVIDERS_MAPPING` so the registration does not claim unsupported
pricing coverage.

Comment on lines +984 to +995
@Test
void calculateCostHandlesReplicateModels() {
// replicate/openai/o1: input 1.5e-05, output 6e-05
// 1000 * 1.5e-05 + 200 * 6e-05 = 0.027
BigDecimal cost = CostService.calculateCost("replicate/openai/o1", "replicate",
Map.of("prompt_tokens", 1000, "completion_tokens", 200), null);

assertThat(cost).isEqualByComparingTo("0.027");
}

/**
* Covers registering {@code watsonx} as a canonical provider so that the 28 non-zero-cost

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicated provider cost tests

calculateCostHandlesReplicateModels and calculateCostHandlesWatsonxModels duplicate the same setup and assertion, so changes must be kept in sync — should we consolidate them into one @ParameterizedTest with a @MethodSource or @CsvSource for the differing values?

Severity

Want Baz to fix this for you? Activate Fixer

Other fix methods

Fix in Cursor

Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`apps/opik-backend/src/test/java/com/comet/opik/domain/cost/CostServiceTest.java` around
lines 984-1008, consolidate `calculateCostHandlesReplicateModels` and
`calculateCostHandlesWatsonxModels` into one `@ParameterizedTest`. Add a `@MethodSource`
(or `@CsvSource`) supplying each model, provider, token-usage map, and expected cost,
then run the shared `CostService.calculateCost` invocation and assertion for both cases,
preserving the existing expected values and adding any necessary JUnit
parameterized-test imports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Backend baz: pending java Pull requests that update Java code 🟢 size/S tests Including test files, or tests related like configuration.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant