Skip to content

[BE]: register replicate and watsonx as canonical providers #7894

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Anuj7411 wants to merge 1 commit into
base: main
Choose a base branch
from
+37 −1
Draft

[BE]: register replicate and watsonx as canonical providers #7894

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion apps/opik-backend/src/main/java/com/comet/opik/domain/cost/CostService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,9 @@ public class CostService {
Map.entry("zai", "zai"),
Map.entry("z-ai", "zai"),
Map.entry("sambanova", "sambanova"),
Map.entry("nebius", "nebius"));
Map.entry("nebius", "nebius"),
Map.entry("replicate", "replicate"),
Map.entry("watsonx", "watsonx")));
Comment on lines +58 to +59

@baz-reviewer baz-reviewer Bot Aug 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Watsonx transcription costs remain zero

Adding watsonx to PROVIDERS_MAPPING admits watsonx/whisper-large-v3-turbo, but its input_cost_per_second and output_cost_per_second fields have no representation in ModelCostData, so buildModelPrice creates zero rates; with mode: audio_transcription, resolveCalculator falls through to defaultCost, and public calculateCost(..., usage, null) reports zero for non-zero transcription usage. Should we add the per-second fields and a duration-keyed transcription calculator with a regression test, or exclude this model/provider shape from PROVIDERS_MAPPING?

Severity

Want Baz to fix this for you? Activate Fixer

Other fix methods

Fix in Cursor

Prompt for AI Agents 
Before applying, verify this suggestion against the current code. In
`apps/opik-backend/src/main/java/com/comet/opik/domain/cost/CostService.java` around
lines 58-59, fix the `watsonx` provider registration because `whisper-large-v3-turbo`
uses `input_cost_per_second` and `output_cost_per_second`, which the current pricing
model and `audio_transcription` calculator ignore, resulting in zero cost. Prefer adding
the per-second fields, a duration-based transcription calculator, and regression
coverage for `calculateCost(..., usage, null)`; if that support is not intended, remove
`watsonx` from `PROVIDERS_MAPPING` so the registration does not claim unsupported
pricing coverage.


// Online evaluation (and OTel ingestion) resolve models to LlmProvider serialized values whose names
// differ from the canonical price-table vocabulary. Normalize those to the single canonical provider
Expand Down
34 changes: 34 additions & 0 deletions apps/opik-backend/src/test/java/com/comet/opik/domain/cost/CostServiceTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -973,4 +973,38 @@ private static Stream<Arguments> provideAggregatorRoutedVendorCases() {
// custom-llm hits the same fallback, as it does for perplexity and moonshot.
Arguments.of("z-ai/glm-4.5", "custom-llm", "0.00104"));
}

/**
* Covers registering {@code replicate} as a canonical provider so that the 40 non-zero-cost
* entries in {@code model_prices_and_context_window.json} tagged with
* {@code litellm_provider: "replicate"} are no longer silently dropped at load time. No Replicate
* model publishes cache rates today, so all Replicate requests route through
* {@link SpanCostCalculator#textGenerationCost}.
*/
@Test
void calculateCostHandlesReplicateModels() {
// replicate/openai/o1: input 1.5e-05, output 6e-05
// 1000 * 1.5e-05 + 200 * 6e-05 = 0.027
BigDecimal cost = CostService.calculateCost("replicate/openai/o1", "replicate",
Map.of("prompt_tokens", 1000, "completion_tokens", 200), null);

assertThat(cost).isEqualByComparingTo("0.027");
}

/**
* Covers registering {@code watsonx} as a canonical provider so that the 28 non-zero-cost
Comment on lines +984 to +995

@baz-reviewer baz-reviewer Bot Aug 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicated provider cost tests

calculateCostHandlesReplicateModels and calculateCostHandlesWatsonxModels duplicate the same setup and assertion, so changes must be kept in sync — should we consolidate them into one @ParameterizedTest with a @MethodSource or @CsvSource for the differing values?

Severity

Want Baz to fix this for you? Activate Fixer

Other fix methods

Fix in Cursor

Prompt for AI Agents 
Before applying, verify this suggestion against the current code. In
`apps/opik-backend/src/test/java/com/comet/opik/domain/cost/CostServiceTest.java` around
lines 984-1008, consolidate `calculateCostHandlesReplicateModels` and
`calculateCostHandlesWatsonxModels` into one `@ParameterizedTest`. Add a `@MethodSource`
(or `@CsvSource`) supplying each model, provider, token-usage map, and expected cost,
then run the shared `CostService.calculateCost` invocation and assertion for both cases,
preserving the existing expected values and adding any necessary JUnit
parameterized-test imports.

* entries in {@code model_prices_and_context_window.json} tagged with
* {@code litellm_provider: "watsonx"} are no longer silently dropped at load time. No Watsonx
* model publishes cache rates today, so all Watsonx requests route through
* {@link SpanCostCalculator#textGenerationCost}.
*/
@Test
void calculateCostHandlesWatsonxModels() {
// watsonx/openai/gpt-oss-120b: input 1.5e-07, output 6e-07
// 1000 * 1.5e-07 + 200 * 6e-07 = 0.00027
BigDecimal cost = CostService.calculateCost("watsonx/openai/gpt-oss-120b", "watsonx",
Map.of("prompt_tokens", 1000, "completion_tokens", 200), null);

assertThat(cost).isEqualByComparingTo("0.00027");
}
}