[BE]: register replicate and watsonx as canonical providers - #7894
[BE]: register replicate and watsonx as canonical providers#7894Anuj7411 wants to merge 1 commit into
Conversation
…so their model prices load
| Map.entry("replicate", "replicate"), | ||
| Map.entry("watsonx", "watsonx"))); |
There was a problem hiding this comment.
Watsonx transcription costs remain zero
Adding watsonx to PROVIDERS_MAPPING admits watsonx/whisper-large-v3-turbo, but its input_cost_per_second and output_cost_per_second fields have no representation in ModelCostData, so buildModelPrice creates zero rates; with mode: audio_transcription, resolveCalculator falls through to defaultCost, and public calculateCost(..., usage, null) reports zero for non-zero transcription usage. Should we add the per-second fields and a duration-keyed transcription calculator with a regression test, or exclude this model/provider shape from PROVIDERS_MAPPING?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`apps/opik-backend/src/main/java/com/comet/opik/domain/cost/CostService.java` around
lines 58-59, fix the `watsonx` provider registration because `whisper-large-v3-turbo`
uses `input_cost_per_second` and `output_cost_per_second`, which the current pricing
model and `audio_transcription` calculator ignore, resulting in zero cost. Prefer adding
the per-second fields, a duration-based transcription calculator, and regression
coverage for `calculateCost(..., usage, null)`; if that support is not intended, remove
`watsonx` from `PROVIDERS_MAPPING` so the registration does not claim unsupported
pricing coverage.
| @Test | ||
| void calculateCostHandlesReplicateModels() { | ||
| // replicate/openai/o1: input 1.5e-05, output 6e-05 | ||
| // 1000 * 1.5e-05 + 200 * 6e-05 = 0.027 | ||
| BigDecimal cost = CostService.calculateCost("replicate/openai/o1", "replicate", | ||
| Map.of("prompt_tokens", 1000, "completion_tokens", 200), null); | ||
|
|
||
| assertThat(cost).isEqualByComparingTo("0.027"); | ||
| } | ||
|
|
||
| /** | ||
| * Covers registering {@code watsonx} as a canonical provider so that the 28 non-zero-cost |
There was a problem hiding this comment.
Duplicated provider cost tests
calculateCostHandlesReplicateModels and calculateCostHandlesWatsonxModels duplicate the same setup and assertion, so changes must be kept in sync — should we consolidate them into one @ParameterizedTest with a @MethodSource or @CsvSource for the differing values?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`apps/opik-backend/src/test/java/com/comet/opik/domain/cost/CostServiceTest.java` around
lines 984-1008, consolidate `calculateCostHandlesReplicateModels` and
`calculateCostHandlesWatsonxModels` into one `@ParameterizedTest`. Add a `@MethodSource`
(or `@CsvSource`) supplying each model, provider, token-usage map, and expected cost,
then run the shared `CostService.calculateCost` invocation and assertion for both cases,
preserving the existing expected values and adding any necessary JUnit
parameterized-test imports.
Details
CostService.PROVIDERS_MAPPINGonly registers a subset of thelitellm_providervalues inmodel_prices_and_context_window.json. Models whose provider is missing from that map have their price dropped at load time (buildModelPricereturns null), so their spans bill atDEFAULT_COST(zero) and cost tracking / the per-evaluation spend budget silently under-report.This registers two more unmapped, per-token-priced providers so their bundled prices load:
replicate(40 priced models) andwatsonx(28 priced models, IBM watsonx.ai). Neither publishes cache rates today, so every request routes through the standardtextGenerationCostpath and noPROVIDERS_CACHE_COST_CALCULATORentry is needed.Part of the ongoing provider-coverage cleanup (#7697).
Testing
Added one
@Testper provider asserting a representative model prices to a non-zero, exact expected cost (it returns zero before this change).Documentation
No documentation changes needed