[Eval] Query batch of test suite runs costs

[buhaiovos]

Page-scoped total run cost from dial-adas (one query for many runs)

Context

GET /api/v1/test-suite-runs/{id}/costs today issues two sequential dial-adas calls per run (one per
eval.phase) and returns two per-call averages (avgTestCaseCost, avgMetricEvalCost). The upcoming
enriched run listing (see the follow-up issue) needs the total cost of many runs at once, and it must
fetch prices exactly once per page — never once per run.

Run cost lives only in dial-adas. Usage-log rows correlate to a run through a single flat baggage string:

eval.phase=execution,eval.run.id=<runId>,eval.suite.id=<suiteId>,run.index=0,testcase.id=<testCaseId>

so a per-run breakdown requires dial-adas to derive the run id inside the query. Whether it can is
unverified: the archived design (openspec/changes/archive/2026-08-20-test-suite-run-costs/design.md,
Risks) explicitly flagged grouping over a json_extract_string expression as unconfirmed, and only the
and / co / count / avg aggregate shape has been validated against a real deployment.

Because testcase.id is part of the baggage, grouping by the raw baggage string is not an option — its
cardinality is one value per test case per phase.

Scope

Phase 1 — spike (blocking, do first)

Probe a real dial-adas with both shapes and record which one it accepts, the exact function names it
exposes, and whether group_by resolves a select alias.

Shape A — group by an extracted run id (preferred; query size independent of page size):

{ "entity": "dial_usage_log", "mode": "aggregate",
  "filter": {"op":"or","args":[
    {"op":"co","args":[ {"type":"fn","name":"json_extract_string",
        "args":[{"type":"field","name":"request_tags"},{"type":"value","value":"baggage"}]},
      {"type":"value","value":"eval.run.id=<r1>"} ]}
    /* …one per run id on the page… */ ]},
  "select": [
    {"expr": {"type":"fn","name":"regexp_extract","args":[
        /* the json_extract_string expression above */,
        {"type":"value","value":"eval\\.run\\.id=([0-9a-f-]{36})"} ]}, "as": "run_id"},
    {"expr": {"type":"fn","name":"sum","args":[{"type":"field","name":"total_price"}]}, "as": "total_cost"} ],
  "group_by": ["run_id"] }

Shape B — one conditional sum per run, group_by: [] (no grouping support needed, but the query
grows with page size):

select: [ sum_if(total_price, co(baggage, 'eval.run.id=<r1>')) as c_<r1>, … ]

Outcome gate: if neither shape works, close this issue with the finding documented and the enriched
listing ships without a cost field. Do not fall back to one call per run.

Phase 2 — implementation (only if the spike succeeds)

• RunCostQueryBuilder.buildPageTotalCostQuery(Collection<UUID> runIds) — reuse the existing private
  jsonExtractBaggage / baggageContains / stringValue helpers. No eval.phase predicate, so the
  total spans both phases and any phase added later. buildAggregateQuery(runId, phase) stays untouched.
• client/dialadas/dto/AdasAggregateRowDto — add @JsonProperty("run_id") String runId and
  @JsonProperty("total_cost") Double totalCost, both nullable. Unknown-property tolerance keeps the
  existing avg path unaffected.
• New service/domain/RunCostFetcher:
  • Map<UUID, RunCost> fetchTotalCosts(Collection<UUID> runIds) — one
    dialAdasClient.executeAggregate(...) call, rows keyed back to run ids.
  • record RunCost(Double value, RunCostStatus status) with
    RunCostStatus { AVAILABLE, NO_DATA, UNAVAILABLE } — a bare null cannot distinguish "no usage rows
    recorded" from "adas did not answer".
  • A run absent from the response → NO_DATA. catch (DialAdasClientException e) → log with the
    exception as the trailing SLF4J argument (LoggingConventionTest) → all runs on the page
    UNAVAILABLE. Never rethrow — a listing must not fail because adas is down.
  • No transaction; must never be invoked with a meta or analytics transaction open.
• Config (defaults in application.yml only; properties class holds structure + validation):
  test-suite-run.enriched-list.cost.enabled / TEST_SUITE_RUN_ENRICHED_COST_ENABLED, default true.
  Add the row to docs/configuration.md with all six columns and amend §5.5, which currently says
  dial-adas is queried only by /costs.

Out of scope

• Any change to GET /api/v1/test-suite-runs/{id}/costs — it keeps its two averages and its 502/504
  failure contract.
• Per-test-case or per-phase cost breakdowns.
• Caching of adas responses.

Acceptance criteria

• Spike outcome recorded in the OpenSpec change's design.md: which shape adas accepts, the exact
  function names, and whether group_by resolves a select alias.
• Costs for N runs are fetched in exactly one HTTP call, asserted by a test
  (verify(dialAdasClient, times(1)).executeAggregate(any())).
• Aggregation stays server-side (no row-mode fetch summed in application code).
• A run with no usage rows yields NO_DATA with a null value — never 0.
• An adas timeout or connection failure yields UNAVAILABLE for the page and no exception escapes
  RunCostFetcher.
• Existing RunCostQueryBuilderTest avg assertions still pass unchanged.

Manual: point DIAL_ADAS_URL at a real deployment and run the two spike payloads with curl against
POST {base}/v1/queries/execute before writing any code.