Skip to content

feat(usage): add pairwise cost comparisons - #925

Merged
wesm merged 1 commit into
kenn-io:mainfrom
rodboev:pr/usage-cost-comparison
Jul 2, 2026
Merged

feat(usage): add pairwise cost comparisons#925
wesm merged 1 commit into
kenn-io:mainfrom
rodboev:pr/usage-cost-comparison

Conversation

@rodboev

@rodboev rodboev commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

The usage dashboard has enough aggregate data to show which models and projects cost the most, but it does not let users compare two choices directly. Issue #361 asks for questions like model A versus model B cost per session or token profile, which currently requires manual filtering and side-by-side arithmetic outside AgentsView.

This adds a pairwise usage comparison endpoint and dashboard panel. The comparison logic lives in the service layer so the direct backend, HTTP backend, server route, and frontend all use one metric definition. Each side applies the same usage filters and the same model or project filter path as the existing summary endpoint, so timezone handling, deduplication, pricing, and eligibility stay aligned with the rest of the usage dashboard.

The UI is additive: existing summary cards, time series, attribution, top sessions, and cache efficiency remain in place. The new panel focuses on model and project comparisons first, with cost, token, and session-based metrics returned by the backend instead of recomputed in the browser.

Fixes #361

@roborev-ci

roborev-ci Bot commented Jun 29, 2026

Copy link
Copy Markdown

roborev: Combined Review (7e82b01)

No issues found.


Panel: ci_default_security | Synthesis: codex | Members: codex_default (codex/default, done, 8m15s), codex_security (codex/security, done, 1m53s) | Total: 10m8s

@wesm

wesm commented Jul 1, 2026

Copy link
Copy Markdown
Member

looking

@wesm
wesm force-pushed the pr/usage-cost-comparison branch from 7e82b01 to 66c45cd Compare July 1, 2026 17:32
@roborev-ci

roborev-ci Bot commented Jul 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (66c45cd)

Summary verdict: one medium correctness issue needs attention; no security regression was found.

Medium

  • frontend/src/lib/stores/usage.svelte.ts:771: A stale first-load pairwise request can leave loading.pairwise stuck true. New summary/full refreshes invalidate the summary version before the old pairwise request finishes, so its finally block skips clearing loading; if the replacement summary fails or returns early, no later pairwise request clears it.
    • Fix: Clear pairwise loading when invalidating/aborting pairwise work, or clear it when the finishing request is still the current pairwise request regardless of summary version. Add a test for summary failure while pairwise first load is in flight.

Panel: ci_default_security | Synthesis: codex, 7s | Members: codex_default (codex/default, done, 7m24s), codex_security (codex/security, done, 1m14s) | Total: 8m45s

@roborev-ci

roborev-ci Bot commented Jul 2, 2026

Copy link
Copy Markdown

roborev: Combined Review (0a1363f)

No issues found.


Panel: ci_default_security | Synthesis: codex | Members: codex_default (codex/default, done, 7m1s), codex_security (codex/security, done, 2m4s) | Total: 9m5s

@roborev-ci

roborev-ci Bot commented Jul 2, 2026

Copy link
Copy Markdown

roborev: Combined Review (518f1c7)

No issues found.


Panel: ci_default_security | Synthesis: codex | Members: codex_default (codex/default, done, 7m46s), codex_security (codex/security, done, 1m39s) | Total: 9m25s

@wesm
wesm force-pushed the pr/usage-cost-comparison branch from 518f1c7 to 2a09ab5 Compare July 2, 2026 17:56
@roborev-ci

roborev-ci Bot commented Jul 2, 2026

Copy link
Copy Markdown

roborev: Combined Review (2a09ab5)

Summary: One medium issue should be fixed before merge.

Medium

  • internal/service/http.go:486: UsagePairwiseComparison drops req.GitBranch when proxying to /api/v1/usage/pairwise-comparison, unlike the other usage endpoints. Branch-scoped comparisons through the HTTP backend or MCP daemon will silently compare all branches.
    • Fix: Add "git_branch": req.GitBranch to the query serialization and cover it in the HTTP backend serialization test; regenerate/update the frontend client so the pairwise method also exposes gitBranch.

Panel: ci_default_security | Synthesis: codex, 9s | Members: codex_default (codex/default, done, 5m53s), codex_security (codex/security, done, 1m59s) | Total: 8m1s

@wesm

wesm commented Jul 2, 2026

Copy link
Copy Markdown
Member

looking

@roborev-ci

roborev-ci Bot commented Jul 2, 2026

Copy link
Copy Markdown

roborev: Combined Review (f0daf29)

Pairwise usage comparison is close, but one medium stale-data bug should be fixed before merge.

Medium

  • frontend/src/lib/stores/usage.svelte.ts:561 - Pairwise comparison data is only cleared when the selected slices change. If the date or session filters change while the same model/project values remain valid, the old pairwise result stays visible. If the new pairwise request then fails, fetchPairwise suppresses the error because pairwiseComparison is non-null, leaving stale metrics displayed for the new filter context.

    Fix: Clear or mark pairwise data stale whenever a new summary/filter version is accepted before starting the pairwise fetch, or track the params used for the current comparison and show an error/empty state when the current request fails.


Panel: ci_default_security | Synthesis: codex, 8s | Members: codex_default (codex/default, done, 8m22s), codex_security (codex/security, done, 2m0s) | Total: 10m30s

- test(usage): align pairwise proof with filter semantics
- fix(usage): match pairwise clients to the wire contract (kenn-io#361)
- fix(usage): drop stale pairwise results during selector refresh (kenn-io#361)
- fix(usage): keep pairwise state and schema aligned (kenn-io#361)
- fix(usage): clear summary-driven pairwise drift (kenn-io#361)
- test(usage): cover MCP pairwise daemon forwarding (kenn-io#361)
- fix(usage): keep generated client drift out of the PR (kenn-io#361)
- fix(usage): narrow generated client changes to pairwise scope (kenn-io#361)
- fix(usage): use shared selectors and clear aborted pairwise loading (kenn-io#361)
- Keep pairwise comparison CI aligned with current source and lint
- fix(usage): keep pairwise selectors reactive and non-blocking (kenn-io#361)
- fix(i18n): add zh-TW pairwise comparison copy
- docs: require synchronized localization catalogs
- fix(usage): lower pairwise comparison priority
- fix(usage): forward branch filters for pairwise comparisons
- fix(serve): reuse daemons published during startup
- fix(usage): clear stale pairwise data on refresh
@wesm
wesm force-pushed the pr/usage-cost-comparison branch from 3487d4d to 7b9f34f Compare July 2, 2026 19:51
@roborev-ci

roborev-ci Bot commented Jul 2, 2026

Copy link
Copy Markdown

roborev: Combined Review (7b9f34f)

No issues found.


Reviewers: 2 done | Synthesis: codex | Total: 13m57s

@wesm
wesm merged commit 80e1c0e into kenn-io:main Jul 2, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Feature: comparative cost analysis

2 participants