Add guarded web model evals

[dzarlax]

Summary

• add POST /models/eval for manual workload evals from the web catalog
• require server-side confirmation for paid/cloud model evals
• keep one eval running at a time per process
• persist and display the latest eval status per model
• add web rendering and paid guard tests

Tests

• go test -count=1 ./internal/adminapi
• go test -count=1 ./...

[coderabbitai]

Warning

Review limit reached

@dzarlax, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 8 minutes and 32 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 8272ac7c-3507-44a3-8ea4-150635fa6355

📥 Commits

Reviewing files that changed from the base of the PR and between 25baf0b and 987c7e8.

📒 Files selected for processing (5)

• internal/adminapi/adminapi_test.go
• internal/adminapi/handlers.go
• internal/adminapi/model_evals.go
• internal/adminapi/server.go
• internal/adminapi/templates/partials_models_row.html

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/web-model-evals

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 987c7e82eb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Package the eval dataset for runtime loads

In the production image I checked, the final Docker stage copies only /app/bin/agent and config/config.yaml into /app, but this new handler reads evals/workload.json from the process working directory. In that container every /models/eval request will fail to load the suite instead of actually running the eval, so the feature is broken for the default deployment unless the suite is embedded or copied into the runtime image.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Require explicit confirmation for zero-priced OpenRouter rows

For OpenRouter rows whose catalog price is zero or missing but whose ID is not a :free variant, .Free is true because Capabilities.Free treats zero price as free/unknown, so this branch silently posts confirm_paid_eval=1. handleModelEval still classifies that model as paid via isPaidModelEval, but it accepts the hidden confirmation, allowing a paid/cloud eval to run without the explicit checkbox warning.

Useful? React with 👍 / 👎.