Add guarded web model evals#13
Conversation
|
Warning Review limit reached
More reviews will be available in 8 minutes and 32 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 987c7e82eb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if s.cfgRef == nil { | ||
| return failedModelEval("missing runtime config") | ||
| } | ||
| suite, err := evalpack.LoadSuite(modelEvalDatasetPath) |
There was a problem hiding this comment.
In the production image I checked, the final Docker stage copies only /app/bin/agent and config/config.yaml into /app, but this new handler reads evals/workload.json from the process working directory. In that container every /models/eval request will fail to load the suite instead of actually running the eval, so the feature is broken for the default deployment unless the suite is embedded or copied into the runtime image.
Useful? React with 👍 / 👎.
| {{if or .Free (eq $root.CatalogProvider "ollama") (eq $root.CatalogProvider "local")}} | ||
| <input type="hidden" name="confirm_paid_eval" value="1"> |
There was a problem hiding this comment.
For OpenRouter rows whose catalog price is zero or missing but whose ID is not a :free variant, .Free is true because Capabilities.Free treats zero price as free/unknown, so this branch silently posts confirm_paid_eval=1. handleModelEval still classifies that model as paid via isPaidModelEval, but it accepts the hidden confirmation, allowing a paid/cloud eval to run without the explicit checkbox warning.
Useful? React with 👍 / 👎.
1 participant
Summary
Tests