Improve model eval timing and ARM64 builds

[dzarlax]

Summary

• Relax model evaluation timing so each case gets its own timeout window and stale runs are tracked over a longer interval.
• Add repository AGENTS.md instructions for future Codex work.
• Move the ARM64 Docker workflow to GitHub's native ubuntu-24.04-arm runner and remove QEMU setup.

Tests

• go test ./internal/adminapi/...

Summary by CodeRabbit

• Documentation

  • Added comprehensive system documentation covering architecture, configuration management, LLM routing, and evaluation framework details.

• Improvements

  • Enhanced model evaluation timeout behavior and staleness detection for improved reliability and consistency.

• Chores

  • Updated CI/CD workflow to support ARM architecture deployments.

[coderabbitai]

Warning

Review limit reached

@dzarlax, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 27 minutes and 26 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 3adbc71a-1dbc-4e09-b3ff-be82120bbf44

📥 Commits

Reviewing files that changed from the base of the PR and between 28b1546 and b1f8f5e.

📒 Files selected for processing (2)

• AGENTS.md
• internal/adminapi/model_evals.go

📝 Walkthrough

Walkthrough

Three unrelated maintenance changes: internal/adminapi/model_evals.go restructures eval timeouts from a single shared timeout to a per-case 90-second constant and switches from a request-scoped context to context.Background; the ARM64 CI workflow migrates from ubuntu-latest with QEMU emulation to the native ubuntu-24.04-arm runner; and AGENTS.md is created with comprehensive repository guidance for AI coding agents.

Changes

Model Eval Timeout Refactor

Layer / File(s)	Summary
Eval timeout constants, context, and suite runner internal/adminapi/model_evals.go	Removes modelEvalTimeout, introduces modelEvalCaseTimeout (90s), sets modelEvalStaleAfter to 30 minutes as a direct constant, switches runModelEval call to context.Background(), and updates evalpack.RunSuite to use modelEvalCaseTimeout instead of the hardcoded 12-second value.

Repo Infrastructure and Documentation

Layer / File(s)	Summary
ARM64 CI native runner migration .github/workflows/docker-arm64.yml	Changes the docker-arm64 job runner from ubuntu-latest to ubuntu-24.04-arm and removes the QEMU setup step that is no longer needed with a native ARM runner.
AGENTS.md repository guidance AGENTS.md	Adds 363-line document covering commands, request-flow architecture, per-package responsibilities, LLM routing priority, semantic memory, tool filtering, SQLite schema, usage/cost tracking, admin UI/API endpoints, KV-backed persistence, OpenRouter model hydration, multimodal types, and MCP server references.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐇 No more QEMU pretending to be ARM,
A native runner keeps the builds warm.
Eval timeouts stretched to ninety whole seconds,
While Background context no longer beckons.
And AGENTS.md tells the bots what to do—
Hippity hoppity, the repo's brand new! 🥕

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly addresses two of the three main changes: improving model eval timing and optimizing ARM64 builds in CI/CD.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/eval-timing-behavior

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 28b1546032

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Restore a bounded request context for model evals

When the requester disconnects or a proxy gives up on this synchronous htmx request, the eval now continues under an unbounded context.Background(). With the current 13-case workload and the new 90s per-case timeout, this can keep modelEvalMu held and continue issuing cloud/paid model calls for roughly 20 minutes after the request is gone; the previous suite deadline stopped new cases after about 45s. Please derive the eval context from r.Context() with an overall deadline, or run evals as an explicit background job with cancellation/status handling.

Useful? React with 👍 / 👎.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@AGENTS.md`:
- Line 354: In the AGENTS.md file, locate the section documenting the Codex
Bridge MCP sync behavior near the MCP_BRIDGE_EXPORT_PATH environment variable
description. The text currently refers to "Codex Desktop format" but should be
updated to use "Claude Desktop format" to align with the correct terminology
used in coding guidelines. Simply replace "Codex Desktop format" with "Claude
Desktop format" in the documentation.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: d20ec653-a75f-4fb3-b63d-7290a6331aa5

📥 Commits

Reviewing files that changed from the base of the PR and between 09679ad and 28b1546.

📒 Files selected for processing (3)

• .github/workflows/docker-arm64.yml
• AGENTS.md
• internal/adminapi/model_evals.go