feat(evals): add multi-provider and native-agent runners

[ericjuta]

Summary

Add multi-provider and native-agent execution adapters behind the existing evaluator. This branch contains 12 commits and changes 28 files relative to main at ff235a40e9063a3cc6d9846ad7e7c1f3f01a76a6.

• Add an AI SDK OpenRouter runner, an isolated API-key Claude CLI runner, and a pinned OMP 18.1.14 runner through official HarnessAgent/createACP with a local Docker sandbox.
• Preserve separate Responses, Codex, OpenRouter, Claude and OMP observation targets. Keep the shared suites, grading, replay validation, exact-report-bound attempt capture and manual publication approval.
• Add runner-specific credential preflight and CLI options. Saved-login reuse remains deferred; Gina MCP authorization is separate from model credentials.
• Preserve canonical MCP tool names while adapting native wire names, require exact OMP startup inventory and guard evidence, and restrict native actions to staged skill reads and canonical MCP tools.
• Support provider-qualified model identities through report/export contracts and browser decoding. Update dependencies, artifact admission and runner documentation.

Related: multi-provider evaluation map and implementation tracker. Deferred authentication and comparison-identity decisions remain open; this PR does not close the map.

Known unresolved failure

Track intermittent OMP MCP-error observation loss contains the public-safe retained evidence summary and receipt fingerprints.

One synthetic MCP-error run failed with generation-failed despite native terminal completion and one host MCP call. It failed closed, but did not retain the expected canonical failed tool observation. The bounded investigation already included exception instrumentation, four diagnostic passes and one final unchanged-source pass. The cause remains unknown; later passes are not a fix. No investigation or native scenario was rerun to open this PR.

Retry and isolation limits

OMP retry.enabled: false and retry.modelFallback: false disable agent-level TurnRecovery and configured fallback. OMP 18.1.14 provider clients still have hard-coded HTTP retries: the retained HTTP500 probes observed 11 Anthropic requests and 12 OpenAI/OpenRouter requests. The original one-request assertions remain failed. There is no universal one-request guarantee and no runtime fork in this PR.

The Docker provider is network-configurable; its default owned network is not an internal-only network. The contained synthetic proof explicitly supplied an Internal:true network from container creation and used offline bootstrap caches prepared separately. Do not confuse that proof configuration with the default production network policy. The Gina bearer stays on the host; the provider credential is forwarded into the isolated native runtime, not brokered outside it.

Existing verification

Head: 4fa4c6f2578268c60b60c454a3fc1a495dc0b64a, GitHub signature verified.

• CI passed on this head.
• CodeQL passed on this head.
• Latest local source verification: non-incremental typecheck covering 984 files, changed-runtime lint with zero diagnostics, and 29 focused tests: runner.test.ts 20 and live-cli.test.ts 9.
• A prior source-bound CLI-help case timed out once; the isolated case and settled focused batch passed without raising its timeout. The original failure remains recorded.
• Retained native proof: query/path calls keep canonical SQL identity; intent tracing is absent from the final native tool schema; empty input makes zero MCP calls and retains a failed observation; missing guard makes zero provider requests; the final MCP-error smoke retains a canonical failed observation. Query/path baselines also passed, so no attribution regression was causally reproduced.

These are source, CI and synthetic protocol results. They are not measured model quality, production Gina connectivity, or evidence that a model independently discovered the plugin. Private raw captures and host paths are not published here.

Separately approved next milestone

Paid evaluations are not authorized or executed by this PR. Before the planned four cases × three repetitions per selected runner, separately approve the runner/model selection, reasoning and cost limits, and provide the required provider credentials and Gina MCP read authorization privately. Retain failures, aggregate reports and exact-report-bound attempts. Public results still require explicit review of their exact content; neither this PR nor a successful run grants publication approval.

No merge, deployment or release is part of this PR-opening step.