Choose admitted venue runner and comparison profiles

[ericjuta]

Part of #65.

Spot, Perps and Predictions MCP evaluations

Current checkpoint, 2026-09-11

Canonical implementation and admission checkpoint. PR #79 and PR #80 are merged. Current source uses stock HarnessAgent/adapters with local process execution and explicit native OMP profile authentication alongside API-key mode. This supersedes the earlier missing-implementation, patched-adapter, Docker, descriptor-relay and custom auth-custody blockers. Historical comments remain unchanged.

Source landing and account-free synthetic proof are not runner or venue admission. Suitable real authentication/profile access, independent Gina MCP authorization, applicable approved runtime identity, routing/reasoning and effective-setting evidence remain required. Dedicated venue execution evidence remains incomplete. Concrete campaign bounds, UTC expiry, run approval and separate exact-content publication approval also remain open. The upstream ACP typed provider-error limitation remains tracked in oh-my-pi #11644.

Status remains NOT_READY_FOR_APPROVAL. M1-M16, all four profiles, all 34 candidate goals, controlled Sol/medium and Spot-first sequencing remain settled. #73 stays open and #71 stays blocked. No live, paid, venue, measured or publication acceptance is claimed.

Question

Which existing runner profiles may evaluate each dedicated MCP, and what additional admission evidence is required before their scores can be compared?

Human decisions

• Distinguish controlled model tool loops, native agents and installed-plugin activation evidence. Existing Gina Read runner support is not automatic venue MCP support.
• Set venue-specific tool visibility, discovery/schema access, cross-venue denial and competing-tool conditions. Do not confuse permitted MCP bash with permission to use a local shell or unrestricted networking.
• Require exact effective connection/catalog/runtime/configuration identity, compatible observation semantics and disclosed provider retry/step/deadline behavior. Define unavailable/not-admitted profiles without fabricating passes.
• Extend rather than duplicate the established fixed-versus-tuned comparison policy. Exact model/reasoning/budget identity and credential custody remain on their existing decisions.

The public evidence inventory and linked safety, authentication and identity decisions are resolved. Their existing dependency relationships remain; this venue-admission decision is still open. No runner changes, native diagnostics or model calls occur while resolving it.

Confirmed identity handoff

Reuse the owner-confirmed identity and evidence policy without reopening its model, reasoning, budget, package or omission choices. Its explicit admission deferrals require:

• Return the exact compatible existing HarnessAgent/Codex runtime versions and executable hashes for owner approval before any credential-bearing evaluator run, including admission smoke. Native Codex remains required but not yet admitted.
• Return the exact per-profile OpenRouter endpoint slugs and verified reasoning translation for the same pre-execution approval. One exact upstream endpoint is required, with fallbacks disabled and parameter support required; missing control or endpoint mismatch blocks admission.
• Bind the exact non-secret serialized request settings and approved intentional omissions. Unknown effective provider defaults remain unavailable, not inferred. The labels-only allowance for missing immutable model binding does not waive required controls.

This handoff admits no runner and authorizes no implementation, credential access, model/MCP execution or renewed native diagnostics. Measured acceptance remains a separate decision.

[ericjuta]

Owner-confirmed venue runner policy

The owner confirmed the consolidated policy and the proposal to set numeric action caps against the concrete case matrix. This records the policy agreement, not completion of runtime admission. This ticket remains open for the outstanding configuration evidence and owner approvals below.

Approved profile and tool conditions

• Target the three controlled model profiles and native Codex specified in the identity and evidence policy for each of Spot, Perps and Predictions. Admit each runner, venue, permission/capability and evidence-mode combination separately. Success on one combination does not admit another. Native coverage remains required; Claude/Pi remain staged. No additional runner is implicitly admitted.
• Expose the target venue's normal admitted catalogue and discovery/schema access for its frozen permission profile, consistently across equivalent trials. Keep legitimate distractors; do not tailor the catalogue, schemas or instructions to reveal the expected answer. Reduced-catalogue experiments are separate conditions.
• The restricted baseline has one dedicated venue MCP and only the minimum permitted native plugin/skill-loading mechanisms. No competing web search or MCP connections, unrestricted local shell, direct provider networking or cross-venue actions, including inside venue MCP bash. Permission to use venue MCP bash does not authorize the host shell. Competing-tool studies require separate profiles rather than pooled scores.

First-live task rejection policy

During the first live campaign, after a definitively rejected task order with trusted evidence of no effect, end further task execution in that case. The agent may not correct and resubmit the order. Retain the failure evidence and apply the already-approved grading rules. Exercise repair/retry behavior in non-live cases initially.

This does not change transport-retry accounting, already-authorized risk-reducing cleanup or the existing campaign-stop conditions. An ordinary definite rejection need not end the campaign, but required case-owned cleanup still precedes the next case. Uncertain mutations retain the established reconciliation, stop and human-handoff rules. Cleanup is not permission to resume the rejected task.

Engineering obligations derived from existing decisions

No profile is admitted until the applicable version-bound enforcement, pre-effect denial, lifecycle/cleanup and required observation obligations are proven.

The owner asked that inferable engineering choices be handled without further approval questions. Version-bound behavioral proof at the actual enforcement boundaries, pre-effect denial, deadline/cancellation/cleanup proof and required observation completeness follow the locked authentication, safety, identity and grading decisions. Configuration review alone is not runtime proof. These are derived implementation requirements, not evidence that they have been implemented or passed.

Approved sequencing and outstanding gates

Set numeric per-case and campaign mutation-attempt caps with the concrete case matrix and measured acceptance proposals. No numeric cap, rejected-attempt/retry allowance or cleanup allocation was selected by this confirmation. Those values still require owner approval before live execution; the existing financial, time and confirmation limits remain unchanged. Outer tool-call budgets do not establish inner mutation bounds.

For the applicable profile, before any credential-bearing evaluator run, including admission smoke, return for owner approval:

• Exact compatible existing HarnessAgent/Codex runtime versions and executable hashes.
• Exact per-profile OpenRouter endpoint slugs and verified reasoning-translation evidence, with fallbacks disabled and parameter support required as already agreed.

Separately, each campaign requires its concrete cases, evidence modes, run/trial counts, bounded retry policy, cost ceilings, UTC expiry and explicit run approval. Financial execution additionally requires the approved dedicated accounts, instrument/chain/provider allowlists, price and exposure controls, mutation-attempt caps, rejected-attempt accounting and cleanup allocation. Assign fixture or recorded-response evidence to each capability excluded from live execution. These campaign proposals belong with the case matrix and still need owner approval; a separately authorized read-only admission smoke does not require a financial campaign profile. Neither this policy nor provisional case-matrix discussion waives the applicable admission or execution prerequisites.

Keep this runner-admission ticket open and keep the measured-acceptance ticket's dependency on it intact until the outstanding admission work is satisfied. Case-matrix discussion may proceed provisionally; measured acceptance is not resolved or unblocked by this comment.

No runner was admitted, no runtime or route identity was approved, and no implementation, credential access, renewed native diagnostics, model/MCP call, transaction, deployment or measured-results publication was authorized or performed by this policy-recording step.

[ericjuta]

Published admission research checkpoint

The runtime assessment records installed/public package facts, credential-free offline proof and its limits, the missing requested HarnessAgent/Codex integration, and unproven controlled endpoint enforcement/native reasoning translation. Existing authentication, isolation, identity, and venue policies remain unchanged.

Status remains NOT_READY_FOR_APPROVAL and this ticket remains open. Exact compatible native pins, routing/settings evidence, and applicable owner approval are still required before credential-bearing smoke. No native profile was dropped or replaced; no execution was authorized.

[ericjuta]

Source and admission checkpoint, 2026-09-10

Status remains NOT_READY_FOR_APPROVAL. This records completed engineering and its proof, not a new policy decision, runner admission, campaign approval, or measured result.

Public evaluator source

Published branch feat/eval-multi-runner is at 99f6fd10cae2ac18e551d94d4480db0872bc604c, not merged into main at this checkpoint.

• Durable trial starts precede dispatch. Report binding requires every planned trial to finish and binds the exact report bytes and run identity.
• The runner retains provider-label evidence and checks required labels before tool dispatch. Provider labels do not prove an exact endpoint, region, or effective reasoning setting.
• OpenRouter current-key admission checks finite non-resetting provider limits. Direct requested-cap comparisons now tolerate only machine-precision roundoff, preserve the original numeric evidence, and reject a nearby genuine excess. This is not a mathematical end-to-end spending guarantee against in-flight usage, other key users, or provider overshoot.
• Configuration-v2 and failure-retained journals accompany unchanged v1 report contracts. Explicit alpha Gina-read resource selection is supported; it is not dedicated venue execution support.

Proof on that revision: 324 tests across 30 files, nonincremental typecheck, quality and artifact verification passed. The public-boundary scan covered 510 files and 12 archives with zero findings. CI and CodeQL passed. Built-package mechanism checks used synthetic inputs/transports, not paid model or measured venue execution.

Remaining gates

• Exact compatible existing HarnessAgent/Codex pins and the required configuration, tool/network isolation, native-auth custody, and exclusive shared-store lifecycle still need admission. No replacement runner or weaker isolation is substituted.
• Exact controlled OpenRouter endpoint identity and native reasoning translation still need independently observed evidence and the applicable owner approval. Requested routing and synthetic provider-label checks do not close this gate.
• Dedicated venue execution evidence remains incomplete. This source checkpoint does not complete venue admission.
• Concrete accounts, quantities, funding, executable inputs, run counts, aggregate caps, UTC expiry, and campaign approval remain with the case matrix and measured acceptance gate. Exact-content measured-results publication remains a separate approval.

Keep all locked decisions and required coverage, including M1-M16, the four profiles, all 34 candidate goals, and controlled Sol/medium with Spot first. No new routine preference questionnaire is introduced. The retained OMP MCP-error limitation is unchanged; its completed investigation was not reopened.

This admission ticket stays open. The case-matrix/acceptance ticket stays open and blocked by it. No dependency or decision is resolved by this source checkpoint, and no merge, deployment, live evaluation, trade, or measured-publication approval follows from this tracker update.

[ericjuta]

Source preparation checkpoint, 2026-09-10

Status remains NOT_READY_FOR_APPROVAL. Public evaluator branch feat/eval-multi-runner is published at 0f554b80ca271756cfedc3a13821fb4afbfabf7c, not merged into main.

The new commit corrects the output-collision regression fixture to use the emitted configuration-v2 companion. It does not change runner behavior. Source review of the preceding implementation found no introduced landing defect. On the published head, 324 tests across 30 files, quality checks, artifact verification, package-install smoke, eval-app typecheck, and all 230 target-conformance checks passed. The public-boundary scan covered 510 files and 12 archives with zero findings. CI and CodeQL passed on this exact revision.

This is source and offline proof, not a measured evaluation. The previous source checkpoint still describes the implementation and evidence limits.

Remaining gates are unchanged:

• Compatible native HarnessAgent/Codex versions, required isolation, native-auth custody, and exclusive saved-login lifecycle still need admission. No replacement runner or weaker isolation is substituted.
• Exact OpenRouter endpoint identity and native reasoning translation still need independently observed evidence and applicable owner approval before credential-bearing evaluation.
• Dedicated venue execution evidence remains incomplete.
• Concrete campaign inputs, counts, costs, financial bounds, UTC expiry, and run approval remain unfrozen. Exact-content measured-results publication is a separate approval.

All four profiles, all 34 candidate goals, and controlled Sol/medium with Spot first remain required. #73 stays open; #71 stays open and blocked by #73. No locked decision, dependency, merge, deployment, live evaluation, trade, or measured-publication approval follows from this update.

[ericjuta]

Owner-authorized Codex adapter patch, 2026-09-10

The owner clarified that execution must remain through the existing AI SDK HarnessAgent, then explicitly authorized patching its Codex adapter. The path remains evaluator → HarnessAgent → Codex adapter → official Codex. This is not permission to replace HarnessAgent with a custom CLI runner.

Source implementation and offline verification of the necessary adapter/dependency patch are authorized. Preserve SDK-owned configuration, personal-configuration exclusion, restricted native tools/sandbox, native saved-login custody without copies, and exclusive shared-store lifecycle. The implementation must fail closed wherever those controls cannot be enforced; an API surface or patched capability flag is not runtime proof.

This settles the adapter-patch-versus-wait decision. It does not admit a runtime or authorize credential access, live model/MCP calls, paid evaluation, financial execution, PR-triggered preview deployment, merging into main, or publication of measured results. #73 remains open and NOT_READY_FOR_APPROVAL; #71 remains blocked. The four profiles, 34 goals and controlled Sol/medium with Spot first are unchanged.

[ericjuta]

Restricted Codex adapter source checkpoint, 2026-09-10

Published 77f2a75 on feat/eval-multi-runner; not merged into main. The implementation remains evaluator → HarnessAgent → patched official Codex adapter → official Codex. This follows the adapter-patch authorization.

The package bundles the paired HarnessAgent/adapter changes, restrictive exec configuration, Unix-socket relay, explicit host auth-store lease handling, and cleanup/unsupported-lifecycle rejection. Clean installed consumers receive the patched code, types, and bootstrap assets. Artifact verification independently rebuilds the committed source and compares exact bytes, including admitted dependency source maps.

Source/offline proof: 359 tests across 32 files; 230 target-conformance checks; quality/type checks; dependency audit; clean artifact/package-install verification; public-boundary scan of 519 files and 12 archives with zero findings. The installed package also passed 26 adapter lifecycle regressions. A separate synthetic SDK argument-capture and Unix-relay smoke passed; its IPv4/IPv6 denial proof is not proof of the native sandbox. CI and CodeQL passed on this exact head.

Admission remains NOT_READY_FOR_APPROVAL. A trusted filtered saved-auth view, all-process store exclusivity/refresh custody, and actual native sandbox enforcement remain unproven. No native credential-bearing CLI runner is enabled. Exact routing/reasoning evidence, dedicated venue execution evidence, and campaign approval also remain outstanding. This source checkpoint is not approval for live/paid evaluation, financial execution, deployment, or measured-results publication.

All four profiles, all 34 goals, controlled Sol/medium and Spot-first sequencing remain unchanged. #73 stays open; #71 remains blocked. No ticket closure or dependency change.

[ericjuta]

Source merged, admission unchanged

Under the user's explicit merge authorization, PR #77 merged into main as d0bff12 on 2026-09-10. It preserves the HarnessAgent → official Codex adapter path and current-main candle/prediction and Devin packaging changes.

The integrated head passed 359 tests across 32 files, 231 target checks, quality/type checks, independent artifact/package-install verification, and a public-boundary scan of 520 files and 12 archives with zero findings. The merged main tree exactly matches that verified tree. Main CI and main CodeQL passed on the merge commit.

This is a source landing, not a live/native or venue evaluation. Native credential-bearing execution remains disabled pending independent filtered auth-store, all-process exclusivity/refresh custody and actual sandbox proof. Routing/reasoning evidence, dedicated venue evidence and campaign approval remain outstanding. NOT_READY_FOR_APPROVAL remains; #73 stays open and #71 remains blocked. No ticket closure or dependency change. All four profiles, 34 goals, Sol/medium and Spot-first sequencing remain settled.