Skip to content

Fix native OMP eval catalog compatibility and tool evidence - #85

Merged
ericjuta merged 5 commits into
askgina:mainfrom
0xashrk:fix/omp-connected-catalog
Sep 11, 2026
Merged

Fix native OMP eval catalog compatibility and tool evidence#85
ericjuta merged 5 commits into
askgina:mainfrom
0xashrk:fix/omp-connected-catalog

Conversation

@0xashrk

@0xashrk 0xashrk commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Correct native OMP live-eval compatibility and accounting without changing the suite, prompts or grader. No new OAuth Responses shim is introduced; the existing native OAuth path is used.

  • Accept the exact canonical 31-tool research catalog or exact connected 32-tool catalog, still rejecting missing, unknown or duplicate names. Expose/record only the 31 research tools; exclude gina.renderReadOnlyDashboard.
  • Permit legitimate JSON Schema union types in AJV while retaining strict validation and no evaluator-side argument coercion.
  • Explicitly launch OMP with --tools read; the ACP builtin declaration alone does not enforce runtime tool filtering. Supply a required string path read schema rather than depending on a fragile title.
  • Keep native read activation/failure evidence separate from canonical research routing. Reads are not confined to staged skills or a filesystem sandbox.
  • Correlate native MCP mirrors to captured host executions one-to-one using trusted server/tool/correlation metadata, equal arguments and matching success/error outcome. Retain every real execution, surplus/mismatched mirrors and unknown native calls; do not blanket-drop acp_* calls.

Verification

  • Six focused regression tests: five fail against the previous source; all six pass with the fix. Coverage includes real repeated executions, surplus mirrors, argument mismatch, unknown native evidence and failed native reads.
  • Evaluation package: 236 tests passed. Root typecheck and build passed.
  • Actual OAuth metadata preflight verified false duplicate routing before the fix and one canonical execution afterward.
  • Two real production-MCP spot runs, four cases × three repetitions, clean chats, medium reasoning, native OpenAI OAuth (no OpenAI API key), separate Gina OAuth tools:read grant. Same OMP 18.1.17 executable, harness-acp 1.0.40, suite and grader.
Recorded metric GPT-5.5 / medium GPT-5.6 Sol / medium
Implemented-grader overall pass 11/12 11/12
Routing 11/12 11/12
Arguments / completion 12/12 each 12/12 each
P50 latency 26.633s 27.949s
P95 / max latency 54.557s 55.121s
Reported total tokens 185,084 335,030
Within declared 30s target (diagnostic) 8/12 6/12
Max reported result bytes 76,565 76,565

Both models failed one AAVE metadata trial for two actual spot.getTokenMetadata executions: GPT-5.5 repetition 3 and Sol repetition 1. Neither had a recorded tool-call error. Those failures were retained, not replaced; the stored evidence does not explain the extra invocation.

Grading limits — not broader quality or safety claims

  • Only the three simple-price trials assert specific argument values (ids=ethereum, vs_currencies=usd); the other nine argument scores automatically pass for lack of a case-specific constraint. Host schema validation is separate, and OMP may coerce arguments before forwarding.
  • Completion checks completed status and recorded tool errors, not answer semantics. Final-answer grounding, chart presentation and correctness are not graded.
  • The YAML's 30s / 100,000-byte budgets are not pass/fail gates.
  • Safety and skill activation are N/A/ungraded, not passes. Observed forbidden tools/scopes still fail safety, but missing requested_scope evidence can omit its score.
  • Native reads are allowed; this is not filesystem confinement, Codex CLI plugin-install verification or an MCP App rendering test.
  • This small live sample is not a general model ranking or monetary-cost comparison. Usage totals exclude setup/debug work. Full answers and raw tool argument/result bodies were not retained.

Reproducibility / evidence

  • GPT-5.5 run: spot-openai-oauth-20260911T135127Z; aggregate SHA-256 d12167198c14054d5b0599c9fd7fc3758a2d9c1aaa784009ae7352fb91fdfb0f.
  • Sol run: spot-openai-oauth-sol-20260911T142646Z; aggregate SHA-256 4f4d0f5a544905aa200abebdfec431bc93daba21ce95d3710f2d3a82d6a64a71.
  • Tested source: 5f98d54cf0f6514c90116a4c7c1d889cdd7dc485; Sol tip 780dc809e9956a329d9e60c72503de449477699b changes only one README sentence.
  • A three-page comparison PDF and checksummed archive were delivered separately to the requester: both original reports, all 24 valid attempts, journals, normalized tool-name/status traces, exact suite/grading snapshots and source patch. They are not committed or publicly attached to this PR. Temporary credential profiles/files and diagnostic wrapper were removed; no credentials are included.

These changes repair runner evidence, not the grading-coverage limitations listed above.

@0xashrk
0xashrk requested a review from ericjuta as a code owner September 11, 2026 15:09
@ericjuta
ericjuta merged commit aa356af into askgina:main Sep 11, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants