Capability grounding: folder-mode agents get their manifest back; gateway honors its own contract - #2626
Merged
Merged
Conversation
added 2 commits
September 3, 2026 20:05
… the gateway honors its own contract Root-caused from live user reports (custom agents 'can't find' connected MCP tools; a model announcing 'file_write wasn't available in this session'). Git forensics traced the keystone to PR #2300 (Aug 5): the enabled-capabilities manifest gained a usesHostFolderTools exclusion whose one-line rationale ('host-folder agents retain their workspace-only prompt') deleted the ENTIRE manifest - MCP and plugin ids included - for exactly the 'research the web and save files to my folder' configuration. The model was never told any of its capabilities existed; its only route was a blind capabilities search it had no reason to issue correctly. The polite small-model failure mode made it read as model stupidity for a month. Fixes, each pinned by a test or eval: - Folder-mode custom agents compose the enabled-capabilities manifest again (both gate sites). The capabilities gateway was already in the folder-mode schema and loads already worked; the model just was never told what to load. - The gateway's advertised pagination is now a callable contract: the bare-call result says 'Next page: {"list": "enabled", "page": N}' but list/page were undeclared, so the strict preflight rejected the harness's own hint - an agent with >40 enabled capability lines could never see page 2. This is the literal 'errors when model checks capabilities' report. - capabilities joins fullBootstrapToolNames: it was skeletonized on custom agents from the day it replaced the legacy pair, stripping the load-bearing 'IDs are values, never callable function names' prose. - The bare-call path resolves the agent via ChatExecutionContext before activeAgent, so delegated/background runs list THEIR capabilities, not another window's. - Bare-name id rescue: {"ids": ["Exa_search"]} resolves to tool/<name> when it names a registered dynamic tool, mirroring the registry's existing reverse rescue. Unknown bare names keep the format error. - Workspace file tools dead-end actionably: the opaque refusal became 'ask the user to attach a folder via the Folder chip; use share_artifact meanwhile' in both the scope gate and the capabilities load path. The refusal itself stands - no execution boundary moves. - Discovery stops masking built-ins: the per-agent grant is documented as scoping dynamic tools but was applied to every hit, and built-ins can never be IN the grant - seeded agents could not discover any indexed built-in. Built-ins are exempted; configure tools stay masked; authoritatively-gated tools never enter the index. Evals: new floors-gated CapabilityGrounding suite (deterministic, token-free) pins folder-mode manifest presence via the prompt-surface census - PROVEN fail-closed: both cases FAIL against the unfixed composer (section 'enabledManifest' MISSING) and PASS with the fix. The prompt_surface runner gains enableTools fixture parity and a composeModel override so floors-gated cases cannot skip vacuously on tiny run models.
…non-discoverable set Adversarial audit of the previous commit found its workspace-tool guidance was keyed on builtInToolNames — but workspace tools are RUNTIME-MANAGED, which made one of the two new messages dead code and bypassed the other in the common production state (folder tools register process-wide the first time any chat mounts a folder, then read .alreadyLoaded and the loadable-hint steered the model into a capabilities load the dynamic gates refuse). Observed live before this fix: the model improvised a scratchpad artifact and told the user to 'enable a file-write tool in the agent's capability settings'. - Scope gate: the Folder-chip guidance now precedes the loadable-hint and keys purely on coreWorkspaceToolNames membership - byte-identical answer whether or not the tools are registered. - Load path: the workspace refusal moved out of the isBuiltIn branch and ahead of the notRegistered guard - same state-independence. - isDynamicRegisteredTool now matches listDynamicTools exactly (excludes runtime-managed too), so a bare 'file_write' id is no longer rescued into the dynamic-load gates' opaque refusals. - nonDiscoverableBuiltInToolNames gains the spawn trio + applescript/ mac_query: same authoritative per-agent contract as computer_use/ browser_use; discovering them with the flag off produced a discover->load dead loop. - Tests upgraded to dual-state (folder tools registered AND unregistered, same message both ways, never a loader steer); the folder-mode section- ordering pin gets an explicit empty grant so the restored manifest can't make it registry-state dependent. Live-proven on the dev build: no-folder ask answers honestly with the artifact fallback; + -> Add Folder -> Select Working Directory -> proof-workspace chip; file_write then writes notes.txt (verified on disk, 5 bytes 'hello'); a bare capabilities call in the folder-attached chat returns the paginated enabled list and the model relays it correctly.
EvalCatalogManifestTests regenerates the suite->case-id map from disk and diffs it against the committed manifest; the new suite's two cases were on disk but not in the manifest, failing the evals harness lane.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The regression (root-caused with commit-level proof)
Live user reports (r/osaurus): custom agents "can't find" connected MCP tools (Exa, fetch) or file tools; a model announced "file_write wasn't available in this session" and silently fell back to an artifact; thread titled "Errors when model checks capabilities". All on small local models.
Git forensics traced the keystone to PR #2300 (Aug 5,
42e64580a):resolveEnabledManifestgained a!executionMode.usesHostFolderToolsguard whose one-line rationale ("host-folder agents retain their workspace-only prompt") deleted the entire enabled-capabilities manifest — MCP and plugin ids included — for exactly the "research the web and save files to my folder" configuration. The model was never told any of its capabilities existed. Supporting timeline: #2250 (manifest gone for all custom agents, Aug 1–5), #2393 (bare-capabilities dead loop, fixed Aug 14), #2308 (MCP index race, fixed Aug 7) — F1 was the piece still shipping.Fixes
capabilitiesgateway was already in the folder-mode schema and loads already worked — the model just was never told what to load.{"list": "enabled", "page": N}was undeclared, so the strict preflight rejected the harness's own "Next page" hint — >40 enabled lines meant page 2 was unreachable.capabilitiesjoinsfullBootstrapToolNames— it had been skeletonized on custom agents since it replaced the legacy pair, stripping the load-bearing "IDs are values, never callable function names" prose.ChatExecutionContext.currentAgentId— delegated/background runs listed another agent's capabilities.{"ids": ["Exa_search"]}resolves totool/<name>for registered dynamic tools (mirrors the registry's reverse rescue). Unknown names keep the format error.builtInToolNames; workspace tools are runtime-managed, making one message dead code and the other bypassed — caught by adversarial audit, repaired in the second commit, pinned by dual-state tests).Evals (fail-closed proven)
New floors-gated
CapabilityGroundingsuite (deterministic, token-free, prompt-surface census through the production compose path): FAILS against the unfixed composer (section 'enabledManifest' MISSING, exit 1) and PASSES with the fix. It would have caught #2300 on merge day. Kit:enableToolsfixture parity in the prompt_surface runner +composeModeloverride so floors-gated cases can't skip vacuously on tiny run models.Live proof (dev build, JANG_2L)
proof-workspacechip;file_writewritesnotes.txt(verified on disk: 5 bytes,hello); a barecapabilitiescall in the folder-attached chat returns the paginated enabled list and the model relays it correctly — the reported failure scenario, healthy.Unit tests: pagination contract through the registry preflight, bare-name rescue + unknown-name error, dual-state workspace dead-ends (registered AND unregistered), section-ordering pin made registry-state independent.