Skip to content

Reject invalid tool calls in agent loops#27

Open
gold-silver-copper wants to merge 14 commits into
mainfrom
codex/core-tool-call-validation
Open

Reject invalid tool calls in agent loops#27
gold-silver-copper wants to merge 14 commits into
mainfrom
codex/core-tool-call-validation

Conversation

@gold-silver-copper

@gold-silver-copper gold-silver-copper commented May 22, 2026

Copy link
Copy Markdown
Owner

Summary

Adds a generic agent-loop validation boundary for model-emitted tool calls before they can be dispatched or exposed to streaming callers, and aligns validation with the effective provider tool contract where Rig can infer or construct that contract.

Changes include:

  • Add typed invalid tool-call reporting with CompletionError::InvalidToolCall.
  • Add ToolCallContract / ToolCallNameValidator for request-declared tools, provider-native function declarations, hosted provider tools, and tool-choice allowlists.
  • Add typed UnknownToolCallError for model-requested tools that are not registered with the prompt.
  • Validate non-streaming tool calls after response hooks but before history/tool dispatch.
  • Validate completed streaming tool calls before public yield/dispatch.
  • Validate streamed tool-call name deltas before hooks or public yield, and reject argument deltas before a validated name.
  • Keep unknown tool dispatch failures as hard agent-loop errors instead of stringifying ToolNotFoundError into successful tool results.
  • Parse raw native tool declarations from additional_params.tools for OpenAI/Gemini-style shapes and Anthropic-style { name, input_schema } function tools.
  • Include provider-native function declarations in validation while still requiring a matching local Rig tool before agent execution.
  • Treat matching local tools as execution-only when a provider-native function declaration with the same name is present, avoiding duplicate provider declarations.
  • Move Gemini raw tool-choice parsing out of the generic contract and into Gemini provider contract construction.
  • Reject duplicate generic/raw Gemini tool-choice controls in request conversion.
  • Normalize provider-native function schemas through provider-specific conversion paths, including input_schema support for OpenAI Responses, Gemini REST, Gemini Interactions, and xAI.
  • Reject Gemini REST code_execution until executableCode / codeExecutionResult response parts are preserved instead of accepting partial request-only support.
  • Fail closed for providers that do not send tools, including Hyperbolic, Mira, and Perplexity, instead of warning and dropping tool declarations.
  • Add focused streaming and non-streaming regressions for undeclared, disallowed, provider-hosted, native-function, duplicate-declaration, and unsupported-provider tool cases.

Intentionally out of scope for this PR:

  • A provider repair retry loop after typed validation errors.
  • Live default_api provider canaries.
  • A separate external-routing hook for provider-native function calls that are not backed by local Rig tools.
  • Gemini REST code-execution response preservation.

Validation

Passed locally:

  • cargo fmt
  • git diff --check
  • cargo test -p rig-core tool_call_contract --lib
  • cargo test -p rig-core provider_function_tool_normalizes_input_schema --lib
  • cargo test -p rig-core raw_additional_params_tools --lib
  • cargo test -p rig-core gemini_contract_uses_raw --lib
  • cargo test -p rig-core code_execution --lib
  • cargo test -p rig-core provider_native_function_names_filter_duplicate_local_declarations --lib
  • cargo test -p rig-core --lib (706 passed, 8 ignored)
  • cargo check --all-features
  • cargo clippy --all-targets --all-features

@gold-silver-copper

Copy link
Copy Markdown
Owner Author

Pushed follow-up fix commit cb852225.

This addresses the remaining review items:

  • normalizes provider tool declarations so raw additional_params.tools is extracted into typed tools for OpenAI-compatible adapters or rejected where ambiguous;
  • filters duplicate local/provider-native function declarations for Bedrock, VertexAI, and Gemini gRPC;
  • pre-validates all non-streaming tool calls before dispatch so unknown tools cannot race after valid side-effecting tools;
  • rejects unsupported multi-name ToolChoice::Specific construction for Bedrock and Anthropic.

Validation passed locally:

  • cargo fmt
  • git diff --check
  • cargo check --all-features
  • cargo clippy --all-targets --all-features
  • cargo test
  • targeted regressions for rig-core, rig-bedrock, rig-vertexai, rig-gemini-grpc, and --test openai.

@gold-silver-copper

Copy link
Copy Markdown
Owner Author

Pushed 7cabb99c to address the remaining tool_choice/request divergence findings.

Changes:

  • ToolChoice::None now serializes as no tool declarations for Ollama, Llamafile, and Bedrock.
  • Ollama and Llamafile now reject Required / Specific tool choices because those require provider-side enforcement.
  • Ollama and Llamafile reject raw additional_params.tool_choice instead of letting it diverge from Rig validation.
  • Added regression coverage for the new Ollama/Llamafile behavior and updated the Bedrock None expectation.

Validation run locally:

  • cargo fmt
  • git diff --check
  • cargo test -p rig-core tool_choice --lib
  • cargo test -p rig-bedrock tool_choice --lib
  • cargo check --all-features
  • cargo test -p rig-core --lib
  • cargo test -p rig-bedrock --lib
  • cargo clippy --all-targets --all-features
  • cargo test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant