Skip to content

feat(google): support structured output + tool combination for Gemini 3#4848

Draft
dsfaccini wants to merge 4 commits intopydantic:mainfrom
dsfaccini:google-structured-outputs
Draft

feat(google): support structured output + tool combination for Gemini 3#4848
dsfaccini wants to merge 4 commits intopydantic:mainfrom
dsfaccini:google-structured-outputs

Conversation

@dsfaccini
Copy link
Copy Markdown
Collaborator

@dsfaccini dsfaccini commented Mar 25, 2026

Summary

Plan-only PR — this PR contains a PLAN.md for review before implementation begins.

Google Gemini 3 supports:

The plan proposes:

  • Two new profile flags gating these restrictions for Gemini 3+
  • A new tests/models/google/test_structured_output.py test file (9 test cases)
  • Removing 3 superseded tests from test_google.py
  • SDK investigation for include_server_side_tool_invocations (added in google-genai 1.68.0)

See PLAN.md for full details.

Pre-Review Checklist

  • Any AI generated code has been reviewed line-by-line by the human PR author, who stands by it.
  • No breaking changes in accordance with the version policy.
  • Linting and type checking pass per make format and make typecheck.
  • PR title is fit for the release changelog.

Pre-Merge Checklist

  • New tests for any fix or new behavior, maintaining 100% coverage.
  • Updated documentation for new features and behaviors, including docstrings for API docs.

@claude
plan: Google structured output + tool combination for Gemini 3
256aa2f 
Covers pydantic#4801 (NativeOutput + function tools) and pydantic#4788 (function + builtin tools).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@github-actions github-actions bot added size: S Small PR (≤100 weighted lines) feature New feature request, or PR implementing a feature (enhancement) labels Mar 25, 2026
DouweM
DouweM requested changes Mar 25, 2026
View reviewed changes
PLAN.md Outdated
if model_request_parameters.builtin_tools:
if model_request_parameters.function_tools:
if not GoogleModelProfile.from_profile(self.profile).google_supports_function_tools_with_builtin_tools:
raise UserError('Google does not support function tools and built-in tools at the same time.')
Copy link
Copy Markdown
Collaborator

@DouweM DouweM Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should specify that it's a model-specific issue, as frontier models do support it

PLAN.md Outdated
if model_request_parameters.function_tools:
if not GoogleModelProfile.from_profile(self.profile).google_supports_native_output_with_function_tools:
raise UserError(
'Google does not support `NativeOutput` and function tools at the same time. Use `output_type=ToolOutput(...)` instead.'
Copy link
Copy Markdown
Collaborator

@DouweM DouweM Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same

PLAN.md Outdated

#### 2c. No changes to `prepare_request`

The existing `prepare_request` (line 286-301) handles `output_tools + builtin_tools` using `google_supports_native_output_with_builtin_tools`. This already correctly resolves auto mode to 'native' for Gemini 3 when builtin_tools + output_tools are present.
Copy link
Copy Markdown
Collaborator

@DouweM DouweM Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that's "correctly" -- since Gemini 3+ now supports function tools and builtin tools at the same time, we can use output tools + builtin tools just fine, and don't need to fallback on native.

Claude here seemed confused judging by " output_tools + builtin_tools using google_supports_native_output_with_builtin_tools", as it conflates "output tools" with "native output"

PLAN.md Outdated
The existing `prepare_request` (line 286-301) handles `output_tools + builtin_tools` using `google_supports_native_output_with_builtin_tools`. This already correctly resolves auto mode to 'native' for Gemini 3 when builtin_tools + output_tools are present.

For auto mode + function_tools + builtin_tools on Gemini 3:
1. `prepare_request`: output_tools + builtin_tools → auto converts to 'native' (existing logic)
Copy link
Copy Markdown
Collaborator

@DouweM DouweM Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above: this is wrong, if the default is still tool output mode, we can remove the native workaround, and just use the standard behavior

PLAN.md Outdated
| NativeOutput + function + builtin | error | **works (new)** |
| function + builtin (no output type) | error | **works (new)** |
| auto + function + builtin + output_type | error | **works (new, auto->native)** |
| ToolOutput + builtin_tools | error | error (correct, use NativeOutput) |
Copy link
Copy Markdown
Collaborator

@DouweM DouweM Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wrong i believe

PLAN.md Outdated
```python
# In _build_content_and_config, when both tool types present:
if has_function_tools and has_builtin_tools:
config['include_server_side_tool_invocations'] = True
Copy link
Copy Markdown
Collaborator

@DouweM DouweM Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should always set this flag if any builtin tools are enabled and build BuiltinToolCall/BuiltinToolParts from the new toolCall/toolReturn parts we get back. We may be able to remove some of the current logic that tries to reconstruct those from e.g. groundingMetadata.

hetzner and others added 2 commits March 26, 2026 20:37
@claude
plan: update plan with SDK bump, single flag, and prepare_request cha…
3659d72 
…nges

Incorporates review feedback from DouweM:
- Single `google_supports_function_tools_with_builtin_tools` flag
- Skip prepare_request workaround for Gemini 3
- Always set `include_server_side_tool_invocations` with builtin tools
- Bump google-genai minimum to >=1.68.0
- ToolOutput + builtin_tools now works on Gemini 3 (no workaround)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@claude
plan: address review feedback on plan v2
b47e962 
- Rename flag to `google_supports_tool_combination` (matches Google docs)
- Replace line number refs with function/class names
- Negate condition instead of `if flag: pass`
- Add deferred docs section (output.md, builtin-tools.md, models/google.md)
- Clarify edge case table with UserError vs workaround distinction
- Expand Context with three distinct API mechanisms

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@github-actions github-actions bot added size: M Medium PR (101-500 weighted lines) and removed size: S Small PR (≤100 weighted lines) labels Mar 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DouweM DouweM DouweM requested changes

Assignees

No one assigned

Labels

feature New feature request, or PR implementing a feature (enhancement) size: M Medium PR (101-500 weighted lines)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Google Function Calling with Structured Output Google support both function tools and built-in tools

2 participants

@dsfaccini @DouweM