Improve capabilities docs by dsfaccini · Pull Request #4832 · pydantic/pydantic-ai

dsfaccini · 2026-03-24T23:33:11Z

Summary

Dissected PR Add Capability abstraction, AgentSpec, Hooks, unified thinking setting, per-run toolset isolation, builtin tool fallback to local tools #4640 (Capability abstraction) and identified 6 bugs, each confirmed with a reproducing test
Added doc gap analysis (DOCS-QUESTIONS.md) and niche caveat notes (NICHE-DOCS.md)
Minor xref/formatting fixes in capabilities docs

Bugs found

Hooks after_* fires forward, should reverse — violates the documented contract in docs/hooks.md ("after_* hooks fire in reverse order" including "on the same Hooks instance")
DynamicToolset for_run_step() no error recovery — if factory or __aenter__ raises, old toolset is lost with no rollback
on_*_error handlers chain-replace original error — when handler A raises a new exception, handler B sees the new error, not the original
Capability returning self from for_run() with mutated state uses stale cache — identity check misses mutations when for_run() returns self
Tool retry count persists across DynamicToolset tool swaps — ToolManager tracks retries by name, so swapped implementations inherit old counts
History processor composition creates orphaned tool returns — no validation that processed messages remain semantically consistent

Test plan

uv run pytest tests/test_capabilities_bugs.py -v — 12 tests, all pass
make lint — clean
make typecheck — clean (pyright passes)
AI generated code

🤖 Generated with Claude Code

Dissect PR pydantic#4640 (Capability abstraction) and identify 6 bugs: 1. Hooks after_* fires forward, should reverse (violates docs/hooks.md contract) 2. DynamicToolset for_run_step() has no error recovery on factory/enter failure 3. on_*_error handlers chain-replace original error, losing context 4. Capability returning self from for_run() with mutated state uses stale cache 5. Tool retry count persists across DynamicToolset tool swaps 6. History processor composition can create orphaned tool returns Each bug has a reproducing test in tests/test_capabilities_bugs.py. DOCS-QUESTIONS.md lists doc gaps found by a separate review. NICHE-DOCS.md has caveat notes for bugs 2, 5, 6 (niche edge cases). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

docs/capabilities.md

docs/hooks.md

pydantic_ai_slim/pydantic_ai/capabilities/hooks.py

tests/test_capabilities_bugs.py

DOCS-QUESTIONS.md

NICHE-DOCS.md

Exit old toolset before calling factory in for_run_step(), so that if the factory or new toolset's __aenter__ raises, the old toolset has been properly cleaned up. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

devin-ai-integration

Devin Review found 1 new potential issue.

View 7 additional findings in Devin Review.

devin-ai-integration · 2026-03-25T19:52:53Z

pydantic_ai_slim/pydantic_ai/toolsets/_dynamic.py

+        # Exit old toolset before evaluating factory
+        old_toolset = self._toolset
+        self._toolset = None
+        if old_toolset is not None:
+            await old_toolset.__aexit__(None, None, None)

-        # Manage the transition in-place
-        if self._toolset is not None:
-            await self._toolset.__aexit__(None, None, None)
+        new_toolset = await self._evaluate_factory(ctx)
        self._toolset = new_toolset
        if self._toolset is not None:
            await self._toolset.__aenter__()


🚩 DynamicToolset identity check removal changes behavior for stable-instance factories

The old for_run_step had an optimization: if new_toolset is self._toolset: return self which skipped the __aexit__/__aenter__ cycle when the factory returned the same toolset instance. The new code always exits the old toolset before evaluating the factory, making this optimization impossible. For factories that return the same stateful toolset (e.g., one with real connection resources in __aenter__/__aexit__), this means every run step will close and reopen those resources unnecessarily.

The existing test test_dynamic_toolset_for_run_step_same_instance_skips_transition at tests/test_toolsets.py:1116 still passes, but only because FunctionToolset has no-op lifecycle methods. Its docstring ("skips transition when factory returns the same instance") and inline comment ("early return without transition") no longer describe actual behavior.

This appears to be a deliberate design tradeoff (cleanup safety over performance), not a bug — but the test documentation is now misleading, and downstream users relying on identity-based skip behavior may notice a behavioral change.

Was this helpful? React with 👍 or 👎 to provide feedback.

Devin found the same on my PR :D #4846 (comment)

DouweM · 2026-03-27T00:44:27Z

@dsfaccini Didn't mean to close this; you'll just want to fix merge conflicts / see what I already fixed that's not needed here anymore.

# Conflicts: # docs/capabilities.md # pydantic_ai_slim/pydantic_ai/toolsets/_dynamic.py

devin-ai-integration

Devin Review found 4 new potential issues.

View 9 additional findings in Devin Review.

devin-ai-integration · 2026-03-27T15:34:20Z

docs/api/agent_spec.md

+# `pydantic_ai.agent.spec`
+
+::: pydantic_ai.agent.spec
+    options:
+        members:
+            - AgentSpec
+
+::: pydantic_ai._template
+    options:
+        members:
+            - TemplateStr


🟡 New docs/api/agent_spec.md page not added to mkdocs.yml navigation

The new API reference page docs/api/agent_spec.md (documenting AgentSpec and TemplateStr) was created but never added to the mkdocs.yml nav section. All other API reference pages are listed under API Reference > pydantic_ai (see mkdocs.yml:140-187), but api/agent_spec.md is missing. This means users cannot discover the page through site navigation, and the TemplateStr API docs (which are only rendered via the ::: directive in this file) are effectively hidden. The page should be added between api/agent.md and api/builtin_tools.md in the nav.

Prompt for agents

Add docs/api/agent_spec.md to the mkdocs.yml navigation. In mkdocs.yml, find the API Reference > pydantic_ai section (around line 143) where api/agent.md is listed, and add api/agent_spec.md immediately after it, before api/builtin_tools.md.

Was this helpful? React with 👍 or 👎 to provide feedback.

Good point Devin

devin-ai-integration · 2026-03-27T15:34:21Z

docs/agent-spec.md

 ```

-[`Agent.from_spec`][pydantic_ai.Agent.from_spec] accepts a dict or [`AgentSpec`][pydantic_ai.agent.spec.AgentSpec] instance and supports additional keyword arguments that supplement or override the spec:
+[`Agent.from_spec`][pydantic_ai.agent.Agent.from_spec] accepts a dict or [`AgentSpec`](#agentspec-reference) instance and supports additional keyword arguments that supplement or override the spec:


🔴 AgentSpec reference changed from API xref to anchor link, violating doc rules

The AgentSpec reference was changed from [AgentSpec][pydantic_ai.agent.spec.AgentSpec] to [AgentSpec](#agentspec-reference). This violates docs/.cursor/rules.mdc ("Always reference the python code in the docs e.g. ModelSettings should link to [ModelSettings][pydantic_ai.settings.ModelSettings]") and docs/AGENTS.md rule:66 ("Use reference-style links for API elements: [ElementName][module.path.ElementName]"). Notably, the same file still uses proper API xrefs for AgentSpec in other places (docs/agent-spec.md:70, docs/agent-spec.md:173), making this inconsistent.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-03-27T15:34:22Z

docs/agent-spec.md

 ## `AgentSpec` reference

-The [`AgentSpec`][pydantic_ai.agent.spec.AgentSpec] model represents the full spec structure:
+The [`AgentSpec`](#agentspec-reference) model represents the full spec structure:


🔴 Second AgentSpec reference changed from API xref to anchor link, violating doc rules

Same violation as in line 37: AgentSpec is referenced as [AgentSpec](#agentspec-reference) instead of [AgentSpec][pydantic_ai.agent.spec.AgentSpec], violating the mandatory documentation rules in docs/.cursor/rules.mdc and docs/AGENTS.md rule:66 that require API elements to use reference-style links.

Was this helpful? React with 👍 or 👎 to provide feedback.

This now links to this same section lol. Should link to API.

docs/agent-spec.md

DouweM · 2026-03-27T16:58:17Z

docs/api/agent_spec.md

+# `pydantic_ai.agent.spec`
+
+::: pydantic_ai.agent.spec
+    options:
+        members:
+            - AgentSpec
+
+::: pydantic_ai._template
+    options:
+        members:
+            - TemplateStr


Good point Devin

docs/agent-spec.md

DouweM · 2026-03-27T17:00:20Z

docs/agent-spec.md

 ## `AgentSpec` reference

-The [`AgentSpec`][pydantic_ai.agent.spec.AgentSpec] model represents the full spec structure:
+The [`AgentSpec`](#agentspec-reference) model represents the full spec structure:


This now links to this same section lol. Should link to API.

DouweM · 2026-03-27T17:00:50Z

docs/api/agent_spec.md

+::: pydantic_ai.agent.spec
+    options:
+        members:
+            - AgentSpec


I wouldn't mind this one being documented on api/agent.md

DouweM · 2026-03-27T17:01:00Z

docs/api/agent_spec.md

+        members:
+            - AgentSpec
+
+::: pydantic_ai._template


@dsfaccini You know this is wrong :) Please review your own PRs!

DouweM · 2026-03-27T17:01:51Z

docs/capabilities.md

@dsfaccini Weren't there a ton of docs gaps that were going to be addressed as part of this PR?

devin-ai-integration

Devin Review found 5 new potential issues.

View 9 additional findings in Devin Review.

devin-ai-integration · 2026-03-27T20:45:48Z

docs/agent-spec.md

 ## Template strings

-[`TemplateStr`][pydantic_ai.TemplateStr] provides Handlebars-style templates (`{{variable}}`) that are rendered against the agent's [dependencies](dependencies.md) at runtime. In spec files, strings containing `{{` are automatically converted to template strings:
+`TemplateStr` provides Handlebars-style templates (`{{variable}}`) that are rendered against the agent's [dependencies](dependencies.md) at runtime. In spec files, strings containing `{{` are automatically converted to template strings:


🔴 TemplateStr link removed in docs/agent-spec.md line 74, violating mandatory docs linking rule

The mandatory rule in docs/.cursor/rules.mdc requires: "Always reference the python code in the docs e.g. ModelSettings should link to [ModelSettings][pydantic_ai.settings.ModelSettings]." This line was changed from [TemplateStr][pydantic_ai.TemplateStr] to bare `TemplateStr`, actively removing the API reference link. TemplateStr is exported from pydantic_ai (pydantic_ai_slim/pydantic_ai/__init__.py:3).

Suggested change

`TemplateStr` provides Handlebars-style templates (`{{variable}}`) that are rendered against the agent's [dependencies](dependencies.md) at runtime. In spec files, strings containing `{{` are automatically converted to template strings:

[`TemplateStr`][pydantic_ai.TemplateStr] provides Handlebars-style templates (`{{variable}}`) that are rendered against the agent's [dependencies](dependencies.md) at runtime. In spec files, strings containing `{{` are automatically converted to template strings:

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-03-27T20:45:49Z

docs/agent-spec.md

 Template variables are resolved from the fields of the `deps` object. When a `deps_type` (or [`deps_schema`](#deps_schema)) is provided, template variable names are validated at construction time.

-In Python code, [`TemplateStr`][pydantic_ai.TemplateStr] can be used explicitly, but a callable with [`RunContext`][pydantic_ai.tools.RunContext] is generally preferred for IDE autocomplete and type checking:
+In Python code, `TemplateStr` can be used explicitly, but a callable with [`RunContext`][pydantic_ai.tools.RunContext] is generally preferred for IDE autocomplete and type checking:


🔴 TemplateStr link removed in docs/agent-spec.md line 82, violating mandatory docs linking rule

Same mandatory rule violation as BUG-0001. This line was changed from [TemplateStr][pydantic_ai.TemplateStr] to bare `TemplateStr`, removing the API reference link.

Suggested change

In Python code, `TemplateStr` can be used explicitly, but a callable with [`RunContext`][pydantic_ai.tools.RunContext] is generally preferred for IDE autocomplete and type checking:

In Python code, [`TemplateStr`][pydantic_ai.TemplateStr] can be used explicitly, but a callable with [`RunContext`][pydantic_ai.tools.RunContext] is generally preferred for IDE autocomplete and type checking:

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-03-27T20:45:51Z

docs/capabilities.md

+`before_model_request` hooks see the full `request_context.messages` list, including any [message history](message-history.md) passed to `agent.run()`, and can modify it.
+
+!!! note "Skip and chain behavior"
+    All skip exceptions (`SkipModelRequest`, `SkipToolValidation`, `SkipToolExecution`) short-circuit the hook chain: remaining capabilities' `before_*` hooks do not fire, and `after_*` hooks are not called for the skipped operation. A skip raised from `wrap_*` propagates immediately — inner capabilities' wrap hooks never execute.


🔴 SkipModelRequest, SkipToolValidation, SkipToolExecution unlinked in new docs content

The mandatory rule in docs/.cursor/rules.mdc requires all Python code references to be linked. Line 558 introduces new content with three bare-backtick exception class references (SkipModelRequest, SkipToolValidation, SkipToolExecution) that are not linked to their API paths. These are all in pydantic_ai.exceptions and are linked correctly elsewhere in the same file (e.g., docs/capabilities.md:553).

Suggested change

All skip exceptions (`SkipModelRequest`, `SkipToolValidation`, `SkipToolExecution`) short-circuit the hook chain: remaining capabilities' `before_*` hooks do not fire, and `after_*` hooks are not called for the skipped operation. A skip raised from `wrap_*` propagates immediately — inner capabilities' wrap hooks never execute.

All skip exceptions ([`SkipModelRequest`][pydantic_ai.exceptions.SkipModelRequest], [`SkipToolValidation`][pydantic_ai.exceptions.SkipToolValidation], [`SkipToolExecution`][pydantic_ai.exceptions.SkipToolExecution]) short-circuit the hook chain: remaining capabilities' `before_*` hooks do not fire, and `after_*` hooks are not called for the skipped operation. A skip raised from `wrap_*` propagates immediately — inner capabilities' wrap hooks never execute.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-03-27T20:45:52Z

docs/capabilities.md

+| [`before_tool_validate`][pydantic_ai.capabilities.AbstractCapability.before_tool_validate] | `(ctx: `[`RunContext`][pydantic_ai.tools.RunContext]`, *, call: `[`ToolCallPart`][pydantic_ai.messages.ToolCallPart]`, tool_def: `[`ToolDefinition`][pydantic_ai.tools.ToolDefinition]`, args: `[`RawToolArgs`][pydantic_ai.capabilities.RawToolArgs]`) -> `[`RawToolArgs`][pydantic_ai.capabilities.RawToolArgs] | Modify raw args before validation (e.g. JSON repair) |
+| [`after_tool_validate`][pydantic_ai.capabilities.AbstractCapability.after_tool_validate] | `(ctx: `[`RunContext`][pydantic_ai.tools.RunContext]`, *, call: `[`ToolCallPart`][pydantic_ai.messages.ToolCallPart]`, tool_def: `[`ToolDefinition`][pydantic_ai.tools.ToolDefinition]`, args: `[`ValidatedToolArgs`][pydantic_ai.capabilities.ValidatedToolArgs]`) -> `[`ValidatedToolArgs`][pydantic_ai.capabilities.ValidatedToolArgs] | Modify validated args |
+| [`wrap_tool_validate`][pydantic_ai.capabilities.AbstractCapability.wrap_tool_validate] | `(ctx: `[`RunContext`][pydantic_ai.tools.RunContext]`, *, call: `[`ToolCallPart`][pydantic_ai.messages.ToolCallPart]`, tool_def: `[`ToolDefinition`][pydantic_ai.tools.ToolDefinition]`, args: `[`RawToolArgs`][pydantic_ai.capabilities.RawToolArgs]`, handler: `[`WrapToolValidateHandler`][pydantic_ai.capabilities.WrapToolValidateHandler]`) -> `[`ValidatedToolArgs`][pydantic_ai.capabilities.ValidatedToolArgs] | Wrap the validation step |
+| [`on_tool_validate_error`][pydantic_ai.capabilities.AbstractCapability.on_tool_validate_error] | `(ctx: `[`RunContext`][pydantic_ai.tools.RunContext]`, *, call: `[`ToolCallPart`][pydantic_ai.messages.ToolCallPart]`, tool_def: `[`ToolDefinition`][pydantic_ai.tools.ToolDefinition]`, args: `[`RawToolArgs`][pydantic_ai.capabilities.RawToolArgs]`, error: ValidationError | `[`ModelRetry`][pydantic_ai.exceptions.ModelRetry]`) -> `[`ValidatedToolArgs`][pydantic_ai.capabilities.ValidatedToolArgs] | Handle validation errors (see [error hooks](#error-hooks)) |


🔴 ValidationError unlinked in on_tool_validate_error signature table

The mandatory rule in docs/.cursor/rules.mdc requires all Python code references to be linked. In the changed on_tool_validate_error signature on line 573, ValidationError (from pydantic) is bare-backticked while ModelRetry next to it is properly linked. The old version had Exception (which is a builtin), but the new ValidationError is a pydantic class that should be linked as [ValidationError][pydantic.ValidationError] per the docs rule.

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-03-27T20:45:53Z

docs/capabilities.md

+| [`get_instructions()`][pydantic_ai.capabilities.AbstractCapability.get_instructions] | [`AgentInstructions`][pydantic_ai.agent.AgentInstructions] \| `None` | [Instructions](agent.md#instructions) (static strings, [template strings](agent-spec.md#template-strings), or callables) |
+| [`get_model_settings()`][pydantic_ai.capabilities.AbstractCapability.get_model_settings] | [`AgentModelSettings`][pydantic_ai.agent.AgentModelSettings] \| `None` | [Model settings](agent.md#model-run-settings) dict, or a callable for per-step settings |


🚩 AgentInstructions and AgentModelSettings link paths changed in configuration methods reference table

Lines 432-433 change the cross-reference paths for AgentInstructions from pydantic_ai._instructions.AgentInstructions to pydantic_ai.agent.AgentInstructions, and AgentModelSettings from pydantic_ai.agent.abstract.AgentModelSettings to pydantic_ai.agent.AgentModelSettings. This aligns with the new __all__ export added in pydantic_ai_slim/pydantic_ai/agent/__init__.py:114 ('AgentInstructions'). However, AgentInstructions is defined in pydantic_ai._instructions — these links will only resolve if mkdocstrings can follow the re-export chain. Worth verifying with make docs-serve that these links actually resolve.

Was this helpful? React with 👍 or 👎 to provide feedback.

github-actions bot added size: L Large PR (501-1500 weighted lines) bug Report that something isn't working, or PR implementing a fix labels Mar 24, 2026

This comment was marked as resolved.

Sign in to view

readd crossrefs and sequence diagram

2df76aa

This comment was marked as resolved.

Sign in to view

move mermaid

b6edd2e

DouweM requested changes Mar 25, 2026

View reviewed changes

fix docs

ee9002c

github-actions bot added size: M Medium PR (101-500 weighted lines) and removed size: L Large PR (501-1500 weighted lines) labels Mar 25, 2026

This comment was marked as resolved.

Sign in to view

dsfaccini and others added 2 commits March 25, 2026 13:17

merge upstream/main, fix DynamicToolset exit ordering

306802c

Exit old toolset before calling factory in for_run_step(), so that if the factory or new toolset's __aenter__ raises, the old toolset has been properly cleaned up. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

remove remaining files

d6f1367

devin-ai-integration bot reviewed Mar 25, 2026

View reviewed changes

DouweM mentioned this pull request Mar 25, 2026

Fix capability type inference, support sync prepare_tools functions, MCP capability with just a URL #4846

Merged

4 tasks

add new plan about dfoc improvments

0f00d20

DouweM closed this in #4846 Mar 25, 2026

DouweM reopened this Mar 27, 2026

Merge remote-tracking branch 'upstream/main' into bugs-in-capabilities

6256d9e

# Conflicts: # docs/capabilities.md # pydantic_ai_slim/pydantic_ai/toolsets/_dynamic.py

This comment was marked as resolved.

Sign in to view

dsfaccini added 2 commits March 27, 2026 10:25

fix xrefs

a2772d2

remove plan

37f63f6

devin-ai-integration bot reviewed Mar 27, 2026

View reviewed changes

DouweM changed the title ~~bug: Capabilities review~~ Improve capabilities docs Mar 27, 2026

DouweM requested changes Mar 27, 2026

View reviewed changes

dsfaccini added 2 commits March 27, 2026 12:57

xrefs and backticks

583f6d0

enhance capabilities docs

0374cd0

devin-ai-integration bot reviewed Mar 27, 2026

View reviewed changes

	`TemplateStr` provides Handlebars-style templates (`{{variable}}`) that are rendered against the agent's [dependencies](dependencies.md) at runtime. In spec files, strings containing `{{` are automatically converted to template strings:
	[`TemplateStr`][pydantic_ai.TemplateStr] provides Handlebars-style templates (`{{variable}}`) that are rendered against the agent's [dependencies](dependencies.md) at runtime. In spec files, strings containing `{{` are automatically converted to template strings:

	In Python code, `TemplateStr` can be used explicitly, but a callable with [`RunContext`][pydantic_ai.tools.RunContext] is generally preferred for IDE autocomplete and type checking:
	In Python code, [`TemplateStr`][pydantic_ai.TemplateStr] can be used explicitly, but a callable with [`RunContext`][pydantic_ai.tools.RunContext] is generally preferred for IDE autocomplete and type checking:

		\| [`get_instructions()`][pydantic_ai.capabilities.AbstractCapability.get_instructions] \| [`AgentInstructions`][pydantic_ai.agent.AgentInstructions] \\| `None` \| [Instructions](agent.md#instructions) (static strings, [template strings](agent-spec.md#template-strings), or callables) \|
		\| [`get_model_settings()`][pydantic_ai.capabilities.AbstractCapability.get_model_settings] \| [`AgentModelSettings`][pydantic_ai.agent.AgentModelSettings] \\| `None` \| [Model settings](agent.md#model-run-settings) dict, or a callable for per-step settings \|

Conversation

dsfaccini commented Mar 24, 2026

Summary

Bugs found

Test plan

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DouweM commented Mar 27, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot Mar 27, 2026

Choose a reason for hiding this comment

devin-ai-integration bot Mar 25, 2026 •

edited

Loading