feat: add support for structured output schemas and improve agent capabilities

[frisbeeman]

• Integrated StructuredOutputFormat with workflows to enable JSON Schema-constrained outputs.
• Updated LLM provider to enforce schema validation for agent replies.
• Enhanced testing coverage for backward compatibility and schema behavior.
• Added a new structured_output_agent example demonstrating schema-constrained workflows.
• Migrated default LLM to OpenAI backend in tests for improved compatibility.
• Incremented dependencies with schemars integration for schema generation.

Summary by CodeRabbit

• New Features

  • Added support for structured JSON output schemas in agent workflows, allowing constraints on LLM-generated responses.
  • New example demonstrating structured output with strict schema validation.

• Chores

  • Updated existing examples to explicitly configure output schema settings.

[coderabbitai]

Walkthrough

This PR adds optional structured output schema support to the agent framework, enabling LLMs to produce strictly validated JSON responses. It extends workflow and activity state with an output_schema field, wires it through the LLM call chain, introduces a complete example with tool integration, and strengthens test validation with schema-driven assertions.

Changes

Structured Output Schema Feature

Layer / File(s)	Summary
State contract and serialization src/state.rs	AgentInput and LlmChatInput add optional output_schema: Option<StructuredOutputFormat> fields with serde defaults. Compaction preserves schemas. New tests verify roundtripping, null-field omission, and backward compatibility with legacy JSON.
LLM module integration src/llm.rs, src/prelude.rs	llm::chat signature extended to accept and pass output_schema to LLMProvider::chat_with_tools. StructuredOutputFormat is imported and re-exported via prelude.
Activity and workflow propagation src/activities.rs, src/workflow.rs	llm_chat activity logs and forwards output_schema to llm::chat. Workflow constructs activity input by copying output_schema from workflow input, ensuring end-to-end schema propagation.
Structured output example and dependencies Cargo.toml, examples/structured_output_agent/main.rs	New complete example with typed WeatherReport result, LookupTemperature tool with JSON parsing and error handling, strict JSON schema generation, OpenAI-compatible LLM worker, and client-side output deserialization and validation.
Existing example updates examples/interactive_math_agent/main.rs, examples/pipelined_math_agent/main.rs, examples/simple_math_agent/main.rs	All three math agent examples explicitly set output_schema: None in workflow inputs.
Test strengthening with structured output tests/agent_workflow.rs	Workflow test updated to use OpenAI-compatible LLM backend with Ollama, introduces typed Expression struct with JSON Schema validation, wires schema into workflow input, and replaces string assertions with field-level validation of operands and computation.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

• #5: This PR directly implements the structured output feature by adding AgentInput.output_schema field and wiring it through LlmChatInput, activities::llm_chat, and AgentWorkflow::run as described in the issue.

Poem

🐰 Schemas bloom like clover in the spring,
Constraining outputs to a structured thing,
Weather queries structured, tools align,
LLM responses crystalline—refined!
From state to chat, the schema takes the flight,
Making agent outputs perfectly tight! ✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main changes: adding support for structured output schemas and improving agent capabilities through integration with the LLM provider and workflow enhancements.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/structured-output

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (2)

tests/agent_workflow.rs (1)

200-216: ⚡ Quick win

Consider a slightly looser tolerance than f64::EPSILON.

At magnitude ~15-20, one ULP is already ~3.55e-15, while f64::EPSILON ≈ 2.22e-16. The current bounds hold only if the model emits the exact JSON literals from the prompt (15.4, 4.5, 19.9) — which is the assumption stated in the comment, but it's fragile:

• The add tool returns parsed.a + parsed.b, serialized via ryu. If the model echoes the tool's computed sum verbatim instead of recomputing/printing 19.9, the round-trip f64 can differ from 19.9_f64 by ~1 ULP, exceeding f64::EPSILON.
• Small reasoning models also occasionally normalize numbers (e.g., 19.90, 19.9000000000000004).

A tolerance like 1e-9 (or even 1e-6) preserves the "no hallucinated math" intent while making the assertion robust to a single rounding step. The schema/typed-shape check is what actually catches structural deviations.

♻️ Proposed tolerance loosening

-    // Tight epsilon — LLM emits JSON literals so values round-trip exactly,
-    // but clippy::float_cmp insists on `abs() < eps` over `==`.
-    assert!(
-        (parsed.left_operand - 15.4).abs() < f64::EPSILON,
+    // Allow one ULP of slack: the tool sums in f64 before serializing, so the
+    // model may echo the rounded sum rather than the literal.
+    const TOL: f64 = 1e-9;
+    assert!(
+        (parsed.left_operand - 15.4).abs() < TOL,
         "left_operand mismatch: got {}",
         parsed.left_operand
     );
     assert!(
-        (parsed.right_operand - 4.5).abs() < f64::EPSILON,
+        (parsed.right_operand - 4.5).abs() < TOL,
         "right_operand mismatch: got {}",
         parsed.right_operand
     );
     assert!(
-        (parsed.result - 19.9).abs() < f64::EPSILON,
+        (parsed.result - 19.9).abs() < TOL,
         "result mismatch: got {}",
         parsed.result
     );

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/agent_workflow.rs` around lines 200 - 216, The assertions using
f64::EPSILON are too strict for values ~15–20; replace the epsilon with a looser
tolerance (e.g., const TOL: f64 = 1e-9) and update the three checks that compare
parsed.left_operand, parsed.right_operand, and parsed.result to use (value -
expected).abs() < TOL instead of f64::EPSILON so the test tolerates one
ULP/serialization variance while still catching incorrect math or shape errors.

examples/structured_output_agent/main.rs (1)

209-238: 💤 Low value

Optional: surface stop_reason and treat parse failure as a demo signal.

final_answer is only schema-constrained when the agent reaches StopReason::FinalAnswer. If the loop hits MaxTurns or the provider doesn't honor the schema, the deserialization will silently fall through to the Err branch and the example exits Ok(()) — easy to miss for someone running this as a smoke test of structured outputs. Consider gating the parse on the stop reason and/or returning an error on parse failure so the demo loudly distinguishes "model ignored schema" from "demo succeeded."

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@examples/structured_output_agent/main.rs` around lines 209 - 238, The example
currently always attempts to parse out.final_answer into WeatherReport and
treats parse failure as a non-fatal demo message; instead first gate parsing on
out.stop_reason (compare against StopReason::FinalAnswer) and if the stop reason
is not FinalAnswer, print/emit a clear warning including out.stop_reason and
return an Err to fail the demo; if stop reason is FinalAnswer then attempt
serde_json::from_str::<WeatherReport>(&out.final_answer) and on parse errors
return an Err (with context about final_answer and the deserialize error) rather
than silently exiting Ok, using the AgentOutput variable name out and the
WeatherReport/StopReason symbols to locate the logic.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@examples/structured_output_agent/main.rs`:
- Around line 209-238: The example currently always attempts to parse
out.final_answer into WeatherReport and treats parse failure as a non-fatal demo
message; instead first gate parsing on out.stop_reason (compare against
StopReason::FinalAnswer) and if the stop reason is not FinalAnswer, print/emit a
clear warning including out.stop_reason and return an Err to fail the demo; if
stop reason is FinalAnswer then attempt
serde_json::from_str::<WeatherReport>(&out.final_answer) and on parse errors
return an Err (with context about final_answer and the deserialize error) rather
than silently exiting Ok, using the AgentOutput variable name out and the
WeatherReport/StopReason symbols to locate the logic.

In `@tests/agent_workflow.rs`:
- Around line 200-216: The assertions using f64::EPSILON are too strict for
values ~15–20; replace the epsilon with a looser tolerance (e.g., const TOL: f64
= 1e-9) and update the three checks that compare parsed.left_operand,
parsed.right_operand, and parsed.result to use (value - expected).abs() < TOL
instead of f64::EPSILON so the test tolerates one ULP/serialization variance
while still catching incorrect math or shape errors.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ff315663-b349-417d-9e1a-4216029d5b01

📥 Commits

Reviewing files that changed from the base of the PR and between f76b384 and 7cda31c.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (11)

• Cargo.toml
• examples/interactive_math_agent/main.rs
• examples/pipelined_math_agent/main.rs
• examples/simple_math_agent/main.rs
• examples/structured_output_agent/main.rs
• src/activities.rs
• src/llm.rs
• src/prelude.rs
• src/state.rs
• src/workflow.rs
• tests/agent_workflow.rs