Skip to content

Commit fe1609f

Browse files
kyegomezkyegomez
authored
Merge pull request #1501 from Steve-Dusty/feat/planner-generator-evaluator
feat: implement PlannerGeneratorEvaluator multi-agent harness
2 parents 0b683f5 + 78d91a0 commit fe1609f

8 files changed

Lines changed: 1873 additions & 0 deletions

File tree

‎docs/mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -282,6 +282,7 @@ nav:
282282

283283

284284

285+
- PlannerGeneratorEvaluator: "swarms/structs/planner_generator_evaluator.md"
285286
- DebateWithJudge: "swarms/structs/debate_with_judge.md"
286287
- MajorityVoting: "swarms/structs/majorityvoting.md"
287288
- MAKER: "swarms/structs/maker.md"

‎docs/swarms/structs/planner_generator_evaluator.md‎

Lines changed: 293 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,293 @@
1+
# `PlannerGeneratorEvaluator`
2+
3+
The `PlannerGeneratorEvaluator` is a domain-agnostic three-agent orchestration harness inspired by the GAN-style architecture described in [Anthropic's harness design research](https://www.anthropic.com/engineering/harness-design-long-running-apps). It coordinates long-running autonomous tasks from a short natural-language prompt, using an iterative generate-evaluate feedback loop to converge on high-quality output across any domain.
4+
5+
All three agents communicate through a single shared file on disk.
6+
7+
```mermaid
8+
graph TD
9+
A[User Prompt] --> B[Planner]
10+
B --> C[Plan + Evaluation Criteria]
11+
C --> D{For Each Step}
12+
D --> E[Generator Proposes Contract]
13+
E --> F[Evaluator Reviews Contract]
14+
F --> G[Generator Executes Step]
15+
G --> H[Generator Self-Assessment]
16+
H --> I[Evaluator Scores Output]
17+
I --> J{All Criteria Pass?}
18+
J -->|Yes| D
19+
J -->|No| K{Retries Left?}
20+
K -->|Yes + Scores Improving| L[REFINE: Fix Issues]
21+
K -->|Yes + Scores Declining| M[PIVOT: New Approach]
22+
L --> G
23+
M --> G
24+
K -->|No| D
25+
D -->|Done| N[Final Output + Shared State File]
26+
```
27+
28+
The harness follows this workflow:
29+
30+
1. **Planning**: Planner expands a short prompt into an ambitious plan with steps and evaluation criteria
31+
2. **Contract Negotiation**: Generator proposes what "done" looks like for each step; Evaluator reviews
32+
3. **Execution**: Generator produces concrete output and self-assesses before handoff
33+
4. **Evaluation**: Evaluator scores output per-criterion with hard thresholds — any criterion below its threshold fails the step
34+
5. **Feedback Loop**: On failure, Generator receives scores + trajectory signal (refine or pivot) and retries
35+
6. **All state on disk**: The shared state file is the single append-only record of the entire run
36+
37+
38+
## Key Features
39+
40+
| Feature | Description |
41+
|---------|-------------|
42+
| **GAN-Style Separation** | Distinct Generator and Evaluator agents prevent self-evaluation bias |
43+
| **Step Contracts** | Generator and Evaluator agree on success criteria before execution |
44+
| **Hard Threshold Enforcement** | Any single criterion below its threshold fails the step |
45+
| **Score Trajectory** | Tracks score trends across retries — signals Generator to refine or pivot |
46+
| **Self-Assessment** | Generator self-evaluates before Evaluator handoff |
47+
| **Shared State File** | Single append-only `.md` file for all inter-agent communication |
48+
| **Domain-Agnostic** | Planner defines evaluation criteria tailored to the task domain |
49+
| **Custom Agents** | Pass pre-configured agents with tools, MCP, or any Agent settings |
50+
| **Configurable Thresholds** | Default thresholds plus Planner-defined per-criterion thresholds |
51+
52+
53+
## Constructor
54+
55+
### `PlannerGeneratorEvaluator.__init__()`
56+
57+
| Parameter | Type | Default | Required | Description |
58+
|-----------|------|---------|----------|-------------|
59+
| `model_name` | `str` | `"gpt-4.1"` | No | Model identifier for all three agents |
60+
| `planner_model_name` | `str` | `None` | No | Override model for the Planner |
61+
| `generator_model_name` | `str` | `None` | No | Override model for the Generator |
62+
| `evaluator_model_name` | `str` | `None` | No | Override model for the Evaluator |
63+
| `max_steps` | `int` | `10` | No | Upper bound on plan steps to execute |
64+
| `max_retries_per_step` | `int` | `3` | No | Max evaluation failures before advancing |
65+
| `working_directory` | `str` | `os.getcwd()` | No | Directory where output is produced |
66+
| `shared_state_path` | `str` | `None` | No | Path for the shared state file (auto-generated if None) |
67+
| `default_thresholds` | `Dict[str, float]` | `{}` | No | Fallback score thresholds by criterion name |
68+
| `output_type` | `OutputType` | `"dict"` | No | Format for output (dict, str, list, final, json, yaml) |
69+
| `verbose` | `bool` | `False` | No | Enable verbose logging |
70+
| `planner_agent` | `Agent` | `None` | No | Pre-configured Agent for planning |
71+
| `generator_agent` | `Agent` | `None` | No | Pre-configured Agent for generation (e.g., with file/code tools) |
72+
| `evaluator_agent` | `Agent` | `None` | No | Pre-configured Agent for evaluation (e.g., with Playwright MCP) |
73+
74+
#### Raises
75+
76+
| Exception | Condition |
77+
|-----------|-----------|
78+
| `ValueError` | If `max_steps < 1`, `max_retries_per_step < 0`, or `model_name` is empty |
79+
80+
---
81+
82+
## Core Methods
83+
84+
### `run()`
85+
86+
Execute the full PGE harness pipeline from a short prompt to completed output.
87+
88+
| Parameter | Type | Default | Required | Description |
89+
|-----------|------|---------|----------|-------------|
90+
| `task` | `str` | — | **Yes** | A short natural-language description of the desired task |
91+
92+
#### Returns
93+
94+
| Type | Description |
95+
|------|-------------|
96+
| `Any` | Formatted conversation history according to `output_type` |
97+
98+
After `run()` completes, access `harness.last_result` for structured metadata:
99+
100+
| Field | Type | Description |
101+
|-------|------|-------------|
102+
| `output_path` | `str` | Path to the shared state file |
103+
| `plan` | `str` | The generated plan text |
104+
| `step_logs` | `List[Dict]` | Per-step metadata (contract, scores, retries) |
105+
| `total_duration` | `float` | Wall-clock time in seconds |
106+
| `total_steps_completed` | `int` | Number of steps that passed evaluation |
107+
| `total_retries` | `int` | Total retry attempts across all steps |
108+
109+
---
110+
111+
### `batched_run()`
112+
113+
Run the harness on multiple tasks sequentially.
114+
115+
| Parameter | Type | Default | Required | Description |
116+
|-----------|------|---------|----------|-------------|
117+
| `tasks` | `List[str]` | — | **Yes** | List of task prompts to process |
118+
119+
#### Returns
120+
121+
| Type | Description |
122+
|------|-------------|
123+
| `List[Any]` | List of results, one per task |
124+
125+
---
126+
127+
## Usage Examples
128+
129+
### Basic Usage
130+
131+
```python
132+
from swarms import PlannerGeneratorEvaluator
133+
134+
harness = PlannerGeneratorEvaluator(
135+
model_name="gpt-4.1",
136+
max_steps=3,
137+
max_retries_per_step=2,
138+
output_type="final",
139+
verbose=True,
140+
)
141+
142+
result = harness.run(
143+
"Write a comprehensive guide on the benefits and risks of intermittent fasting"
144+
)
145+
146+
print(result)
147+
print(f"Steps completed: {harness.last_result.total_steps_completed}")
148+
print(f"Duration: {harness.last_result.total_duration:.1f}s")
149+
```
150+
151+
### Custom Agents with Tools
152+
153+
Pass pre-configured agents with tools so the Generator can write files and the Evaluator can verify them on disk:
154+
155+
```python
156+
from swarms import Agent, PlannerGeneratorEvaluator
157+
158+
159+
def write_file(filename: str, content: str) -> str:
160+
"""Write content to a file."""
161+
with open(filename, "w") as f:
162+
f.write(content)
163+
return f"Written: {filename}"
164+
165+
166+
def read_file(filename: str) -> str:
167+
"""Read content from a file."""
168+
with open(filename, "r") as f:
169+
return f.read()
170+
171+
172+
generator = Agent(
173+
agent_name="PGE-Generator",
174+
model_name="gpt-4.1",
175+
max_loops=1,
176+
tools=[write_file],
177+
)
178+
179+
evaluator = Agent(
180+
agent_name="PGE-Evaluator",
181+
model_name="gpt-4.1",
182+
max_loops=1,
183+
tools=[read_file],
184+
)
185+
186+
harness = PlannerGeneratorEvaluator(
187+
model_name="gpt-4.1",
188+
generator_agent=generator,
189+
evaluator_agent=evaluator,
190+
max_steps=3,
191+
)
192+
193+
result = harness.run("Create a Python module for string manipulation utilities")
194+
```
195+
196+
### Evaluator with Playwright MCP (Web App Testing)
197+
198+
For web application development, give the Evaluator browser automation via Playwright MCP so it can test the running app like a real user:
199+
200+
```python
201+
from swarms import Agent, PlannerGeneratorEvaluator
202+
203+
evaluator = Agent(
204+
agent_name="PGE-Evaluator",
205+
model_name="gpt-4.1",
206+
max_loops=1,
207+
mcp_config={"url": "http://localhost:3000/playwright"},
208+
)
209+
210+
harness = PlannerGeneratorEvaluator(
211+
model_name="gpt-4.1",
212+
evaluator_agent=evaluator,
213+
max_steps=5,
214+
max_retries_per_step=3,
215+
)
216+
217+
result = harness.run("Build a todo app with React frontend and FastAPI backend")
218+
```
219+
220+
### Custom Thresholds
221+
222+
Provide default score thresholds that apply when the Planner doesn't define them:
223+
224+
```python
225+
from swarms import PlannerGeneratorEvaluator
226+
227+
harness = PlannerGeneratorEvaluator(
228+
model_name="gpt-4.1",
229+
default_thresholds={
230+
"accuracy": 8.0,
231+
"clarity": 7.0,
232+
"completeness": 7.0,
233+
},
234+
max_retries_per_step=4,
235+
)
236+
237+
result = harness.run("Write a technical specification for a rate-limiting middleware")
238+
```
239+
240+
---
241+
242+
## Architecture Details
243+
244+
### Shared State File
245+
246+
All inter-agent communication flows through a single append-only markdown file. Each section is timestamped and labeled:
247+
248+
```
249+
# PGE Harness Shared State
250+
251+
## User Prompt
252+
[original prompt]
253+
254+
---
255+
### [PLANNER OUTPUT] (2026-03-25 10:30:00)
256+
[plan with steps and evaluation criteria]
257+
258+
---
259+
### [STEP 1 CONTRACT PROPOSAL] (2026-03-25 10:30:15)
260+
[Generator's proposed contract]
261+
262+
---
263+
### [STEP 1 CONTRACT REVIEW] (2026-03-25 10:30:25)
264+
[Evaluator's review — APPROVED or AMENDMENTS REQUIRED]
265+
266+
---
267+
### [STEP 1 WORK LOG] (2026-03-25 10:30:45)
268+
[Generator's output + self-assessment]
269+
270+
---
271+
### [STEP 1 EVALUATION] (2026-03-25 10:31:00)
272+
[Evaluator's per-criterion scores, findings, and feedback]
273+
```
274+
275+
### Refine vs. Pivot
276+
277+
When a step fails evaluation, the harness computes a score trajectory across retries:
278+
279+
- **Scores improving** → REFINE: keep the current direction, fix specific issues
280+
- **Scores declining or stagnant** → PIVOT: take a fundamentally different approach
281+
282+
This signal is passed to the Generator alongside the Evaluator's feedback.
283+
284+
### Evaluation Criteria
285+
286+
The Planner defines domain-appropriate criteria as part of the plan. Each criterion has:
287+
288+
| Field | Description |
289+
|-------|-------------|
290+
| **Name** | Short label (e.g., "accuracy", "clarity") |
291+
| **Weight** | Relative importance (high, standard, low) |
292+
| **Description** | What it measures and what good/bad looks like |
293+
| **Threshold** | Minimum passing score (1-10). Any criterion below threshold = step fails |

‎examples/multi_agent/planner_generator_evaluator/pge_example.py‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
"""
2+
Planner-Generator-Evaluator Harness — Basic Example
3+
4+
Demonstrates the PGE harness with default agents. The Planner expands
5+
the prompt into a plan, the Generator executes each step, and the
6+
Evaluator scores output against criteria in an iterative feedback loop.
7+
8+
To run:
9+
python examples/multi_agent/planner_generator_evaluator/pge_example.py
10+
"""
11+
12+
from swarms import PlannerGeneratorEvaluator
13+
14+
if __name__ == "__main__":
15+
harness = PlannerGeneratorEvaluator(
16+
model_name="gpt-4.1",
17+
max_steps=3,
18+
max_retries_per_step=2,
19+
output_type="final",
20+
verbose=True,
21+
)
22+
23+
result = harness.run(
24+
"Write a comprehensive guide on the benefits and risks of intermittent fasting"
25+
)
26+
27+
print(result)
28+
29+
print(f"\nSteps completed: {harness.last_result.total_steps_completed}")
30+
print(f"Total retries: {harness.last_result.total_retries}")
31+
print(f"Duration: {harness.last_result.total_duration:.1f}s")
32+
print(f"Shared state file: {harness.last_result.output_path}")

0 commit comments

Comments
 (0)