feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop

[ryaneggz]

Summary

Implement a first-class evaluator subsystem for Orchestra that separates generation from evaluation, inspired by Anthropic's harness design research. Agents are reliably bad at self-evaluation — a dedicated evaluator running in a separate context window with a skeptical posture produces dramatically better quality control.

Key Features

• EvaluatorConfig on Assistant schema — attach an evaluator assistant with grading criteria to any generator assistant
• Structured grading criteria — named dimensions (e.g., accuracy, completeness), weights, and per-criterion pass/fail thresholds
• Generate → evaluate → critique → regenerate loop — automated feedback loop in LLMController with configurable max iterations
• Separate context window — evaluator cannot see generator's conversation history, preventing self-evaluation bias
• Default skeptical evaluator prompt — battle-tested system prompt tuned for critical assessment
• Playwright MCP integration — evaluators can interact with live UIs before scoring (optional)
• Frontend display — evaluation scores, pass/fail indicators, iteration badges on chat messages
• Configurable fail actions — retry, flag, or stop when evaluation fails

Spec

Full specification: .claude/specs/evaluator-agent.md

Implementation plan: .claude/plans/evaluator-agent.md

Implementation Phases

1. Schema & Data Layer — EvaluationCriterion, EvaluatorConfig, EvaluationResult models + Alembic migration
2. Orchestration Loop — generate-evaluate-critique-regenerate in LLMController
3. Default Evaluator Prompt — skeptical-posture system prompt with calibration examples
4. Frontend Display — EvaluationBadge.tsx with scores, pass/fail indicators, iteration badges
5. Testing & Documentation — unit/integration tests + example notebook

File Changes

• backend/src/schemas/entities/evaluation.py — new
• backend/src/schemas/entities/llm.py — add evaluator field
• backend/src/controllers/llm.py — orchestration loop
• backend/src/static/prompts/md/evaluator.md — new
• backend/migrations/versions/xxx_add_evaluator.py — new
• frontend/src/components/chat/EvaluationBadge.tsx — new
• examples/agents/evaluator_example.ipynb — new
• Tests in backend/tests/unit/ and backend/tests/integration/

Risks

• Evaluator leniency: LLMs are naturally generous. Mitigation: strong skepticism prompting + few-shot calibration.
• Cost multiplication: each iteration doubles token spend. Mitigation: configurable max iterations (default 3), budget cap.
• Model mismatch: weaker evaluator than generator yields unreliable scores. Mitigation: document same-tier recommendation.

[ryaneggz]

Playwright MCP Feasibility Analysis

Research-only assessment of integrating Playwright MCP with the Evaluator Agent feature.

---

1. Current State of Playwright/Browser in Orchestra

Playwright MCP is NOT integrated as a tool in Orchestra today. The README files (README.md:211, backend/README.md:173) document running @playwright/mcp locally as a developer tool via ngrok, but it is not wired into the agent runtime.

What does exist for browser automation:

Component	Location	Role
agent-browser CLI	Installed globally in the sandbox Dockerfile (sandboxes/ubuntu/Dockerfile:21-22)	Playwright/Chromium wrapper usable via execute tool in the exec_server
agent-browser skill	.claude/skills/agent-browser/SKILL.md	Claude Code skill for dev workflows, not an agent tool
Playwright Chromium deps	sandboxes/ubuntu/Dockerfile:22 (npx playwright install-deps chromium)	System libraries for headless Chromium in sandbox container

The sandbox container (exec_server) already has Chromium + Playwright system deps installed. An agent using the MCP sandbox can already run agent-browser open <url> via the execute tool, but this is shell-level, not structured MCP tool-level.

---

2. How MCP Tools Are Wired into Agent Graphs

The MCP integration path is mature and well-defined:

1. Assistant.mcp — Optional[dict] on the Assistant schema (backend/src/schemas/entities/llm.py:107) holds MCP server configs.
2. init_tools() — In backend/src/agents/__init__.py:162-190, when mcp is present, it creates a MultiServerMCPClient(mcp) and appends the discovered tools to the agent's tool list.
3. init_subagents() — In backend/src/agents/__init__.py:193-209, each subagent gets its own init_tools() call with its own subagent.mcp dict, meaning subagents have fully independent MCP tool sets.
4. ToolService.mcp_tools() — In backend/src/services/tool.py:61-84, MCP servers are filtered by enabled flag and cleaned before passing to MultiServerMCPClient.

Key finding: The architecture already supports giving a subagent its own MCP servers independently from the generator. An evaluator subagent can have Playwright MCP configured while the generator does not.

---

3. Can an Evaluator Subagent Use MCP Tools Independently?

Yes. The init_subagents() function at backend/src/agents/__init__.py:203 passes each subagent's mcp dict individually:

"tools": await init_tools(subagent.tools, subagent.a2a, subagent.mcp, service_context)

The evaluator, as designed in the spec (separate context window, separate assistant), would have its own Assistant record with its own mcp config. The EvaluatorConfig.assistant_id references a distinct assistant, which naturally gets its own MCP connections.

However, the spec's orchestration loop in LLMController invokes the evaluator as a separate agent call — not as a subagent within the same graph. This means the evaluator's MCP would be resolved via construct_agent() independently, which is correct for isolation.

---

4. Infrastructure for Running a Browser in Backend

Environment	Browser Support	Notes
exec_server sandbox (Docker)	Chromium installed, agent-browser CLI available	1 CPU / 1GB RAM limit in docker-compose.services.yml:57. Headless only.
Daytona sandbox	Depends on image	Not guaranteed to have Chromium
StateBackend (local)	Host system dependent	No isolation; security concern for evaluator browsing
Playwright MCP as external server	Full Playwright server via @playwright/mcp	Runs separately, exposed via streamable_http transport

Two viable approaches for evaluator browser access:

Option A — Playwright MCP as external MCP server: Run npx @playwright/mcp@latest --port 8931 as a sidecar service (or add to docker-compose.services.yml). Configure the evaluator assistant's mcp field to point to it. This gives the evaluator structured Playwright tools (browser_navigate, browser_click, browser_screenshot, etc.) without shell-level indirection.

Option B — agent-browser via exec_server: The evaluator uses the existing MCP sandbox's execute tool to run agent-browser CLI commands. Already works today, but is less structured (string parsing of CLI output vs. structured MCP tool responses).

---

5. Should Playwright MCP Be Phase 2?

Recommendation: Yes, defer to a follow-up phase. Rationale:

• The core evaluator value proposition (separate context, skeptical grading, critique loop) works entirely without browser access. The spec already marks it as "optional."
• Playwright MCP adds a new infrastructure dependency (sidecar service, Chromium runtime, increased resource requirements).
• The agent-browser via execute already provides a stopgap for browser-based evaluation in the sandbox.
• The MCP wiring for it is trivial once the evaluator assistant exists — it is just a JSON config on the assistant's mcp field, e.g.:

  {
    "playwright": {
      "transport": "streamable_http",
      "url": "http://playwright-mcp:8931/mcp",
      "headers": {}
    }
  }

• Separating it reduces the initial PR scope and allows the evaluator loop to be validated independently.

Suggested phasing:

• Phase 1-4 (from spec): Ship evaluator without Playwright MCP
• Phase 5 (new): Add playwright-mcp service to docker-compose.services.yml, document configuration, add evaluator prompt instructions for UI testing

---

6. Resource & Security Implications

Concern	Assessment	Mitigation
Memory/CPU	Chromium headless requires ~200-500MB RAM per instance. Current sandbox limit is 1GB, so single concurrent browser session is feasible.	Increase exec_server memory limit to 2GB if Playwright MCP is a sidecar; or set explicit browser pool limit.
Evaluator browsing live UIs	If evaluator can navigate arbitrary URLs, it could access internal services, cloud metadata endpoints, or be used for SSRF.	Restrict Playwright MCP to allowlisted URLs (the evaluated app only). The @playwright/mcp server supports --allowed-origins.
Token cost	Screenshots in vision mode add ~1000-2000 tokens per screenshot. Multiple screenshots per evaluation round significantly increases cost.	Default to text-based snapshot mode (agent-browser snapshot) over screenshots. Make vision mode opt-in per criterion.
Timeout	Browser interactions are slow (page loads, animations). Current exec timeout is 120s.	Evaluator should have extended timeout or async evaluation pattern.
Container isolation	Browser runs in sandbox container (good). But Playwright MCP as a sidecar shares the Docker network with backend.	Run Playwright MCP on the services network (same as exec_server), not the backend network. Use profiles: ["tools"] for opt-in.
Concurrent evaluations	Multiple users triggering evaluations simultaneously would spawn multiple browser instances.	Single Playwright MCP server can handle concurrent sessions, but resource limits apply. Consider per-user session pooling.

---

Summary

Question	Answer
Is Playwright MCP already available as a tool?	No. Chromium + agent-browser CLI exist in the sandbox image, but no structured Playwright MCP integration.
How do MCP tools get wired into agent graphs?	Via Assistant.mcp dict -> MultiServerMCPClient -> tools appended to agent. Mature, well-tested path.
Can evaluator use MCP independently from generator?	Yes. Subagents/separate assistants get their own mcp config and tool resolution.
Infrastructure for browser in backend?	Sandbox has Chromium. Playwright MCP would need a new sidecar service.
Should Playwright MCP be phase 2?	Yes. The evaluator loop delivers value without it; browser integration is additive and increases scope/risk.
Resource/security implications?	Manageable with URL allowlisting, memory limit increases, and opt-in activation via Docker compose profiles.

[ryaneggz]

Frontend Display Analysis for Evaluator Agent

1. Current Message Rendering Architecture

The chat message pipeline is:

• ChatMessages (frontend/src/components/lists/ChatMessages.tsx) — virtualized list via @tanstack/react-virtual that renders <Message> components per message.
• Message (same file, memoized) — dispatches on message.type:
  • human/user — right-aligned bubble with edit support.
  • tool_input — renders DefaultTool with JSON viewer.
  • tool — wraps in ToolTimeline (vertical dot timeline with expand/collapse).
  • ai/default — renders MarkdownCard with optional SubagentBadge for sub-agent attribution.
• Streaming: useChat.ts receives SSE events (messages, values, metadata, error, aborted, done) and feeds them through StreamMessageHandler, which appends/updates messages in the in-memory array. The loadingMessage string (e.g., "Calling search_engine tool...") shows below the message list with a Loader2 spinner.

Key takeaway: there is no concept of message metadata beyond agent_name, model, tool_call_id, and status today. Evaluation results would need to be attached as a new field on the message object (e.g., evaluation or evaluation_results).

---

2. Reusable Component Patterns

Component	Pattern	Reuse for Evaluator
SubagentBadge (components/badges/SubagentBadge.tsx)	Inline <span> with icon + text, text-xs, rounded-md	Direct pattern for an iteration badge ("Iteration 2/3")
Badge (components/ui/badge.tsx)	CVA-based with default, secondary, destructive, outline variants	Use default (green) for pass, destructive (red) for fail
ToolTimeline + ToolTimelineItem (components/timeline/)	Vertical dot timeline with green/red status dots, collapsible content panels	Excellent pattern for showing evaluation iterations as a timeline
Accordion (components/ui/accordion.tsx)	Radix-based expand/collapse with chevron animation	Used by TodoList for collapsible sections — good for hiding evaluator critique by default
Collapsible (components/ui/collapsible.tsx)	Radix primitive, lighter than accordion	Alternative for inline critique toggle
Tabs (components/ui/tabs.tsx)	Radix tab list with active state styling	Could be used for "Output" vs "Evaluation" tab on a message, but may be too heavy
Card (components/ui/card.tsx)	Standard card with header/content/footer sections	Container for a summary evaluation card
TodoList (components/lists/TodoList.tsx)	Accordion with status icons (pending/in-progress/completed), statusConfig map	Near-identical pattern for per-criterion score rows with pass/fail icons
ChatUtilityRow (components/chat/ChatUtilityRow.tsx)	Rounded pill buttons with icon + label + count, popover on click	Pattern for a compact "Eval: 4/5 Pass" pill in the utility row
ThreadSandboxStatus (components/status/ThreadSandboxStatus.tsx)	Popover selector with health dot indicator (green/red circle)	Green/red dot pattern for pass/fail per criterion

No existing Progress bar component exists in components/ui/. If per-criterion score bars are desired, a new Progress component (shadcn pattern) would need to be added.

---

3. Recommended Display Approach

A. Per-Message Evaluation Summary (Inline)

On AI messages that have evaluation results, render below the message content (above the model name / tok/s line):

[Pass Badge] Evaluation: 3/3 criteria passed (Iteration 2/3) [v Expand]

• Use Badge variant=default (green) for pass, destructive for fail.
• Use SubagentBadge-style iteration count: "Iteration 2/3".
• Collapsible detail section (via Collapsible or Accordion) showing per-criterion breakdown.

B. Per-Criterion Detail (Collapsible)

When expanded, show a TodoList-like layout:

[CheckCircle2 green] accuracy      0.85 / 0.60 threshold
[CheckCircle2 green] completeness  0.72 / 0.60 threshold
[XCircle red]        clarity       0.45 / 0.60 threshold  <-- FAILED

• Reuse statusConfig pattern from TodoList with CheckCircle2 (green) and XCircle (red).
• Show score as a simple fraction or inline bar (no Progress component needed initially — a div with bg-green-500 width percentage works).

C. Evaluator Critique (Collapsed by Default)

Below the criterion rows, show the evaluator's critique text in a MarkdownCard inside a Collapsible:

[v] Show evaluator feedback
  "The response lacks specific examples for the third requirement..."

This reuses MarkdownCard directly, which already handles all markdown rendering.

D. Iteration Display in Thread

Recommended: show only the final output to the user, with iteration metadata. Intermediate iterations should NOT appear as separate messages in the thread — this would clutter the conversation. Instead:

• The spec says "Buffer intermediate iterations; only stream final accepted output to user" (from the risk mitigations).
• The iteration badge on the final message tells the user how many rounds occurred.
• Expanding the evaluation section shows the full critique history if desired.

Alternative considered (rejected): Showing all iterations as separate collapsible messages would require a message grouping mechanism that does not exist today. The virtualizer renders flat messages, and adding nesting would require significant refactoring.

---

4. Gaps in the Spec

4.1 Real-Time Evaluation Progress (Streaming State)

Gap: The spec does not account for real-time feedback during evaluation iterations. The existing loadingMessage mechanism (e.g., "Calling search_engine tool...") could be extended to show "Evaluating response (iteration 1/3)..." but the spec does not define:

• What SSE event type carries evaluation progress (new evaluation event type needed?)
• Whether the user should see intermediate scores streaming in real-time or only the final result.

Recommendation: Add a new SSE event type evaluation_progress with { iteration: number, max_iterations: number, status: "evaluating" | "regenerating" | "passed" | "failed" } that drives the loadingMessage and a future progress indicator.

4.2 Evaluation Data Shape on Messages

Gap: The spec defines EvaluationResult as a backend Pydantic model but does not specify how it arrives at the frontend. Options:

• Attached to the values event as message metadata (like files and todos today).
• Attached directly on the AI message object in the messages event.
• Sent as a separate event type.

Recommendation: Attach on the AI message object as message.evaluation_results: EvaluationResult[] (array for all iterations). This is the simplest path — the Message component already receives the full message object and can check for this field.

4.3 Should Users See the Critique?

Recommendation: Yes, but collapsed by default. The critique text is valuable for:

• Debugging why an agent's response went through multiple iterations.
• Understanding quality gates for transparency.
• Power users tuning their evaluator criteria.

Users who do not care can simply ignore the collapsed section.

4.4 Max Iterations Reached Without Pass

Gap: When max_iterations is reached and the output still fails, the spec's fail_action: "flag" case needs a clear visual treatment.
Recommendation: Show a yellow/amber warning badge: "Evaluation: 2/3 criteria passed (max iterations reached)" with variant="outline" badge + amber border, distinct from both pass (green) and hard-fail (red, for fail_action: "stop").

4.5 No Evaluation Config on Message Display

When an assistant has no evaluator configured, the Message component should render exactly as it does today — zero visual changes. The evaluation UI should be purely additive.

---

5. Proposed New Component

frontend/src/components/chat/EvaluationBadge.tsx — a self-contained component that:

1. Accepts evaluationResults: EvaluationResult[] prop.
2. Renders nothing if evaluationResults is null/undefined/empty.
3. Shows a compact inline summary badge (pass/fail + iteration count).
4. Expands to show per-criterion scores and critique via Collapsible.
5. Uses existing Badge, Collapsible, MarkdownCard, and lucide icons.

This component would be imported in the Message component's AI message branch (line ~194-233 of ChatMessages.tsx), rendered between the MarkdownCard and the model/streaming-rate footer.

6. Integration Points

Layer	Change Needed
SSE Events (lib/entities/stream.ts)	Add evaluation_progress event type to SSEEventType union
useChat.ts (hooks/useChat.ts)	Handle new event in handleMessages — update loadingMessage with iteration info
StreamMessageHandler (lib/utils/message.ts)	Preserve evaluation_results field when processing AI message chunks
ChatMessages.tsx (components/lists/ChatMessages.tsx)	Pass message.evaluation_results to new EvaluationBadge component
No new shadcn components needed	Existing Badge, Collapsible, Accordion cover all needs

---

7. Summary of Recommendations

1. Inline summary badge on AI messages, collapsed by default — reuse Badge + SubagentBadge pattern.
2. Per-criterion rows in collapsible section — reuse TodoList status icon pattern.
3. Critique text in nested collapsible — reuse MarkdownCard.
4. Only show final output in thread; iteration count as metadata badge.
5. New SSE event type evaluation_progress for real-time "Evaluating..." loading state.
6. Evaluation data should be attached as message.evaluation_results array on the AI message object.
7. Amber/warning state for max-iterations-reached without full pass.
8. Zero visual changes for messages without evaluation — purely additive.

[ryaneggz]

Loop Architecture Analysis

After studying the codebase end-to-end — agents/__init__.py, controllers/llm.py, utils/stream.py, schemas/entities/llm.py, the deepagents library internals (graph.py, middleware/subagents.py, middleware system), the frontend streaming pipeline (stream.ts, fetchStreamReader.ts, useChat.ts, message.ts), and the spec on feat/evaluator-agent — here are the key architecture decisions and gaps.

---

1. Where should the evaluator loop live?

Recommendation: Post-generation step in LLMController / stream_generator, NOT a LangGraph subgraph or middleware.

Rationale:

• Not a middleware. Deepagents middleware intercepts individual LLM model calls (wrap_model_call) and tool calls (wrap_tool_call). The evaluator needs the complete generator output — all tool calls resolved, all text finalized — before it can evaluate. Middleware fires per-turn, not after the full agentic loop completes. You would have to fundamentally change when middleware runs.

• Not a LangGraph subgraph. Subagents in deepagents are invoked by the generator via the task tool. The evaluator must run after the generator finishes, outside the generator's control. Making the evaluator a subagraph would mean the generator evaluates itself (calling its own evaluator), which is exactly the anti-pattern the spec wants to avoid.

• Post-generation in the controller is the natural fit. Both llm_invoke() and stream_generator() already have a clear "generator is done" boundary — after agent.invoke() returns or after the async for chunk in agent.astream() loop completes. The evaluator loop wraps this boundary: while iteration < max_iterations: generate → evaluate → if passed: break → else: inject critique and re-generate.

• Precedent in codebase. The controller already handles retry logic (Daytona/MCP fallbacks reconstruct the agent and re-invoke). The evaluator loop is structurally similar — reconstruct input with critique feedback and re-invoke.

---

2. What does "separate context window" mean?

Recommendation: A separate, ephemeral Orchestra instance with its own graph — NOT a separate thread, NOT a shared graph.

The spec says "evaluator cannot see generator's conversation history." Concretely:

• Create a fresh Orchestra via construct_agent() using the evaluator's Assistant config (its own system prompt, model, tools). No checkpointer (ephemeral).
• Pass only the generator's final output + evaluation criteria as a single HumanMessage — the evaluator never sees the generator's multi-turn history.
• This is exactly how subagents work today: _validate_and_prepare_state() in middleware/subagents.py creates subagent_state["messages"] = [HumanMessage(content=description)] — fresh context window each time.
• The evaluator Orchestra instance is constructed once per evaluation loop entry and reused for each iteration's evaluation call. It gets no checkpointer, so no history leaks between iterations.

Implementation detail: resolve the evaluator's Assistant by evaluator_config.assistant_id using assistant_service.get(), then build its tools/prompt/model the same way llm_service.assistant() does.

---

3. Streaming during the evaluate-regenerate loop

This is the hardest design question. Three options:

Option A: Buffer intermediate, stream only final (recommended for v1).

• During the loop, run agent.invoke() (not astream()) for each generation attempt.
• Only after the evaluator passes (or max iterations reached), stream the final accepted output to the user.
• Send a custom SSE event at the start: ("evaluation", {"status": "evaluating", "iteration": N, "max": M}) so the frontend can show a progress indicator.
• Send final evaluation metadata: ("evaluation", {"status": "complete", "scores": {...}, "passed": true, "iterations": N}).
• Pros: Simplest. User doesn't see flickering/replaced content. Clean UX.
• Cons: Higher latency — user sees nothing until final output. For long generations this could be 30-60+ seconds of silence per iteration.

Option B: Stream each iteration, mark superseded.

• Stream each generation attempt normally (via astream()).
• After each stream completes, send evaluation results as a custom SSE event.
• If evaluation fails, send ("evaluation", {"status": "regenerating", "iteration": N}) and then stream the next attempt.
• Frontend replaces/collapses previous attempts and shows only the latest.
• Pros: User sees progress. Lower perceived latency.
• Cons: Frontend must handle message replacement/collapse. More complex. User might read a response that gets replaced.

Option C: Stream final only, but show evaluator reasoning.

• Like Option A but also stream the evaluator's assessment in real-time as a "thinking" step.
• Pros: User sees what the evaluator is thinking. Transparency.
• Cons: Evaluator output is not really useful to end users in most cases.

For the SSE transport: the existing handle_multi_mode() in stream.py already handles "custom" mode chunks as pass-through (if mode == "custom": return chunk). The frontend SSEEventType union in stream.ts would need a new "evaluation" type. This is a clean extension point.

---

4. Can the existing subagent pattern handle the evaluator?

No, and it should not. Key differences:

Aspect	Subagents	Evaluator
Invoked by	Generator (via task tool)	Controller (after generator completes)
Context	Receives delegated subtask	Receives generator output + criteria
Return path	ToolMessage back to generator	EvaluationResult to controller loop logic
Lifecycle	Inside generator's agentic loop	Outside and around the generator's loop
Streaming	Subgraph streaming via subgraphs=True	Separate stream or buffered

The evaluator is architecturally a peer of the generator, not a child. The controller orchestrates both.

---

5. How does the evaluator interact with tool calls the generator made?

The generator's final state (from agent.graph.aget_state(config)) contains the full messages list including all ToolMessage results and AIMessage tool calls. The evaluator should receive:

1. The final AI text response — what the user would see.
2. A summary of tool calls and their results — so the evaluator can judge whether the generator used tools correctly.
3. The original user query — so the evaluator can judge relevance.

This should be formatted into a structured prompt for the evaluator, not raw message history. Something like:

<user_query>...</user_query>
<generator_response>...</generator_response>
<tool_calls>
  <tool name="search_engine" status="success">
    <input>...</input>
    <output>...</output>
  </tool>
</tool_calls>
<evaluation_criteria>
  <criterion name="accuracy" weight="0.4" threshold="0.6">...</criterion>
  ...
</evaluation_criteria>

---

6. Key implementation gaps in the spec

Gap 1: Streaming strategy is unspecified. The spec says "stream to user" but doesn't address what happens during intermediate iterations. This is the most impactful UX decision. Recommend Option A (buffer intermediate) for v1.

Gap 2: Critique injection mechanism. When the evaluator fails a generation, how does the critique get back to the generator? Two options:

• Append as a new HumanMessage to the generator's thread: "The evaluator found these issues: {critique}. Please revise your response." Then re-invoke the same graph with the same config (checkpointer preserves history). This is clean and uses the existing conversation model.
• Replace the system prompt with critique context. Worse — pollutes the system prompt.

Recommend the HumanMessage injection approach. The generator's checkpointer preserves its conversation, so it can see its own previous attempt + the critique.

Gap 3: Cost/token accounting. Each evaluation iteration multiplies cost. The spec mentions max_iterations but not token budget caps. Consider adding max_evaluation_tokens to EvaluatorConfig.

Gap 4: Concurrent evaluation with streaming. stream_generator() is an async generator. The evaluation loop must wrap it. The cleanest approach: extract a helper _run_generator() that returns the final state (using invoke), and only call astream() for the final accepted output.

Gap 5: Database persistence of evaluation results. The spec mentions persisting on thread metadata. The existing _persist_final_state() in stream.py already writes arbitrary data to thread state (todos, files). Add evaluation_results as another field in the same pattern — no migration needed if using the existing JSONB metadata column.

Gap 6: Evaluator structured output. The evaluator must return a parseable EvaluationResult. Use response_format parameter on create_deep_agent() / create_agent() which is already supported. This forces the evaluator LLM to return structured JSON matching the schema.

---

7. Proposed control flow (pseudocode)

# In LLMController.llm_stream() or stream_generator()

async def generate_with_evaluation(generator_agent, evaluator_config, ...):
    evaluator_assistant = await assistant_service.get(evaluator_config.assistant_id)
    evaluator_agent = await construct_agent(evaluator_assistant, checkpointer=None)  # ephemeral
    
    for iteration in range(evaluator_config.max_iterations):
        # Generate (non-streaming for intermediate, streaming for final)
        if iteration == 0:
            result = await generator_agent.invoke(input, config=config)
        else:
            # Inject critique as HumanMessage
            critique_msg = HumanMessage(f"Evaluator feedback: {last_critique}. Revise.")
            result = await generator_agent.invoke({"messages": [critique_msg]}, config=config)
        
        # Evaluate in separate context
        eval_input = format_for_evaluation(result, criteria)
        eval_result = await evaluator_agent.invoke(eval_input)  # fresh context
        evaluation = parse_evaluation_result(eval_result)
        
        if evaluation.passed:
            break
        last_critique = evaluation.critique
    
    # Now stream the final accepted output (or re-stream from checkpoint)
    yield evaluation_metadata_event
    async for chunk in generator_agent.astream(final_input, ...):
        yield chunk

---

8. Recommended file changes (refined from spec)

File	Change
backend/src/schemas/entities/evaluation.py	New: EvaluationCriterion, EvaluatorConfig, EvaluationResult
backend/src/schemas/entities/llm.py	Add evaluator: Optional[EvaluatorConfig] to Assistant
backend/src/controllers/llm.py	Add _evaluate_loop() helper, integrate into llm_invoke()
backend/src/utils/stream.py	Add _evaluate_and_stream() wrapper in stream_generator(), handle "evaluation" custom events
backend/src/static/prompts/md/evaluator.md	New: default skeptical evaluator system prompt
frontend/src/lib/entities/stream.ts	Add EvaluationEvent type to SSEEventType union
frontend/src/components/chat/EvaluationBadge.tsx	New: badge component for scores/iterations
backend/tests/unit/schemas/test_evaluation.py	New: schema validation tests
backend/tests/integration/test_evaluator_loop.py	New: end-to-end loop tests

No Alembic migration needed if evaluation config is stored in the existing Assistant JSONB and evaluation results in thread metadata JSONB.

[ryaneggz]

Downstream Dependency: Sprint Contracts (#910) Requirements

Sprint Contracts (#910) is a downstream consumer of the Evaluator and introduces specific interface requirements that should be addressed during Evaluator implementation to avoid costly retrofitting.

What Sprint Contracts Needs from the Evaluator

1. Contract Grading Mode (Binary Pass/Fail per Criterion)

The Evaluator spec defines continuous 0-1 scoring via EvaluationCriterion(name, description, weight, threshold). Sprint Contracts requires a different mode: binary pass/fail grading against a list of TestCriterion(description, verification_method) objects, with reporting like "17/27 criteria passed".

Recommendation: Design EvaluationResult with a flexible criterion_results field from the start:

class CriterionResult(BaseModel):
    criterion_id: str           # Maps back to input criterion
    passed: bool
    score: Optional[float]      # For continuous mode; None for binary mode
    notes: str                  # Evaluator's reasoning

class EvaluationResult(BaseModel):
    scores: dict[str, float]            # Aggregate scores (existing)
    passed: bool                        # Overall pass/fail (existing)
    critique: str                       # Detailed feedback (existing)
    iteration: int                      # Loop iteration (existing)
    criterion_results: Optional[list[CriterionResult]] = None  # NEW: per-criterion breakdown

This supports both the evaluator's native weighted scoring AND sprint contract binary grading without schema changes later.

2. Structured Counter-Proposal Output (Not Just Critique)

Sprint Contracts requires the evaluator to counter-propose changes to the contract scope and criteria during negotiation. The current spec's EvaluationResult.critique is free-form text — insufficient for programmatic negotiation.

Recommendation: Add an optional counter_proposal field:

class EvaluationResult(BaseModel):
    # ... existing fields ...
    counter_proposal: Optional[dict] = None  # Structured changes for negotiation

Or define a separate NegotiationResponse schema that the evaluator can return when invoked in contract review mode.

3. Accept External Criteria as Input (Not Just Pre-Configured)

The current evaluator spec assumes criteria are pre-configured on EvaluatorConfig. Sprint Contracts needs to pass criteria dynamically at invocation time (the contract's test_criteria list). The evaluator should support both:

• Configured mode: Use EvaluatorConfig.criteria from the assistant definition
• Dynamic mode: Accept criteria as part of the invocation payload (for sprint contract grading)

Recommendation: The orchestration loop in LLMController should support passing criteria alongside the content to evaluate, not only reading from the assistant's config.

4. Multi-Round Orchestration Support

Sprint Contracts requires the evaluator to participate in a negotiation loop (propose → review → counter-propose → revise, up to N rounds). This is distinct from the evaluator's generate-evaluate-critique-regenerate loop.

The negotiation loop is a new orchestration pattern that doesn't exist in the codebase. The current subagent pattern (deepagents task tool) is fire-and-forget — a single HumanMessage in, single ToolMessage out. The negotiation loop must be orchestrated at the LLMController level.

Key finding from codebase analysis: Files in StateBackend ARE shared between parent and subagents (the _EXCLUDED_STATE_KEYS set in deepagents/middleware/subagents.py excludes messages but not files). This means the file-based communication channel (/contracts/sprint-N.md) proposed by Sprint Contracts is viable with the current architecture — the orchestration controller just needs to manage the round-trip loop.

Impact on Implementation Order

The evaluator's Phase 1 (Schema & Data Layer) and Phase 2 (Orchestration Loop) should be designed with these requirements in mind. Specifically:

1. Make EvaluationResult extensible with criterion_results from day one
2. Design the evaluator invocation interface to accept dynamic criteria, not just pre-configured ones
3. Ensure the orchestration loop in LLMController is structured as a composable primitive that Sprint Contracts can extend for negotiation

[ryaneggz]

Dependency Graph Validation

Position in Dependency Graph

Evaluator Agent (#899) — INDEPENDENT (Layer 0)
  └─ depended upon by: Sprint Contracts (#906), Harness Templates (#910)
  └─ soft dep from: Agent Observability (#904/#905), Social Media Campaign (#902), Cost Tracking (#905)

Declared Dependencies

• None declared — claims to be independent.

Validated Dependencies

Correct. Evaluator Agent is genuinely independent. It introduces:

• EvaluationCriterion, EvaluatorConfig, EvaluationResult schemas (new file evaluation.py)
• evaluator field on Assistant schema in llm.py
• Generate-evaluate-critique-regenerate loop in LLMController

Implicit Dependencies Found

1. Cost Tracking (feat: Per-run cost tracking, phase-level metering, and budget caps #905) — The spec's "Risks" section explicitly says "Cost multiplication: each iteration doubles token spend. Mitigation: configurable max iterations, budget cap." The "budget cap" reference implies integration with Cost Tracking. This is a soft implicit dependency — the evaluator should work without it, but the budget cap feature creates a cross-cutting concern. The spec should clarify: does max_iterations alone handle cost control, or is there a planned integration point with RunBudget from feat: Per-run cost tracking, phase-level metering, and budget caps #905?
2. Playwright MCP — The spec mentions "Playwright MCP integration so evaluators can interact with live UIs before scoring." This uses the existing MCP infrastructure (assistant.mcp field already on Assistant), so this is a satisfied implicit dependency.
3. LLMController modification — Both this spec and Planner (feat: Planner Agent — auto-expand short prompts into full product specs #898) modify LLMController.stream()/invoke() to add pre/post-execution phases. If built in parallel, there will be merge conflicts in backend/src/controllers/llm.py.

Interface Contracts Needed

1. EvaluatorConfig schema — should be finalized NOW. Sprint Contracts (feat: Context reset with structured handoff artifacts for long-running sessions #906) and Harness Templates (feat: Sprint contracts — agents negotiate scope and pass/fail criteria before execution #910) both import and extend it:

   class EvaluatorConfig(BaseModel):
       assistant_id: str
       criteria: list[EvaluationCriterion]
       max_iterations: int = 3
       fail_action: Literal["retry", "flag", "stop"] = "retry"

2. EvaluationResult schema — Sprint Contracts (feat: Context reset with structured handoff artifacts for long-running sessions #906) needs to extend this with per-criterion pass/fail. The contract between these specs is: evaluator returns EvaluationResult with scores: dict[str, float] keyed by criterion name. Sprint Contracts adds test_criteria grading on top.
3. Orchestration hook point — The exact insertion point in LLMController where the evaluate loop runs needs to be defined relative to the planner phase. Order should be: planner (first message only) → generator → evaluator loop.

Schema Conflict Risk

• High. Both this spec and Planner (feat: Planner Agent — auto-expand short prompts into full product specs #898) add fields to Assistant in llm.py AND modify LLMController. Alembic migration ordering must be coordinated. Recommendation: Build Planner first (simpler), then Evaluator, OR agree on a single migration that adds both fields.

Critical Path Assessment

On critical path. This is the single most depended-upon spec. Sprint Contracts (#906) has a hard dependency on it. Harness Templates (#910) cannot define its "Reviewed" or "Full Harness" templates without it. Should be Wave 1.

[ryaneggz]

Migration Strategy Note

Schema Changes Required

• EvaluatorConfig, EvaluationCriterion, EvaluationResult are new Pydantic models.
• EvaluatorConfig is added as an optional field on Assistant.
• EvaluationResult is attached as metadata to thread messages (or stored in thread metadata dict).

Migration Needs

• None for Assistant changes. Assistants are stored in LangGraph Store (schemaless JSON), so adding evaluator: Optional[EvaluatorConfig] = None requires no Alembic migration.
• None for evaluation results. Evaluation results can be persisted as thread metadata (also in LangGraph Store) or as structured messages. No new SQL table required for v1.
• If you later want to query evaluation results across threads (analytics), a dedicated evaluation_results SQL table could be added, but that is a v2 concern and not needed for the core feedback loop.

Parallel Development Safety

• No Alembic files touched, so zero migration conflict risk with any other spec.
• Will have a merge conflict in backend/src/schemas/entities/llm.py if feat: Planner Agent — auto-expand short prompts into full product specs #898 (planner) is also in flight, since both add optional fields to Assistant. Resolution is trivial (both are additive optional fields).

Shared Fields with Other Specs

• EvaluationCriterion is referenced by both this spec (feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899) and Harness Templates (feat: Harness Templates — reusable planner→generator→evaluator architectures #911) and Sprint Contracts (feat: Sprint contracts — agents negotiate scope and pass/fail criteria before execution #910). Define it in evaluation.py and import from there to avoid duplication.
• The evaluator's token counts and cost overlap with Cost Tracking (feat: Per-run cost tracking, phase-level metering, and budget caps #905) and Observability (feat: Agent phase observability — structured traces with frontend timeline #904). The evaluator should emit the same metrics events that those systems consume, but does not need its own storage.

Ordering Consideration

• This feature should land before Sprint Contracts (feat: Sprint contracts — agents negotiate scope and pass/fail criteria before execution #910) and Harness Templates (feat: Harness Templates — reusable planner→generator→evaluator architectures #911), which both depend on EvaluatorConfig existing. feat: Planner Agent — auto-expand short prompts into full product specs #898 (planner) is independent and can land in any order relative to this.

[ryaneggz]

Testing Strategy Recommendations

Infrastructure Context

The existing test suite provides excellent patterns for this feature. Key references:

• tests/unit/services/test_checkpoint_hitl.py — mock graph with aget_state/ainvoke, MockInterrupt pattern, IsolatedAsyncioTestCase
• tests/unit/utils/test_compacting.py — middleware testing with @patch("...init_chat_model") to mock LLM calls
• tests/unit/utils/test_middleware_integration.py — middleware stack ordering assertions
• conftest.py mock_external_services autouse fixture — prevents real API calls globally

Recommended Tests

Unit Tests (tests/unit/schemas/test_evaluation.py)

• EvaluationCriterion: weight defaults to 1.0, threshold defaults to 0.6, invalid threshold (>1 or <0) rejected
• EvaluatorConfig: max_iterations defaults to 3, fail_action defaults to "retry", only valid literal values accepted
• EvaluationResult: passed computed correctly from scores vs thresholds, handles missing criteria gracefully

Unit Tests (tests/unit/controllers/test_evaluator_loop.py) — CRITICAL

• Loop termination at max_iterations: mock generator + evaluator LLM calls, verify loop runs exactly max_iterations when evaluator always fails. Use AsyncMock with side_effect list for successive LLM responses
• Loop exits early on pass: evaluator returns passing scores on iteration 2 of 3 → loop stops at 2
• Critique feedback format: verify the critique from evaluator is injected as a new message to generator (check ainvoke call args)
• Separate context windows: verify evaluator does NOT receive generator's conversation history — only the output artifact + criteria
• fail_action="flag": loop completes without retry, metadata includes failure flag
• fail_action="stop": loop raises/returns error after first failed evaluation
• Zero criteria configured: graceful no-op or sensible default behavior

How to test the evaluate-regenerate loop WITHOUT expensive LLM calls:

# Pattern: mock init_chat_model to return AsyncMock with staged responses
@patch("src.controllers.llm.init_chat_model")
async def test_loop(mock_init):
    # Generator responses (each iteration)
    gen_mock = AsyncMock()
    gen_mock.ainvoke.side_effect = [
        AIMessage(content="Draft v1"),  # iteration 1
        AIMessage(content="Draft v2"),  # iteration 2 (after critique)
    ]
    # Evaluator responses
    eval_mock = AsyncMock()
    eval_mock.ainvoke.side_effect = [
        AIMessage(content='{"scores":{"accuracy":0.3},"passed":false,"critique":"Too vague"}'),
        AIMessage(content='{"scores":{"accuracy":0.8},"passed":true,"critique":"Good"}'),
    ]
    mock_init.side_effect = [gen_mock, eval_mock]
    # ... run controller, verify 2 iterations

Integration Test (tests/integration/test_evaluator_flow.py)

• Via async_client: POST /api/llm/stream with evaluator-configured assistant → verify SSE stream contains evaluation events (scores, iteration count)
• Verify evaluation results are persisted as thread metadata

Frontend Tests

• EvaluationBadge.tsx: renders pass/fail state, shows scores tooltip, iteration count badge
• Scores display: renders each criterion with name, score, threshold bar

Gaps in Spec's Test Section

• Missing: Test for evaluator producing malformed JSON (not matching EvaluationResult schema) — the controller needs graceful fallback
• Missing: Cost multiplication guard — test that a budget cap (if configured) can interrupt the loop mid-evaluation
• Missing: Test for evaluator + generator using different models — verify init_chat_model is called with correct model for each
• Missing: Concurrent evaluation — what happens if user sends another message while evaluate-regenerate loop is running?
• Missing: Test that evaluation metadata is included in SSE stream events for real-time frontend display

Shared Fixtures Needed

• mock_evaluation_criteria: reusable list of EvaluationCriterion objects with known thresholds
• mock_evaluation_result_pass / mock_evaluation_result_fail: pre-built JSON strings matching EvaluationResult schema
• evaluator_assistant_fixture: Assistant with evaluator: EvaluatorConfig pre-configured

[ryaneggz]

Downstream Dependency: Harness Templates (#911) Requirements

The Harness Templates feature (#911) bundles EvaluatorConfig and EvaluationCriterion into pre-built multi-agent architectures. After analyzing how templates will consume the evaluator schema, here are the requirements and gaps:

1. EvaluatorConfig Shape Requirements for Templates

Templates need to define evaluator configuration inline without referencing a pre-existing evaluator assistant. The current spec defines:

class EvaluatorConfig(BaseModel):
    assistant_id: str            # The evaluator assistant
    criteria: list[EvaluationCriterion]
    max_iterations: int = 3
    fail_action: Literal["retry", "flag", "stop"] = "retry"

Problem: assistant_id: str (required) creates a chicken-and-egg problem for templates. When a template is defined (especially built-in templates seeded at startup), no evaluator assistant exists yet. The evaluator assistant is only created when a user instantiates the template into a real assistant.

Recommended change: Make assistant_id optional and add inline evaluator config fields:

class EvaluatorConfig(BaseModel):
    assistant_id: Optional[str] = None   # Reference to existing evaluator assistant
    # Inline config (used when assistant_id is None — template mode)
    model: Optional[str] = None          # Model override for evaluator
    system_prompt: Optional[str] = None  # Custom evaluator prompt (defaults to skeptical)
    criteria: list[EvaluationCriterion]
    max_iterations: int = 3
    fail_action: Literal["retry", "flag", "stop"] = "retry"

This way, templates define evaluator behavior inline, and the template instantiation logic creates the evaluator assistant from these inline fields.

2. EvaluationCriterion description Field

Templates define different criteria sets per template:

• Reviewed: code quality criteria
• Full Harness: product depth + functionality + design criteria
• Frontend Design: design quality, originality, craft, functionality (with Playwright MCP)

The description: str field on EvaluationCriterion needs to be rich enough to include few-shot calibration examples (what 1/5 vs 5/5 looks like). This should be documented as supporting multi-line markdown, not just a short label.

3. Playwright MCP Integration Flag

The "Frontend Design" template requires the evaluator to use Playwright MCP for visual testing. The current EvaluatorConfig has no field to specify required MCP servers for the evaluator. Consider adding:

class EvaluatorConfig(BaseModel):
    ...
    mcp_servers: Optional[dict[str, dict]] = None  # MCP servers evaluator needs (e.g., Playwright)

4. What Templates Will Bundle from This Spec

Template	Evaluator?	Criteria	fail_action	max_iterations
Solo	No	(none)	N/A	N/A
Reviewed	Yes	code quality (correctness, readability, test coverage)	retry	3
Full Harness	Yes	product depth, functionality, design, completeness	retry	3
Frontend Design	Yes	design quality, originality, craft, functionality + Playwright	retry	3

5. Schema Stability Request

Since templates will snapshot EvaluatorConfig and EvaluationCriterion at seed time, any breaking changes to these schemas after templates ship will require a template migration. Please consider adding a schema_version to EvaluatorConfig or freezing the schema shape before #911 implementation begins.

Summary of Requested Changes

• Make assistant_id optional on EvaluatorConfig (support inline template mode)
• Add model and system_prompt optional fields for inline evaluator definition
• Consider mcp_servers field for evaluator MCP dependencies (Playwright)
• Document EvaluationCriterion.description as supporting multi-line markdown with calibration examples
• Consider schema_version for forward compatibility with template seeder upgrades

[ryaneggz]

SSE Streaming Design — Evaluation Progress

Context

The evaluator runs a generate-evaluate-critique-regenerate loop with configurable max iterations. The frontend needs real-time visibility into: which iteration is running, evaluation scores per criterion, pass/fail status, and whether the loop is retrying.

Recommended Approach

Use "custom" stream mode with "eval_progress" sub-type. This reuses the same extension mechanism recommended for cost tracking (#905).

Backend — emit evaluation progress events from the orchestration loop:

# In the evaluate-loop inside LLMController (or a dedicated orchestration node):
stream_writer(("custom", {
    "type": "eval_progress",
    "data": {
        "iteration": 2,
        "max_iterations": 3,
        "status": "evaluating",  # "generating" | "evaluating" | "critiquing" | "passed" | "failed"
        "scores": {
            "accuracy": {"score": 0.7, "weight": 0.4, "passed": False, "threshold": 0.8},
            "completeness": {"score": 0.9, "weight": 0.3, "passed": True, "threshold": 0.7},
            "clarity": {"score": 0.85, "weight": 0.3, "passed": True, "threshold": 0.75},
        },
        "weighted_total": 0.80,
        "overall_passed": False,
        "fail_action": "retry",  # "retry" | "flag" | "stop"
    }
}))

Event emission points in the loop:

1. status: "generating" — at the start of each generation pass (with iteration number)
2. status: "evaluating" — when the evaluator begins scoring
3. status: "critiquing" — when the evaluator is producing critique feedback
4. status: "passed" or status: "failed" — terminal states with final scores

Frontend consumption:

1. In useChat.ts, handle custom events where payload.type === "eval_progress"
2. Store in a evalState ref: { iteration, maxIterations, status, scores, overallPassed }
3. The EvaluationBadge.tsx component reads from this state to render:
   • Iteration counter badge (e.g., "Attempt 2/3")
   • Per-criterion score bars with pass/fail coloring
   • Overall status indicator

Why not use setLoadingMessage() for status?

setLoadingMessage is currently used for simple text status ("Processing request...", "Calling web_search tool..."). Evaluation progress is structured data (scores, iterations, pass/fail) that requires a dedicated UI component — not just a text string. The "custom" event carries structured payloads that the frontend can render as a rich component.

Distributed mode: Works automatically — _stream_chunks_to_redis() forwards all handle_multi_mode() output including custom events.