feat: Harness Templates — reusable planner→generator→evaluator architectures

[ryaneggz]

Summary

Harness Templates provide pre-built multi-agent architectures that encode proven orchestration patterns (planner → generator → evaluator). Instead of manually assembling agent configurations, users can select from built-in templates or fork and customize them.

Motivation

Orchestra has all the primitives (subagents, tools, MCP, A2A) but no pre-built orchestration patterns. Every user reinvents the wheel. Anthropic's research identifies specific multi-agent architectures that consistently outperform solo agents. These should be packaged as selectable templates.

Proposed Solution

• HarnessTemplate entity bundling planner + generator + evaluator configs
• Four built-in templates seeded on first run:
  • Solo — No planner, no evaluator. Best for quick tasks, Q&A, simple code.
  • Reviewed — Evaluator with code quality criteria. Best for code generation with quality gates.
  • Full Harness — Planner + evaluator with product depth criteria. Best for long-running app development.
  • Frontend Design — Planner + evaluator with design quality criteria (Playwright MCP). Best for UI/UX work.
• CRUD API at /api/v0/templates with fork support
• Template selector card grid in assistant creation flows
• Model recommendations and cost/time estimates per template

Key Files

• Spec: .claude/specs/harness-templates.md
• Plan: .claude/plans/harness-templates.md

Dependencies

• Requires evaluator-agent spec implementation
• Requires planner-agent spec implementation

Scope

Area	Changes
Backend schema	HarnessTemplate, PlannerConfig, EvaluatorConfig Pydantic models
Backend API	CRUD endpoints + fork support at /api/v0/templates
Seeder	backend/seeds/harness_templates.py for built-in templates
Frontend	TemplateCard, TemplateSelector components
Tests	Unit + integration for schema, API, seeder, and frontend

[ryaneggz]

Dependency Gap Analysis for Harness Templates

After analyzing the current codebase (development branch) against the harness-templates spec and all related specs (#898, #899, #906, #910), here are the identified gaps:

1. Missing Fields on Assistant Schema

The current Assistant model in backend/src/schemas/entities/llm.py has no planner, evaluator, or context_strategy fields. Before templates can bundle these configs, the following must be added:

Field	Type	Required By
planner	Optional[PlannerConfig]	Planner Agent (#898)
evaluator	Optional[EvaluatorConfig]	Evaluator Agent (#899)
context_strategy	Literal["compact", "reset", "auto"]	Context Reset (#906)

Gap: The harness-templates spec references PlannerConfig and EvaluatorConfig but these types do not exist anywhere on development. Template creation will fail schema validation until #898 and #899 land their schema definitions. This is the hard blocking dependency.

2. Fork Mechanism Compatibility

The existing AssistantService.fork() method (lines 150-210 of assistant.py) supports template instantiation reasonably well:

• Deep-copies all fields via model_dump()
• Assigns a new UUID
• Strips public fields (public, owner_id, published_at)
• Sets forked_from in metadata
• Increments fork_count on the source

Gaps for template use:

• No is_builtin guard — the fork method has no concept of immutable/built-in entities. Templates need a deletion/mutation guard that Assistants don't have.
• No template_id provenance — when an assistant is created from a template, there's no field to record which template it came from. The current forked_from in metadata tracks assistant-to-assistant forks, but template-to-assistant is a different relationship.
• Templates are proposed as a separate entity (HarnessTemplate with its own CRUD at /api/v0/templates), so the existing AssistantService.fork() cannot be directly reused. A new HarnessTemplateService with its own fork logic is needed.

3. Template Config Validation Gaps

The spec bundles default_models, evaluation_criteria, context_strategy, and cost/time estimates. Several validation concerns:

• Model validation is absent — default_models: dict[str, str] maps role names to model identifiers (e.g., {"planner": "claude-sonnet-4-20250514", "generator": "claude-sonnet-4-20250514", "evaluator": "claude-sonnet-4-20250514"}), but there's no validation that these model strings are valid providers. The current Assistant.model field coerces empty strings to None but doesn't validate against a model registry.
• EvaluationCriterion cross-validation — the template bundles evaluation_criteria: list[EvaluationCriterion], but this list must be kept in sync with the evaluator's EvaluatorConfig.criteria. The spec doesn't clarify whether the template's criteria override or merge with the evaluator's criteria.
• context_strategy compatibility — the template sets context_strategy but the Assistant schema doesn't have this field yet (feat: Context reset with structured handoff artifacts for long-running sessions #906 dependency). If templates ship before feat: Context reset with structured handoff artifacts for long-running sessions #906, this field will be silently ignored.

4. Templates vs. Pre-configured Assistants

The spec proposes HarnessTemplate as a separate entity with its own table, repo, service, and routes. This is the right call because:

• Templates need is_builtin protection (cannot be deleted/modified by users)
• Templates are system-scoped (no user_id FK for built-ins), unlike Assistants which are always user-scoped in (user_id, "assistants") namespace
• Templates hold metadata that Assistants don't need: estimated_cost_range, estimated_duration, default_models per role

However, the current AssistantService uses LangGraph's BaseStore (key-value) for persistence, not SQLAlchemy models. The spec's plan mentions both an Alembic migration and a repo — the decision on whether templates live in the KV store (like assistants) or in a dedicated SQL table needs to be finalized. Given that built-in templates are system-wide and need is_builtin protection, a SQL table is preferable.

5. Planner/Evaluator Spec Gaps That Templates Would Expose

Planner (#898):

• PlannerConfig has 4 fields: enabled, auto_approve, model, scope_level. Templates need all of these, but also need a way to specify the planner system prompt variant. The spec hardcodes a single prompt at backend/src/static/prompts/md/planner.md, but templates like "Full Harness" vs "Frontend Design" likely need different planner prompts. A prompt_variant or prompt_override field is missing.
• scope_level: Literal["conservative", "ambitious"] is binary. Templates may want a middle ground or a custom description of scope expectations.

Evaluator (#899):

• EvaluatorConfig references assistant_id for the evaluator assistant. For templates, this creates a chicken-and-egg problem: the evaluator assistant doesn't exist yet when the template is defined. Templates should reference evaluator config inline rather than by assistant ID.
• EvaluationCriterion has name, description, weight, threshold. Templates define criteria per template (e.g., "code quality" for Reviewed, "design quality + originality + craft + functionality" for Frontend Design). This schema is sufficient, but the description field should be rich enough to include few-shot calibration examples — the spec is silent on whether this is a short string or allows markdown.
• fail_action: Literal["retry", "flag", "stop"] defaults to "retry", but templates might want different defaults per template. The "Solo" template has no evaluator, "Reviewed" wants "retry", but a "CI Gate" template might want "stop". This is already configurable, no gap here.

6. Seeder Pattern: Versioning and Upgradeability

The existing seeders (user_seeder.py, memory_seeder.py) are simple and idempotent:

• user_seeder.py checks for existence by email before creating
• memory_seeder.py checks for existence by memory_id before creating

Gaps for template seeding:

• No version field — built-in templates will evolve (new criteria, updated model recommendations, adjusted cost estimates). The current seeder pattern only checks "does it exist?", not "is it outdated?". A schema_version: int or version: str field is needed on HarnessTemplate so the seeder can detect and upgrade stale built-in templates.
• No upgrade strategy — when a built-in template is upgraded, should users who forked it be notified? Should forks track their upstream template version? The spec is silent on this. At minimum, the seeder should update built-in templates in place (upsert by a stable builtin_slug identifier, not UUID) while leaving forked copies untouched.
• Seeder ordering — the spec correctly notes templates should be system-owned (no user FK for built-ins). The memory_seeder.py runs per-user, but templates should be a one-time global seed. This requires a different seeder pattern — more like user_seeder.py but for a separate harness_templates table.

7. Cross-Feature Dependency Matrix

Feature	Hard Dep on Templates?	Templates Hard Dep on Feature?	Notes
Planner (#898)	No	Yes — PlannerConfig schema required	Templates bundle PlannerConfig
Evaluator (#899)	No	Yes — EvaluatorConfig + EvaluationCriterion schemas required	Templates bundle EvaluatorConfig + criteria
Context Reset (#906)	No	Soft — context_strategy field nice-to-have	Templates set context_strategy, but Solo/Reviewed work without it
Sprint Contracts (#910)	No	No	Sprint contracts layer on top of evaluator, not bundled in templates
Cost Tracking (#905)	No	No	Templates include estimated_cost_range but this is a static string, not tied to live metering
Observability (#904)	No	No	Observability traces phases; templates define phases

Recommended Implementation Order

1. feat: Planner Agent — auto-expand short prompts into full product specs #898 (Planner) and feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899 (Evaluator) — land PlannerConfig and EvaluatorConfig schemas first (even as stubs)
2. feat: Harness Templates — reusable planner→generator→evaluator architectures #911 (Templates) — build on top of those schemas
3. feat: Context reset with structured handoff artifacts for long-running sessions #906 (Context Reset) — can proceed in parallel, templates adopt context_strategy when available
4. feat: Sprint contracts — agents negotiate scope and pass/fail criteria before execution #910 (Sprint Contracts) — layers on top of evaluator, fully decoupled from templates

Action Items Before Implementation

• Finalize EvaluatorConfig to use inline criteria (not assistant_id reference) for template compatibility
• Add prompt_variant or prompt_override to PlannerConfig for template differentiation
• Add schema_version to HarnessTemplate for upgradeable built-in templates
• Decide KV store vs. SQL table for template persistence (recommend SQL for is_builtin protection)
• Add template_id: Optional[str] to Assistant.metadata for provenance tracking

[ryaneggz]

Migration Strategy Note

Schema Changes Required

• HarnessTemplate is a new entity bundling PlannerConfig, EvaluatorConfig, default models, evaluation criteria, context strategy, and cost/time estimates.
• Built-in templates are seeded via a seeder script (backend/seeds/harness_templates.py).

Migration Needs

• None for the primary data model. Templates should be stored in LangGraph Store (the same schemaless JSON document store used for Assistants), using a namespace like ("system", "harness_templates") for built-in templates and (user_id, "harness_templates") for user-created/forked templates. This approach:
  • Is consistent with how Assistants are already stored.
  • Requires no Alembic migration.
  • Supports arbitrary nested JSON (PlannerConfig, EvaluatorConfig, criteria lists).
  • Enables the fork workflow naturally (copy between namespaces).

Alternative: SQL Table

• If query flexibility is needed (e.g., "find all templates with evaluators enabled"), a SQL table could be used. This would require a migration creating a harness_templates table with a JSONB config column. However, given the small expected dataset (4 built-in + user forks) and that templates are fetched by list/get, LangGraph Store is the simpler choice.

Dependencies (Important for Ordering)

• Hard dependency on feat: Planner Agent — auto-expand short prompts into full product specs #898 (Planner Agent) -- HarnessTemplate references PlannerConfig.
• Hard dependency on feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899 (Evaluator Agent) -- HarnessTemplate references EvaluatorConfig and EvaluationCriterion.
• Both must be merged before this feature can be implemented, since the template schema composes those models.
• This should be the last feature implemented in the harness design batch.

Parallel Development Safety

• No Alembic files touched, zero migration conflict risk.
• The seeder (harness_templates.py) is a new file with no conflict risk.
• The template schema imports PlannerConfig and EvaluatorConfig -- if those models change during parallel development, the template schema will need updating. Minimize this risk by landing feat: Planner Agent — auto-expand short prompts into full product specs #898 and feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899 first.

Seeder Strategy

• Use a seeder pattern matching backend/seeds/user_seeder.py.
• Built-in templates should be idempotent (check-before-create on a well-known ID).
• Run seeder on app startup or via make seeds.templates command.

Recommendation

• Store templates in LangGraph Store, not SQL. No migration needed.
• Implement as the final feature in the batch after feat: Planner Agent — auto-expand short prompts into full product specs #898, feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899, and ideally feat: Context reset with structured handoff artifacts for long-running sessions #906 are merged.
• Create a HarnessTemplateService following the same pattern as AssistantService.

[ryaneggz]

Dependency Graph Validation

Position in Dependency Graph

Context Reset & Handoff (#911) — SEMI-INDEPENDENT (Layer 0-1)
  └─ soft dep on: Planner (#898) — plan files persist through resets
  └─ soft dep on: Evaluator (#899) — evaluator config persists through resets
  └─ soft dep on: Sprint Contracts (#906) — contract state persists through resets
  └─ depended upon by: Harness Templates (#910) — context_strategy field

Declared Dependencies

• "Planner/evaluator integration — plan files, evaluator configs, and sprint contract state persist through resets"

Validated Dependencies

Partially correct. The spec correctly identifies soft dependencies but mischaracterizes them. The core Context Reset feature (ContextResetMiddleware + HandoffArtifact + context_strategy on Assistant) is genuinely independent. The "integration" with planner/evaluator is about ensuring those artifacts are carried across resets — which only matters if those features exist.

The feature CAN be built standalone: it monitors context utilization and triggers resets with handoff artifacts regardless of whether planner/evaluator are configured. The "integration" bullets are forward-compatibility concerns, not blocking dependencies.

Implicit Dependencies Found

1. CompactingMiddleware coexistence — The spec adds context_strategy: "compact" | "reset" | "auto" to Assistant. The existing CompactingMiddleware (at backend/src/utils/compacting.py) currently runs unconditionally. Context Reset must modify the middleware registration logic (backend/src/utils/middleware.py) to conditionally apply compacting OR reset based on the assistant's context_strategy. This is a non-trivial integration with existing code that the spec doesn't detail.

2. Model detection for auto strategy — The spec says auto selects "compact for Opus-class, reset for Sonnet-class, based on model detection." The model string is available on the Assistant or LLMRequest, but there's no existing model classification utility. A new helper like is_opus_class(model: str) -> bool is needed. This is a small implicit dependency but should be noted.

3. "StateBackend" terminology — Like Planner (feat: Planner Agent — auto-expand short prompts into full product specs #898) and Sprint Contracts (feat: Context reset with structured handoff artifacts for long-running sessions #906), this spec stores files:

   • /HANDOFF-{session_number}.md
   • References /PLAN.md from Planner
   • References /contracts/sprint-N.md from Sprint Contracts

   All three specs use "StateBackend" as a concept but the codebase has no class by that name. This needs to be resolved — either as assistant.files, thread state, or a new abstraction.

4. Session metadata on threads — The spec says "Track session number within a thread. Store mapping of session boundaries in thread metadata." The existing Thread schema (in __init__.py) has no metadata or session fields. This requires either extending the Thread schema or storing session data in the checkpoint state. This is an unaddressed schema extension.

5. Message queuing during reset — The spec's risk section mentions "queue incoming messages during reset, deliver to new session." The current architecture streams messages synchronously. Adding a queue during the reset window is a significant infrastructure change that the spec handwaves.

Interface Contracts Needed

1. context_strategy field on Assistant — This field must be added to Assistant in llm.py. Harness Templates (feat: Sprint contracts — agents negotiate scope and pass/fail criteria before execution #910) references it. It should be:

   context_strategy: Literal["compact", "reset", "auto"] = "auto"

2. Shared file storage abstraction — Three specs need file persistence. Define NOW:

   class AgentFileStore:
       async def read(self, path: str) -> str | None
       async def write(self, path: str, content: str) -> None
       async def list(self, prefix: str) -> list[str]

   Implementation can use assistant.files, thread checkpoint state, or a new backend.

3. HandoffArtifact schema — Define the exact schema so Context Reset middleware and Sprint Contracts can both reference it:

   class HandoffArtifact(BaseModel):
       completed_work: list[str]
       in_progress: Optional[str]
       remaining_tasks: list[str]
       key_decisions: list[str]
       files_modified: list[str]
       current_state_summary: str
       plan_reference: Optional[str]

Schema Conflict Risk

• Medium. Adds context_strategy to Assistant in llm.py — same file modified by Planner (feat: Planner Agent — auto-expand short prompts into full product specs #898) and Evaluator (feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899). Migration ordering required.

Critical Path Assessment

Not on critical path for v0.9.0. The core use case (long-running Sonnet sessions) is valuable but secondary to the Planner + Evaluator foundation. The Anthropic article notes that "Opus 4.6 largely removed context anxiety, making compaction sufficient." Recommended: Wave 2, after Planner + Evaluator ship. However, the context_strategy field on Assistant could be added in the Wave 1 migration as a forward-compatible stub.

[ryaneggz]

Testing Strategy Recommendations

Infrastructure Context

Harness Templates is the highest-level feature — it composes #898 (Planner), #899 (Evaluator), and #906 (Context Reset). It follows existing patterns for CRUD entities: see tests/unit/services/test_assistant_service.py (InMemoryStore-based service testing), tests/unit/repos/test_tool_repo.py (repo layer), and tests/integration/test_public_assistants.py (API endpoint testing). The seeder pattern comes from seeds/user_seeder.py.

Recommended Tests

Unit Tests (tests/unit/schemas/test_harness_template.py)

• HarnessTemplate Pydantic validation: all fields, is_builtin defaults to False
• context_strategy accepts valid enum values
• default_models dict validates model strings (provider:model format)
• evaluation_criteria accepts list of EvaluationCriterion (reuse feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899 schema)
• Optional fields (planner, evaluator, estimated_cost_range, estimated_duration) all nullable

Unit Tests (tests/unit/repos/test_harness_template_repo.py)

• CRUD operations via InMemoryStore: create, read, list, update, delete
• Built-in templates cannot be deleted (verify error/exception)
• User templates can be deleted
• Fork creates a copy with is_builtin=False and new id

Unit Tests (tests/unit/services/test_harness_template_service.py)

• List templates returns both built-in and user-created
• Get template by ID returns correct template
• Create user template with valid data succeeds
• Fork built-in template: creates editable copy, preserves all config
• Fork preserves evaluation criteria and model recommendations
• Delete built-in template raises appropriate error

Unit Tests (tests/unit/routes/test_harness_template_routes.py)

• GET /api/v0/templates: returns all templates, includes built-in
• GET /api/v0/templates/{id}: returns template details, 404 for non-existent
• POST /api/v0/templates: creates user template
• PUT /api/v0/templates/{id}: updates user template, rejects update to built-in
• DELETE /api/v0/templates/{id}: deletes user template, 403 for built-in

Seeder Tests (tests/unit/seeds/test_harness_template_seeder.py)

• Seeder creates exactly 4 built-in templates (Solo, Reviewed, Full Harness, Frontend Design)
• Each template has correct configuration:
  • Solo: no planner, no evaluator, context_strategy="compact"
  • Reviewed: no planner, evaluator with code quality criteria
  • Full Harness: planner (ambitious), evaluator with multi-criteria, context_strategy="auto"
  • Frontend Design: planner, evaluator with design criteria + Playwright MCP reference
• Seeder is idempotent — running twice does not create duplicates
• Seeder does not overwrite modified built-in templates (or does — specify behavior)

Integration Tests (tests/integration/test_template_assistant_flow.py)

• Create assistant from template: verify planner + evaluator config is applied correctly
• Via async_client: POST /api/v0/templates → GET to verify → create assistant from template → POST /api/llm/stream → verify planner/evaluator phases execute
• Template selector API returns templates sorted by complexity (Solo → Full Harness)

Frontend Tests

• TemplateCard.tsx: renders template name, description, model recommendations, cost estimate
• TemplateSelector.tsx: renders card grid, clicking card selects template, "Fork" button visible on built-in
• Template indicator badge on assistant cards: shows which template was used
• Empty state: renders correctly when no user templates exist (only built-in shown)

Gaps in Spec's Test Section

• Missing: Test for template versioning — when a built-in template is updated in a new release, what happens to assistants created from the old version?
• Missing: Test for template with invalid model references — models specified in default_models may not be available to the user
• Missing: Test for template cost/time estimates — are these computed dynamically or static strings?
• Missing: Test for template search/filter — spec mentions card grid but no filtering capability
• Missing: Authorization tests — can non-admin users see/fork all templates? Can they modify built-in templates?
• Missing: Test for template export/import (for sharing between Orchestra instances)

Dependency Integration Tests

These tests validate that templates correctly compose the underlying features:

• Template with planner enabled → verify planner phase runs (feat: Planner Agent — auto-expand short prompts into full product specs #898 contract)
• Template with evaluator enabled → verify evaluate-regenerate loop runs (feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899 contract)
• Template with context_strategy="auto" → verify correct strategy selection (feat: Context reset with structured handoff artifacts for long-running sessions #906 contract)
• Full Harness template end-to-end: planner → generator → evaluator loop → context reset (if threshold hit) — this is the most important integration test across all specs

Shared Fixtures Needed

• builtin_templates: the 4 seeded templates as fixture data for comparison assertions
• user_template_fixture: a custom template created by fork for update/delete tests
• template_assistant_fixture: an Assistant created from a template for integration tests