feat: Sprint contracts — agents negotiate scope and pass/fail criteria before execution

[ryaneggz]

Summary

Implement a Sprint Contract negotiation protocol between generator and evaluator agents. Before each sprint (work chunk), the generator proposes a SprintContract containing features to build and testable acceptance criteria. The evaluator reviews, can counter-propose, and once both sides agree, the contract is persisted and used as the grading rubric.

This mirrors Anthropic's harness pattern where sprint contracts contained 27+ testable criteria per feature chunk, bridging the gap between high-level product specs and testable implementation.

Key Components

• SprintContract schema with TestCriterion entries, status lifecycle (proposed -> negotiating -> agreed -> in_progress -> completed/failed)
• Negotiation flow in LLM controller: generator proposes, evaluator reviews, configurable max rounds (default 3)
• File-based communication via StateBackend (/contracts/sprint-N.md)
• Contract-aware evaluation: evaluator grades each criterion individually, reports "17/27 criteria passed" style results
• Frontend panels: ContractPanel.tsx for viewing contracts, CriteriaChecklist.tsx for pass/fail indicators

Dependencies

• Requires evaluator-agent (hard dependency for negotiation and grading)
• Best used with planner-agent (soft dependency for generating high-level specs)

Spec

See .claude/specs/sprint-contracts.md for full implementation details.

File Changes

File	Action
backend/src/schemas/entities/sprint_contract.py	New
backend/src/controllers/llm.py	Modify
frontend/src/components/panels/ContractPanel.tsx	New
frontend/src/components/lists/CriteriaChecklist.tsx	New
backend/tests/unit/test_sprint_contract.py	New
backend/tests/integration/test_sprint_contract_flow.py	New

[ryaneggz]

Dependency Gap Analysis: Sprint Contracts vs. Evaluator (#899) and Planner (#898)

After analyzing the sprint-contracts spec, evaluator-agent spec, planner-agent spec, and the current codebase (subagent communication patterns in deepagents, LLMController, Assistant schema), here are the identified gaps.

1. Evaluator Must Expose a Contract-Compatible Grading Interface

The sprint-contracts spec assumes the evaluator can grade against a list of TestCriterion objects individually (pass/fail + notes per criterion, "17/27 criteria passed" reporting). However, the evaluator spec defines EvaluationCriterion with weighted float scores and thresholds — a fundamentally different schema from TestCriterion.

Gap: There is no shared criterion type between the two specs. Sprint Contracts uses TestCriterion(description, verification_method) with binary pass/fail semantics. The Evaluator uses EvaluationCriterion(name, description, weight, threshold) with continuous 0-1 scoring. These need to be reconciled or the Evaluator needs to support both modes — a "contract grading mode" for Sprint Contracts and a "continuous scoring mode" for general evaluation.

Recommendation: The Evaluator should support accepting a SprintContract as input and returning per-TestCriterion results. Either EvaluationResult needs a contract_results: list[CriterionResult] field, or the Evaluator needs a second interface entirely.

2. Generator-Evaluator Negotiation Has No Message-Passing Pattern

The sprint-contracts spec describes a negotiation loop: generator proposes, evaluator reviews, counter-proposes, up to N rounds. The codebase's current subagent communication is unidirectional and ephemeral — the parent agent spawns a subagent via the task tool, the subagent runs to completion and returns a single ToolMessage. There is no back-and-forth messaging between subagents.

Key observation from deepagents/middleware/subagents.py: Subagents receive a single HumanMessage(content=description) and return a single final message. The _EXCLUDED_STATE_KEYS set excludes messages but NOT files, meaning files in state ARE shared between parent and subagents. This makes the spec's file-based communication channel (/contracts/sprint-N.md) viable — but the negotiation loop itself must be orchestrated at the LLMController level, not within the subagent middleware.

Gap: Neither the evaluator spec nor the sprint-contracts spec defines where the negotiation loop controller lives. It cannot be inside either subagent (they're ephemeral). It must be a new orchestration pattern in LLMController that:

1. Invokes the generator to propose a contract
2. Writes the proposal to StateBackend (/contracts/sprint-N.md)
3. Invokes the evaluator with the contract for review
4. Reads the evaluator's counter-proposal from state
5. Re-invokes the generator with feedback
6. Repeats up to max_rounds

This is a new orchestration primitive that doesn't exist today.

3. Planner Output to Sprint Contract Input — Missing Linkage

The sprint-contracts spec says contracts "break down" the planner's high-level spec into testable sprints. However:

• The planner spec outputs a single /PLAN.md file (overview, features, user stories, design direction).
• The sprint-contracts spec expects a list of features per sprint with testable criteria.
• There is no defined interface for how /PLAN.md content is consumed to generate sprint proposals.

Gap: The generator needs to read /PLAN.md and decompose it into sprint-sized chunks. This decomposition logic is not specified anywhere. Should the generator do this automatically? Should there be a "sprint planner" sub-phase? The current Planner spec only runs on the first message and has no sprint-chunking capability.

4. Missing Schema Fields

On Assistant (llm.py):

• Neither evaluator: EvaluatorConfig nor planner: PlannerConfig exist yet (both specs propose adding them). Sprint Contracts depends on the evaluator field being present.
• Sprint Contracts may also need a field like contract_negotiation: ContractConfig on Assistant to configure max negotiation rounds, auto-approve thresholds, etc.

On EvaluatorConfig (evaluator spec):

• Missing: ability to accept SprintContract as structured input for grading mode
• Missing: grading_mode: Literal["continuous", "contract"] to switch between weighted scoring and binary pass/fail per criterion
• Missing: contract_results field on EvaluationResult for per-criterion binary results with notes

On PlannerConfig (planner spec):

• Missing: sprint_decomposition: bool — whether the plan should be structured as sprint-sized chunks
• Missing: any output schema that Sprint Contracts can consume programmatically (currently just free-form markdown)

5. Should Sprint Contracts Influence Evaluator Design NOW?

Yes. The Evaluator should be built with contract grading as a first-class capability from day one, not retrofitted. Specific recommendations:

1. EvaluationResult should include an optional criterion_results: list[CriterionResult] field that maps 1:1 with input criteria (whether EvaluationCriterion or TestCriterion)
2. The evaluator's system prompt should support a "contract mode" where it grades against a provided list of criteria rather than its own configured criteria
3. The evaluator's orchestration interface should support being invoked with a SprintContract payload, not just raw generator output
4. The evaluator should be able to return structured counter-proposals (not just critique text) for the negotiation loop

Summary of Critical Gaps

Gap	Severity	Blocks Sprint Contracts?
No shared criterion type between Evaluator and Sprint Contracts	High	Yes — cannot grade contracts without it
No multi-round negotiation orchestration pattern	High	Yes — core feature requires it
No Planner → Sprint decomposition interface	Medium	Soft dependency, can work without
No contract grading mode on Evaluator	High	Yes — evaluator must support binary pass/fail per criterion
Missing ContractConfig on Assistant schema	Medium	Can be added alongside evaluator field

[ryaneggz]

Migration Strategy Note

Schema Changes Required

• SprintContract and TestCriterion are new Pydantic models.
• Contracts are persisted as structured markdown files in StateBackend (/contracts/sprint-N.md), not in a SQL table.
• Contract-aware evaluation results are stored in thread metadata or as structured messages.

Migration Needs

• None. All sprint contract data lives in schemaless stores:
  • Contracts are files in StateBackend (LangGraph graph state).
  • Evaluation results referencing contracts are stored in thread metadata (LangGraph Store).
  • No new SQL tables, no column additions to existing SQL tables.

Dependencies (Important for Ordering)

• Hard dependency on feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899 (Evaluator Agent). Sprint contracts require the evaluator to exist for the negotiation and grading flow. EvaluatorConfig and EvaluationCriterion from feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899 must be available.
• Soft dependency on feat: Planner Agent — auto-expand short prompts into full product specs #898 (Planner Agent). The planner generates the high-level spec that contracts break down, but sprint contracts can function without it.
• This means feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899 should merge before this feature branch is started or at minimum before this PR is opened.

Parallel Development Safety

• No Alembic files touched, zero migration conflict risk.
• Code-level conflict in backend/src/controllers/llm.py: Both this spec and feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899 modify the LLM controller orchestration logic. If developing in parallel, expect merge conflicts in the controller. Consider developing this on a branch based on the feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899 branch.

Shared Schemas

• TestCriterion in this spec is conceptually related to EvaluationCriterion in feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899. Consider whether TestCriterion should extend or compose with EvaluationCriterion to avoid schema fragmentation. The evaluator needs to map contract criteria to its grading rubric.

Recommendation

• Define SprintContract and TestCriterion in backend/src/schemas/entities/sprint_contract.py.
• No migration file needed.
• Wait for feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899 to be at least code-complete before starting implementation to avoid rework on the orchestration loop.

[ryaneggz]

Dependency Graph Validation

Position in Dependency Graph

Harness Templates (#910) — DEPENDENT (Layer 2)
  └─ HARD dep on: Evaluator (#899) — templates bundle EvaluatorConfig
  └─ HARD dep on: Planner (#898) — templates bundle PlannerConfig
  └─ SOFT dep on: Context Reset (#911) — templates set context_strategy
  └─ depended upon by: (nothing — leaf node)

Declared Dependencies

• "Requires evaluator-agent spec implementation"
• "Requires planner-agent spec implementation"

Validated Dependencies

Correct. The HarnessTemplate schema directly imports and bundles PlannerConfig and EvaluatorConfig:

class HarnessTemplate(BaseModel):
    planner: Optional[PlannerConfig] = None
    evaluator: Optional[EvaluatorConfig] = None
    evaluation_criteria: list[EvaluationCriterion] = []
    context_strategy: Literal["compact", "reset", "auto"] = "auto"

This means PlannerConfig, EvaluatorConfig, and EvaluationCriterion must exist as importable schemas before Harness Templates can even define its own schema. These are true hard dependencies.

Implicit Dependencies Found

1. Context Reset (feat: Harness Templates — reusable planner→generator→evaluator architectures #911) — The template schema includes context_strategy: Literal["compact", "reset", "auto"]. This field only makes sense if Context Reset (feat: Harness Templates — reusable planner→generator→evaluator architectures #911) is implemented. Without it, only "compact" is functional. This is an undeclared soft dependency.

2. Seeder infrastructure — The spec creates backend/seeds/harness_templates.py for seeding built-in templates. The existing seeder pattern uses backend/seeds/user_seeder.py. This is an existing pattern, so the dependency is satisfied, but the seeder will fail at runtime if the PlannerConfig and EvaluatorConfig schemas don't exist (because the seed data references them).

3. Assistant creation flow — Templates are applied when creating assistants. The spec says "Template config is applied to the assistant's planner and evaluator fields." This means the Assistant schema must already have planner: Optional[PlannerConfig] and evaluator: Optional[EvaluatorConfig] fields. These are added by feat: Planner Agent — auto-expand short prompts into full product specs #898 and feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899 respectively. If either of those isn't merged, template application silently fails or crashes.

4. Cost estimate data — Templates include estimated_cost_range and estimated_duration. These are static strings ("$5-$50"), not computed from actual Cost Tracking (feat: Per-run cost tracking, phase-level metering, and budget caps #905) data. So Cost Tracking is NOT a dependency, but the estimates will be inaccurate without real cost data to calibrate against.

Interface Contracts Needed

1. Schema import chain — Harness Templates needs these exact types:

   • PlannerConfig from planner.py (defined by feat: Planner Agent — auto-expand short prompts into full product specs #898)
   • EvaluatorConfig from evaluation.py (defined by feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899)
   • EvaluationCriterion from evaluation.py (defined by feat: Evaluator Agent — GAN-inspired generator/evaluator feedback loop #899)

   These schemas MUST be merged before Harness Templates development begins.

2. Template → Assistant application — The spec needs a defined method for applying a template to an assistant:

   def apply_template(template: HarnessTemplate, assistant: Assistant) -> Assistant:
       assistant.planner = template.planner
       assistant.evaluator = template.evaluator
       # What about context_strategy? Does it go on Assistant?

   The context_strategy field is proposed on Assistant by Context Reset (feat: Harness Templates — reusable planner→generator→evaluator architectures #911), but if that isn't built yet, Harness Templates can't set it.

3. Built-in template IDs — The seeder creates templates with specific IDs. Other parts of the system (e.g., frontend default selection, API docs) may reference these IDs. They should be defined as constants.

Schema Conflict Risk

• Low. This spec creates new entities (HarnessTemplate) rather than modifying existing ones. No direct conflicts with other specs. However, it requires other specs' schemas as imports.

Critical Path Assessment

Not on critical path. This is the highest-layer dependent spec — it packages other specs' outputs. Must be built after both Planner (#898) AND Evaluator (#899) are merged. Recommended: Wave 2, after v0.9.0 Planner + Evaluator ship. However, the HarnessTemplate schema definition can be written now against the proposed PlannerConfig/EvaluatorConfig interfaces.

[ryaneggz]

Testing Strategy Recommendations

Infrastructure Context

Sprint contracts depend on both #898 (Planner) and #899 (Evaluator). Testing should assume those features are already implemented and use their fixtures. The negotiation loop follows the same evaluate-regenerate pattern from #899, so the mock strategy (staged AsyncMock.side_effect responses) applies directly.

Recommended Tests

Unit Tests (tests/unit/schemas/test_sprint_contract.py)

• TestCriterion validation: description and verification_method required
• SprintContract validation: status lifecycle (proposed → negotiating → agreed → in_progress → completed/failed)
• Invalid status transitions rejected (e.g., proposed → completed directly)
• negotiation_rounds defaults to 0, increments correctly
• Contract with 0 criteria is valid (edge case)
• Contract with 50+ criteria is valid (stress test the schema, not perf)

Unit Tests (tests/unit/controllers/test_sprint_contract_negotiation.py) — CRITICAL

• Generator proposes: verify contract proposal is sent to evaluator with correct structure
• Evaluator approves: single-round negotiation, contract status transitions proposed → agreed
• Evaluator counter-proposes: evaluator requests changes, generator revises, contract status goes proposed → negotiating → agreed
• Max negotiation rounds: after N rounds (default 3) without agreement, last proposal is auto-accepted or flagged
• Contract persisted to StateBackend: verify /contracts/sprint-{N}.md is written with correct content

How to test negotiation without LLM calls:

@patch("src.controllers.llm.init_chat_model")
async def test_negotiation_counter_proposal(mock_init):
    gen_mock = AsyncMock()
    eval_mock = AsyncMock()
    
    # Generator proposes
    gen_mock.ainvoke.side_effect = [
        AIMessage(content='{"features":["auth"],"test_criteria":[{"description":"Login works"}]}'),
        AIMessage(content='{"features":["auth"],"test_criteria":[{"description":"Login works"},{"description":"Logout works"}]}'),
    ]
    # Evaluator: reject first, accept second
    eval_mock.ainvoke.side_effect = [
        AIMessage(content='{"approved":false,"feedback":"Add logout criterion"}'),
        AIMessage(content='{"approved":true}'),
    ]
    mock_init.side_effect = [gen_mock, eval_mock]
    # ... verify 2 negotiation rounds, final contract includes both criteria

Unit Tests (tests/unit/controllers/test_contract_aware_evaluation.py)

• Evaluator grades each TestCriterion individually — verify per-criterion pass/fail in result
• "17/27 criteria passed" style reporting: verify summary includes counts
• Evaluator receives the agreed contract as context (not the proposal)
• Contract compliance score is included in overall EvaluationResult

Integration Tests (tests/integration/test_sprint_contract_flow.py)

• Full flow: create assistant with evaluator → start thread → generator proposes contract → evaluator reviews → contract agreed → generator builds → evaluator grades against contract criteria
• Via async_client: verify SSE stream contains contract negotiation events and per-criterion evaluation results
• Multi-sprint: after sprint 1 completes, sprint 2 begins with new contract

Frontend Tests

• ContractPanel.tsx: renders current sprint's features and criteria, shows status badge
• CriteriaChecklist.tsx: renders each criterion with pass/fail checkbox after evaluation
• Negotiation history: collapsed by default, expandable, shows each round's changes

File-Based Communication Testing

• Verify /contracts/sprint-1.md format is valid structured markdown
• Verify /contracts/sprint-1-evaluation.md is written after grading
• Both generator and evaluator can read contract files (mock StateBackend read/write)

Gaps in Spec's Test Section

• Missing: Test for contract negotiation timeout — what if evaluator never responds?
• Missing: Test for generator ignoring contract and building something different — evaluator should catch this
• Missing: Test for contract amendment mid-sprint (scope change during execution)
• Missing: Test for contract with criteria that require Playwright verification — how are those marked when Playwright is unavailable?
• Missing: Test for sprint numbering continuity across context resets (feat: Context reset with structured handoff artifacts for long-running sessions #906)
• Missing: Test for concurrent sprint contracts on the same thread (should not be possible — enforce sequential sprints)

Shared Fixtures Needed

• sample_sprint_contract: factory creating SprintContract with configurable features/criteria count
• sample_test_criterion: factory for TestCriterion with various verification methods
• mock_contract_negotiation_exchange: pre-built generator+evaluator response sequences for common negotiation patterns