Critical: Validation evidence collection sends entire worktree instead of agent delta — token explosion blocks parallel runs

[lwgray]

What is Marcus?

Marcus is a multi-agent software development system. Multiple AI agents work in parallel, each picking up a task (a unit of coding work), implementing it, and then submitting it for validation. Validation checks whether the agent's code actually satisfies the acceptance criteria defined for that task.

This issue is about the validation step — the moment when Marcus checks whether an agent's work is correct.

---

The Problem (Plain English)

When an agent finishes a task and asks Marcus to validate the work, Marcus collects all the files in the project directory and sends them to an AI model for review. The problem: by the time the 4th or 5th agent is done, dozens of other agents have already merged their own code into the project. The validator picks up everyone's files, not just the files the current agent wrote.

This is like asking a professor to grade Student 3's essay, but handing the professor the entire class binder — all 47 essays — instead of just Student 3's.

The result: the AI model receives hundreds of thousands of tokens it doesn't need, driving up cost and eventually exceeding the model's token limit, which causes the validation to fail with an AI error and blocks the agent from completing their task.

---

Observed Failure

In a recent Marcus run, agent 1_25 was working on task 471d75026 ("Implement Task Execution Engine with Retry and Failure Handling"). By the time agent 1_25 finished, its worktree contained:

• executor.py — the file the agent actually wrote
• cli.py, logger.py, config.py, dependencies.py — files merged from other agents
• 40+ additional files from accumulated agent merges

The validation system loaded all of it:

Metric	Value
Files collected	47 files
Total size	707 KB
Tokens sent to validator	211,000 tokens
Token limit	~200,000 tokens
Result	AI error — validation blocked

The agent could not complete the task. The task stayed in IN_PROGRESS indefinitely.

---

Measured Cost Across Last 5 Projects

Here is the real-world cost of this bug, calculated from the token_events table in ~/.marcus/costs.db (operation = validate_work, model = claude-haiku-4-5, priced at $1.00/M input tokens, $5.00/M output tokens):

Project	Validation Calls	Input Tokens	Output Tokens	Validation Cost	Avg Tokens/Call	Largest Single Call
cli-task-runner-2	5	421,966	13,652	$0.4902	84,393	122,985
cli-task-runner	6	195,164	9,778	$0.2441	32,527	48,963
full-run	3	67,502	10,191	$0.1185	22,500	27,748
todo-app-1	4	62,355	10,136	$0.1130	15,588	22,082
test_build	0	0	0	$0.0000	—	—
Total	18	746,987	43,757	$0.9658	—	—

Key observation: cli-task-runner-2 spent more on validation ($0.49) than on project setup/planning ($0.32). The largest single validation call was 122,985 input tokens — sent to check one agent's work. The correctly-scoped call should be ~3,000–5,000 tokens (just the changed files).

Cost projection: At the current trajectory, a 10-agent parallel run producing ~15 tasks would spend approximately $1.50–$3.00 on validation alone — just because the validator loads the entire codebase every time.

---

Root Cause

The bug is in src/ai/validation/work_analyzer.py, specifically the _discover_source_files method (line 454) and the gather_evidence method (line 121).

gather_evidence — line 151

source_files = self._discover_source_files(project_root)

This calls _discover_source_files with the full project_root. There is no concept of "what did this specific agent change?"

_discover_source_files — line 477

for root, dirs, files in os.walk(project_root):

os.walk recurses through every file in the directory. It has a 10MB total size guard and a 1MB per-file guard, but no awareness of which files belong to which agent's task.

By the time agent 5 finishes work, os.walk returns 47 files from 5 different agents' work. All 47 files get included in the validation prompt.

---

The Fix: Git-Delta Evidence Collection

Instead of walking the entire directory, the validator should ask git: "What files did THIS agent change for THIS task?"

Step 1 — Snapshot baseline at task assignment

When Marcus assigns a task to an agent, record the current git HEAD commit as a baseline_commit in the task record. This captures the state of the codebase before the agent started working.

Step 2 — Compute the delta at validation time

When gather_evidence is called, run:

git diff <baseline_commit>..HEAD --name-only

This returns only the files the agent added or changed since they started the task. Other agents' merged files are not included because they existed before or were merged after the baseline.

Step 3 — Collect evidence from delta files only

Pass the delta file list into _discover_source_files as an allowlist instead of walking the whole tree.

Expected result

Metric	Before (current)	After (fix)
Files per validation	47 (and growing)	2–5 (stable)
Tokens per validation	122,000+	~3,000–5,000
Cost per validation	~$0.12	~$0.003
Token limit failures	Yes (>200k)	No
Scales with project size?	No — gets worse	Yes — constant

---

Why We Cannot Use File Ownership Declarations Instead

An alternative fix would be to have each task's contract declare which files it owns upfront (e.g., "this task owns executor.py and test_executor.py"). We cannot do this because:

Agents decide what files to create during implementation. A task that says "implement the retry handler" might produce retry.py, retry_config.py, and test_retry.py — or it might produce handlers/retry.py and handlers/__init__.py. We don't know the filenames at task creation time. The git-delta approach requires no upfront declaration: it just asks what actually changed.

---

Where to Look in the Code

File	Purpose
src/ai/validation/work_analyzer.py:121	gather_evidence() — entry point for evidence collection
src/ai/validation/work_analyzer.py:454	_discover_source_files() — the method that walks the entire tree
src/ai/validation/work_analyzer.py:477	os.walk(project_root) — the line that recurses through everything
src/ai/validation/work_analyzer.py:370	_get_project_root() — resolves worktree path for the agent
src/marcus_mcp/tools/task.py:2700	_validate_and_complete_implementation() — calls the validator
~/.marcus/costs.db	token_events table — operation='validate_work' rows hold the cost data

Glossary

Term	Definition
Marcus	The coordination system that manages the kanban board and validates agent work
Agent	An AI instance that picks up a task, writes code, and submits it for validation
Worktree	A git working directory — each agent may have its own copy of the repo
validate_work	The operation name for LLM-based acceptance criteria validation in the cost store
_discover_source_files	The method in work_analyzer.py that collects files for the validation prompt
baseline_commit	The git commit SHA at the moment a task is assigned — the "before" snapshot
git delta	The set of files changed between two commits (git diff --name-only)
token_events	Table in ~/.marcus/costs.db recording every AI API call with token counts and cost

---

How to Verify the Bug Today

1. Run a Marcus experiment with 3+ parallel agents on any multi-task project
2. After agents start merging branches, watch the Marcus server logs:

   grep "Discovered.*files" ~/.marcus/logs/marcus_*.log | tail -20
   

3. You will see file counts growing with each merge: 5 files → 12 files → 30 files → 47 files
4. Check validation token costs after the run:

   import sqlite3
   conn = sqlite3.connect('~/.marcus/costs.db')
   cur = conn.cursor()
   cur.execute("SELECT input_tokens, output_tokens, timestamp FROM token_events WHERE operation='validate_work' ORDER BY timestamp DESC LIMIT 10")
   for r in cur.fetchall(): print(r)

5. Expected (broken): input tokens >50,000 per call for mid/late tasks
6. Expected (fixed): input tokens <10,000 per call regardless of how many agents have merged

---

How to Verify the Fix

After implementing the git-delta approach:

1. Run the same experiment
2. Check the logs — file count should be 2–5 per validation regardless of how many agents have merged
3. Check token_events — validate_work input tokens should be <10,000 per call
4. No validation should fail with an AI token-limit error
5. Total validation cost for a 10-task run should be <$0.05

---

Related

• Simon blocker: df7d0aea — "Agent validation blocked in parallel runs: evidence collection submits entire merged worktree instead of task-scoped delta"
• src/ai/validation/work_analyzer.py — primary file to change
• Issue with token_events project_id='unassigned' for validate_work events (separate tracking bug — validate_work events are not linked to their run_id, making cost attribution to projects impossible)

[lwgray]

Fixed by PR #697, squash-merged to develop as 0cc06191. GitHub didn't auto-close this because the repo's default branch is main and #697 merged into develop (per the project's develop-only merge workflow). Two follow-up Codex P2s were addressed in the same PR before merge: empty-git-delta now returns [] (surfaces the correct 'no source files' failure instead of a full-worktree scan), and delta paths are resolved + bounds-checked to block a source-named symlink from leaking external file content into the validation prompt.