feat(huge-repo): huge-repo-native tier (Tier J) — 6 gaps fixed + validated end-to-end - #4
Merged
Merged
Conversation
…etry Fold the manual env workarounds an unbuildable 358k-file repo required into auto-applied, capacity-aware behaviour so a huge/unbuildable repo runs with zero .autodev/config.json edits and zero env vars. Additive, idempotent, no-op on small repos, gated by huge_repo_overrides_disabled. - adapter binding-gap fix (get_adapter binds cfg before the preflight probe) + spawn-agent isolation (--setting-sources user / --strict-mcp-config) so target-repo hooks+MCP no longer inflate cold start - configurable + huge-scaled probe timeout; conservative parallelism (parallelism_multiplier + ceiling); retry jitter; empty-result -> infra-subtype retry; scaled circuit_breaker_window_s - auto-applied huge-repo profile (runtime/ephemeral, never written to disk); C++/CMake unbuildable detection + auto test soft-pass - configurable budget_escalation ceilings; non-task-role turn scaling (reviewer); task-decomposition guidance + telemetry - run cost + wall-time summary (.autodev/run-summary.jsonl) via a CostRecordingAdapter emitting per-invocation invocation_cost ledger ops (captures tournament spend the enforcer cap misses) 3349 tests green.
… repos
Two gaps surfaced by a live run on a 358k-file Git-LFS repo:
1. Execute-phase worktrees came up NON-sparse (worktree_huge_repo_mode=auto
didn't engage sparse, and create_per_task never passed the huge timeout),
so a full LFS checkout produced ~62MB phantom diffs that tripped the
diff-size guardrail and blocked every task, plus 60s worktree-add timeouts.
- apply_huge_repo_profile now auto-enables worktree_sparse_checkout_enabled
- create_per_task passes worktree_huge_create_timeout_s (600s) on huge repos
- sparse cone falls back to the task's files/extended_scope when the plan
declares no edit_scope
2. The preflight PONG probe used the heavy default model (~9-11s, straddling
the 10s timeout) and the detect-time probe runs unbound (couldn't use the
scaled 15s timeout), so plan/execute/resume intermittently failed the probe
under startup index-refresh contention.
- new adapters.probe_model (default "haiku") -> ~7-8s cold start
- unbound detect-time probe timeout default raised 10s -> 20s (a bound
probe_timeout_s + huge-scaling still wins for the post-bind re-probe)
Additive, idempotent, no-op on small repos. 3360 tests green.
…x.lock cleanup Third gap from the live huge-LFS run: the impl-tournament worktree path is separate from create_per_task and was still non-sparse with a 60s timeout (its WorktreeManager was built non-huge and the tournament engine passed no cone). It timed out, the killed `git worktree add` left a stale main-repo .git/index.lock, and the next `git apply` failed (rc=128) -> 3-way failed -> task blocked, cascading to all subsequent applies. - WorktreeManager gains default_sparse_paths; create() falls back to it, so the tournament engine's scope-less create() goes through the existing sparse (--no-checkout + sparse-checkout set + checkout) machinery - impl_tournament_runner builds its WorktreeManager huge-aware (huge_mode, worktree_huge_create_timeout_s=600, autodev_root) and passes the task cone (files + extended_scope), mirroring the execute path - new git_utils.clear_stale_index_lock(): removes .git/index.lock only when unowned (dead PID) AND aged (>30s); apply_patch_to_main calls it before the main-repo index mutation. Belt-and-suspenders against killed/contended git. Pooled-worktree path (worktree_pool_enabled, default off) noted as follow-up. Additive, idempotent, no-op on small repos. 3375 tests green.
Fourth gap from the live huge-LFS run: a Phase-0 research/empty-diff task had its reviewer-APPROVED empty diff on record, but the developer kept exhausting its turn budget exploring the 358k-file repo before re-emitting an artifact -> error_max_turns_escalation_exhausted -> user_decision_required soft-block that lost the approval and stalled autonomous completion. The execute loop now, at the developer-failure chokepoint (_maybe_accept_approved_on_exhaustion), accepts the approved artifact as complete (empty diff = no-op integration) ONLY when all hold: - the failure subtype is turn-exhaustion (error_max_turns / error_max_turns_escalation_exhausted), - a review verdict of APPROVED is recorded for the task, AND - the in-hand diff is empty/whitespace. Semantic NEEDS_CHANGES/REJECTED, non-turn failures, and non-empty/un-reviewed diffs still block/escalate — no masking of real failures. Walks the legal FSM edges to complete; new accepted_approved_on_exhaustion audit ledger op. Additive, idempotent, no-op on small repos and on genuinely-failing tasks. 3385 tests green.
…/ (containment) Fifth gap from the live huge-LFS run: a corrective task derailed into editing .autodev/evidence/0-drift-verifier.json (AutoDev's OWN internal state) re: its critic_drift_verifier verdict vocabulary, instead of the target repo's code, and that .autodev-only diff was accepted as task work. Root cause: collect_edit_scope_violations skips enforcement entirely when a task declares no edit_scope (the common huge-repo fallback), and nothing guarded against an agent writing to AutoDev-owned paths. Fix (execute_phase._execute_one): after the developer succeeds and BEFORE QA gates / reviewer, if the non-empty diff is confined entirely to .autodev/, emit a containment_violation_autodev_paths ledger op and route through _try_retry_or_escalate (same path as a QA-gate failure) so the task is retried/escalated/blocked and can never reach `complete`. Fires before the reviewer so an APPROVED verdict on a no-op-to-target diff can't carry it through. New _path_is_autodev_owned / _diff_confined_to_autodev helpers; ledger op registered in the Literal + _apply_op audit batch. Empty diffs (research tasks) and any diff touching target-repo code are unaffected. Scoping AutoDev internals out of corrective prompts (the upstream trigger) noted as a follow-up. Additive, idempotent. 3412 tests green.
… into correctives
Sixth gap from the live huge-LFS run (the upstream cause of the gap-5
symptom): when the critic_drift_verifier response is malformed for AutoDev's
parser, drift_verifier._parse_drift_response records AutoDev-INTERNAL plumbing
diagnostics as drift "findings" ("drift_verifier: response missing VERDICT
line", "...non-standard verdict 'PASS'..."). phase_review_runner built the
corrective_direction verbatim from those findings, so the phase review spawned
correctives to fix AutoDev's OWN drift_verifier (0.c2, 0.c3) instead of the
target GLES bug -> phase churned, never completed.
Fix: partition drift findings into AutoDev-meta (prefixes 'drift_verifier:' /
'drift_convergence_failure:') vs substantive target-repo findings
('task X: MISSING|DRIFTED', 'drift report: ...'). Correctives are built from
substantive findings only; if none remain, corrective_direction=None ->
phase review resolves as non-blocking 'skipped'. accept_phase=False is
unchanged so the verdict still drives control flow; only the leaked diagnostic
TEXT is filtered. Fail-open: unknown-prefix findings count as substantive, so
real correctives are never suppressed.
New partition_drift_findings() in drift_verifier.py. Additive, idempotent.
3427 tests green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes AutoDev run on a huge, unbuildable, Git-LFS repo with zero
.autodev/config.jsonedits and zero env vars, folding away the manual workarounds the prior dogfood run required. Each gap below was surfaced by a from-scratch run on a 358k-file / 1.17M-symbol engine repo, root-caused, fixed, and re-validated live.The 6 gaps fixed
bea89fb) — full index build 45s vs ~99 min serial; per-invocationinvocation_costledger op +.autodev/run-summary.jsonl+ a printedRun cost: $X · wall Nssummary (captures tournament spend the in-memoryplan_cost_usdbudget cap misses).00162a2) — non-sparse worktrees on an LFS repo produced ~62 MB phantom diffs that tripped the diff-size guardrail (+ 60sgit worktree addtimeouts); the huge-repo profile now auto-enablesworktree_sparse_checkout_enabledand the execute path usesworktree_huge_create_timeout_s. Probe uses a fast model (adapters.probe_model="haiku") + the unbound detect-time timeout is 10→20s, so the PONG probe stops flaking on slow cold-starts (MCP can stay enabled).index.lockcleanup (02425f4) — the impl-tournament worktree path (separate fromcreate_per_task) now shares the sparse + huge-timeout creation; a killedgit worktree addno longer cascades intogit applyrc=128 (clear_stale_index_lock).ddd296b) — a research/empty-diff task with a reviewer-APPROVED empty diff is accepted on turn-exhaustion instead of stalling atuser_decision_required(strictly gated; semantic failures still block)..autodev/containment (eeaf048) — a developer diff confined to AutoDev's own.autodev/is rejected as invalid task output before review and routed to retry/escalate.f2a380d) —drift_verifierparser diagnostics no longer leak into corrective generation (partition_drift_findingsfilters AutoDev-meta findings), so phase reviews stop spawning "fix the drift_verifier" meta-correctives and stay on the target bug.All changes are additive, idempotent, no-op on small repos, and gated by the existing
huge_repo_overrides_disabledescape hatch.Validation
A clean, fully-autonomous
init → plan → executeon a real huge-repo bug (GLES GPU-profiling timer-query + debug-group-depth lifecycle across context switches) completed end-to-end:Follow-ups (not in this PR)
_version.pybump +ci/release_preflightanchor + retrospective.plan_cost_usdbudget cap to count tournament spend.worktree_pool_enabled, default off) still non-sparse on huge repos.