fix: preserve custom apply workflows and autoplan state

[chenrui333]

Summary

• preserve targeted and generic run-only custom apply workflows while retaining durable PullStatus authorization
• derive convention-plan ownership from the complete workflow: a built-in plan or built-in apply step requires Atlantis-managed plan restoration and validation
• bind each successful Atlantis-managed generation to the SHA-256 digest of the exact persisted plan artifact
• serialize durable plan transitions and their VCS publication with a non-expiring per-pull publication claim
• restore external plan-store loading for built-in-plan/run-only-apply workflows, including re-cloned targeted applies
• treat non-convention .tfplan artifacts in a configured fully custom run-only project's path-safe subtree as user-managed
• protect active plan generations from stale completions, ordinary database writes, concurrent apply requests, and destructive import/state/unlock cleanup
• defer successful per-project plan statuses until durable generation completion, and report concurrent generations as pending
• retain sticky policy approvals across replans, failures, and process restarts
• add deterministic multi-replica and hosted regression coverage for fix: validate current head and base plan state #6605, Regression: Autoplan -> Apply fails #6641, and Regression (v0.46.0): atlantis apply fails with "plan file is missing" for custom workflows #6642

Plan-generation and artifact safety

An active generation is persisted as the existing non-applyable ErroredPlanStatus with PlanGeneration != "". BeginPlanGeneration stores that state before plan steps run and retains the project's policy state, so sticky approvals continue to follow the existing policy-hash and discard rules across replans, failures, and restarts.

Only a matching CompletePlanGeneration may clear PlanGeneration and install final plan results. For an Atlantis-managed plan, that same atomic BoltDB transaction or Redis CAS also stores the accepted generation and SHA-256 digest of the exact local bytes saved by the plan operation. S3 writes an immutable generation-and-digest-addressed object alongside the deterministic canonical object retained for legacy and generic discovery. Durable PullStatus selects the immutable object; S3 metadata records the generation and digest for diagnostics, but PullStatus remains the authorization source.

Apply loads or restores the convention plan once, compares its digest with durable PullStatus, and executes an immutable command-scoped snapshot. The command-local digest is revalidated immediately before every built-in apply step. Managed run-only apply receives the same validated snapshot through PLANFILE; fully custom run-only workflows remain status-only and do not gain artifact hashing or loading.

PullStatus written before this change may not contain a durable digest or accepted generation. Those legacy managed plans retain the historical command-start hash validation for upgrade compatibility; the next successful replan installs the durable binding. Empty legacy identity never matches an active or newer accepted generation.

Ordinary UpdatePullWithResults writes reject the entire update atomically when any targeted project has an active generation. Superseded and pull-changed commands do not overwrite newer project or aggregate statuses/comments. If a targeted replan supersedes one member of an older multi-project generation, the remaining members are atomically cancelled and made non-applyable rather than left pending.

Successful per-project plan statuses are published only after CompletePlanGeneration durably persists the matching generation. A persistence failure publishes failed project and aggregate statuses and never publishes a successful plan comment. Aggregate plan state treats an active generation as pending; a real plan error still takes priority and reports failed.

A durable per-pull publication claim spans each state-transition/VCS-publication critical section. Apply holds it from its final accepted-generation refresh through execution, fenced persistence, terminal publication, and automerge. Import, state removal, policy, approval, unlock, close, workflow hooks, and PR-backed API flows use the same ordering boundary. Ambiguous post-execution persistence or VCS errors retain the claim so a later replica cannot publish over an operation that may still complete.

Unreadable serialized PullStatus data remains fail-closed for apply, but a new plan can recover it safely: BoltDB discards the unreadable record when BeginPlanGeneration starts a fresh non-applyable generation, while Redis compares against the exact corrupt raw value and retries normal CAS conflicts. Genuine backend GET/EVAL errors still propagate.

Custom apply compatibility

Targeted atlantis apply -p <project> and plain atlantis apply support fully custom run-only workflows whose custom plan stage writes multiple .tfplan artifacts below one Atlantis project root. These workflows use durable status-only authorization and do not require or load Atlantis's convention plan.

A workflow with a built-in plan step remains Atlantis-managed even when its apply stage is run-only. Atlantis restores the convention plan from an external plan store after a re-clone, validates PullStatus head/base/project identity, accepted generation, and durable plan digest, and then executes the custom apply command. A built-in apply step likewise requires the convention plan and fails closed if a custom plan stage did not produce it.

Generic discovery excludes only non-convention artifacts owned by a configured fully custom run-only project's matching workspace and path-safe subtree. Convention names derived from runtime.GetPlanFilename are reserved: custom artifacts that collide with <workspace>.tfplan or <project>-<workspace>.tfplan retain managed-plan semantics. Sibling and ancestor paths are not claimed. Nested configured managed projects remain discoverable and retain PullStatus validation, expected plan hashes, and convention-plan protection. Custom artifacts remain user-managed; Atlantis does not validate, hash, restore, or delete their contents.

See Custom Workflows for naming, placement, generic-versus-targeted behavior, and Terragrunt examples.

Redis rolling upgrades

The persisted status value remains compatible with v0.46.0 because old replicas understand and reject ErroredPlanStatus for apply. The generation/digest contract, publication claims, and atomic write protections are new, however, so old and upgraded replicas must not overlap against the same Redis state.

Safe HA upgrade procedure:

1. Enable the global apply lock.
2. Drain and stop all old Atlantis replicas.
3. Start only upgraded replicas.
4. Release the global apply lock.

Operational recovery

Atlantis persists plan and apply results before publishing successful result comments or per-project statuses. Interrupted generations intentionally block apply and policy-result writes until a new plan supersedes them.

If PullStatus serialization is unreadable after a schema transition, running a fresh plan starts a new fail-closed generation and replaces the unreadable state. Genuinely interrupted generations can normally be superseded by replanning or safely cancelled by close/unlock after the active publisher stops.

Publication claims deliberately do not expire because an ambiguous VCS request may still complete remotely after Atlantis loses the response. If a claim is orphaned, stop and verify the owning replica cannot publish. If ownership cannot be established, stop every Atlantis replica and back up the configured database. Recovery first inspects the exact canonical UUID claim token, then atomically compare-and-deletes only that token; missing, malformed, or replaced claims are refused. Follow the documented BoltDB utility or exact Redis Lua procedure before restarting. Close waits for a held claim and is not itself a claim-recovery mechanism.

If apply, import, or state-removal execution succeeds but its durable result cannot be stored, Atlantis retains the claim. Operators must verify infrastructure/state, recover the claim offline, and run a fresh plan before retrying. See Using Atlantis.

Regression and hosted acceptance

The nested hosted fixture was squash-merged in runatlantis/atlantis-tests#21743 as eb1845fa146d1dc3805c4d4cb2a717b9372e343b. It models one root Atlantis project that creates:

• generated/dev/atlantis.tfplan
• generated/staging/atlantis.tfplan

Previous hosted acceptance for the fully custom #6642 workflow:

• Atlantis head: fe7938d3c73a3e68340d34ced4097df90c97dea6
• Tester run 30014671712
• e2e-github job 89231426838
• e2e-gitlab job 89231461938
• custom-plan-path-apply passed using plain atlantis apply
• both descendant plans were created and preserved through apply
• only the root Atlantis project was applied
• no convention-managed plan was created at the project root

For #6641, deterministic tests keep plan success behind final PullStatus persistence and immediately apply the persisted autoplan without a manual plan. The reporter's exact Gitea persistence-failure trigger has not been isolated, so this PR continues to address rather than close that issue.

References

• Addresses Regression: Autoplan -> Apply fails #6641
• Closes Regression (v0.46.0): atlantis apply fails with "plan file is missing" for custom workflows #6642
• Hardening regression: fix: validate current head and base plan state #6605
• Original fixture coverage: test: add v0.46.0 custom plan-path apply regressions atlantis-tests#21611
• Replan and generation fixture coverage: test: expand plan generation and replan regression fixtures atlantis-tests#21632
• Nested-artifact fixture coverage: test: cover nested custom plan artifacts atlantis-tests#21743

[github-code-quality]

Code Coverage Overview

Languages: Go

Go / code-coverage/go

The overall coverage in commit 3d899b4 in the fix/v046-apply-regre... branch is 73%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.

File	main	fix/v046-apply-regre... 3d899b4	+/-
server/events/v...ithub/client.go	—	88%	—
server/events/p...mand_builder.go	—	83%	—
server/events/event_parser.go	—	83%	—
server/events/p...mmand_runner.go	—	82%	—
server/events/v...itlab/client.go	—	80%	—
server/core/boltdb/boltdb.go	—	79%	—
server/events/p...mmand_runner.go	—	77%	—
server/controll...i_controller.go	—	76%	—
server/core/redis/redis.go	—	75%	—
server/server.go	—	69%	—

---

_{Updated August 17, 2026 05:08 UTC}

[Copilot]

Pull request overview

This PR fixes regressions in the plan→apply lifecycle introduced by stricter apply-time validation by (a) scoping “convention planfile” validation to Atlantis-managed apply steps so custom apply workflows can run, and (b) ensuring plan results are persisted before publishing success signals (comments/statuses). It also expands test coverage via a Gitea identity round-trip integration test and new GitHub E2E plan-then-apply scenarios.

Changes:

• Gate planfile/hash validation on presence of a built-in apply step to preserve run-only custom apply workflows.
• Persist PullStatus before publishing autoplan/manual plan success, and fail closed with an actionable error path on persistence failures.
• Extend E2E harness with an explicit plan-then-apply scenario and supporting lifecycle utilities; add a Gitea PullStatus round-trip integration test.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
server/events/project_command_runner.go	Restricts apply-time plan validation/hash capture to workflows containing an Atlantis-managed apply step.
server/events/project_command_runner_test.go	Adds unit coverage for run-only apply, mixed apply workflows, and multiple managed apply-step revalidation.
server/events/project_command_builder.go	Skips expected plan-hash capture for apply commands that have no Atlantis-managed apply step.
server/events/plan_command_runner.go	Persists PullStatus before publishing autoplan/manual plan success; adds persistence-failure handling helper.
server/events/gitea_pull_status_integration_test.go	New integration test ensuring Gitea webhook/API identity round trip can retrieve autoplan PullStatus and build immediate apply.
server/events/command_runner_test.go	Adds tests asserting autoplan/plan fail-closed behavior when PullStatus persistence fails and success isn’t published early.
e2e/vcs.go	Generalizes per-project status querying to be command-specific (plan vs apply).
e2e/testcase.go	Introduces ScenarioPlanThenApply and new expectations for apply status contexts and apply comment markers.
e2e/README.md	Documents lifecycle scenarios and clarifies fixture repo usage and diagnostics behavior.
e2e/on_apply_lock.go	Refactors lock-preservation scenario to reuse shared lifecycle helpers; updates project status assertions to be command-specific.
e2e/on_apply_lock_test.go	Adds tests for new status-prefix helper and new-comment detection; asserts plan-then-apply cases are explicit.
e2e/lifecycle.go	New shared lifecycle utilities for fixture PR creation, git operations, mutation, and cleanup.
e2e/gitlab.go	Updates GetProjectStatuses signature (still unsupported on GitLab).
e2e/github.go	Updates per-project status filtering to use a command-specific status prefix.
e2e/e2e.go	Refactors plan-only flow into shared lifecycle helpers; adds plan-then-apply runner with stale-result rejection and richer diagnostics.

[chenrui333]

Follow-up review finding addressed in the latest commits:

• replanning now persists a non-applyable generation before any built-in or custom plan step can replace artifacts
• BoltDB transactions and Redis compare-and-swap writes preserve unrelated projects and prevent stale completion writes from restoring an older plan
• a failed final write leaves the generation non-applyable, so neither built-in nor run-only custom apply can consume its artifacts
• cleanup is secondary and generation-scoped: only managed plans and generation-created, pull-owned plan locks are considered, and a superseding generation is left untouched
• deterministic tests cover built-in/custom failed replans, partial multi-project updates, start failures, cleanup success/failure, superseding generations, and autoplan success visibility before immediate apply

The active replan and mixed-mutation fixtures are now on runatlantis/atlantis-tests main via runatlantis/atlantis-tests#21632. This PR remains open and unmerged.

[jamengual]

Review

Overall this is a carefully engineered change — fail-closed persistence everywhere, side-effect-free apply validation, solid path-traversal guards in repoRelDirWithinRoot, and the ErroredPlanStatus + PlanGeneration (json:",omitempty") encoding is a clever way to keep v0.46.0 replicas rejecting in-progress plans without a schema bump. Test coverage is extensive (BoltDB + Redis generation suites, builder internal tests, Gitea persistence-failure integration test, hosted e2e run). A few findings below.

High — plan -p <project> matching zero projects fails with a confusing error

In plan_command_runner.go run(), beginPlanGeneration only runs when len(projectCmds) > 0, but the final persistence branch is:

} else if len(projectCmds) == 0 && !cmd.IsForSpecificProject() {
    pullStatus, err = p.dbUpdater.replaceDB(ctx, pull, result.ProjectResults)
} else {
    pullStatus, err = p.dbUpdater.completePlanGeneration(pull, planGeneration, result.ProjectResults)
}

A specific-project plan that builds zero commands with --silence-no-projects off (the default) reaches the else arm with planGeneration == "". CompletePlanGeneration rejects the empty generation, so the user gets failed Plan and Apply commit statuses plus a comment like persisting plan results: plan generation is empty joined with cleaning failed plan generation: ... pull status is missing. Previously this path fell into updateDB's empty-results branch and returned the existing status quietly.

Suggest guarding the else arm (fall back to updateDB when planGeneration == "", or return the no-projects result earlier) and adding a test for plan -p unknown-project with SilenceNoProjects=false.

Medium — unreadable pull status now permanently blocks planning

BeginPlanGeneration fails hard on an undecodable status blob (BoltDB propagates the getPullFromBucket error; Redis uses tolerateUnreadable=false). Since every plan and autoplan begins a generation first, one corrupt/incompatible blob makes all planning on that PR fail until the PR is closed and reopened. The old UpdatePullWithResults path logged a warning and overwrote. Fail-closed is defensible, but consider tolerating unreadable state in BeginPlanGeneration (it rebuilds the tracked projects anyway) or documenting the recovery path.

Medium — stale generation after a crash blocks unrelated writes on that project

A crash between Begin and Complete leaves PlanGeneration set. Apply being blocked with "run atlantis plan again" is the intended fail-closed behavior, but UpdatePullWithResults also rejects the entire batch atomically when any targeted project holds an active generation, so policy-check or approve_policies writes touching that project error out until a replan. On a new commit, rejectAnyActivePlanGeneration over an outdated status has the same effect. It self-heals because autoplan's Begin overwrites the stale marker, but a docs note would help operators who hit the error message first.

Low

• Redis updatePullAtomically: 32 CAS attempts with no backoff/jitter — under contention this spins hot GET+EVAL. A small jittered sleep would be cheap insurance.
• locks_controller.go: the new failure comment says "durable plan state changed", but UpdateProjectStatus can also fail on transient DB errors, so the wording can mislead. The "do not apply" instruction is right either way.
• Comment ordering: updatePull moved after DB persistence in four runners. Deliberate and more honest, but user-visible (comments post later; a crash between write and comment leaves state changed with no comment) — worth a changelog line.
• When applies succeed but the DB write fails, the whole result is now marked errored even though infrastructure was applied. The comment wording could state explicitly that applies may have executed.
• PR scope: the ~1,300-line e2e harness rewrite (clean code — context-aware cleanup, errors.Join, slices.Backward teardown) doubles the review surface; splitting it out would have made this easier to review.

Verified during review

• completePlanGenerationResults synthesizes error results for projects skipped by failed execution-order groups, so the completeness check in CompletePlanGeneration cannot wedge.
• cleanupFailedPlanGeneration re-reads PullStatus and only deletes artifacts still owned by the failing generation, protecting a newer command's plans; lock cleanup respects the pre-existing-lock snapshot.
• hasRunOnlyApplySteps returns false when any apply step exists, so mixed workflows keep full convention validation, consistent with the setExpectedPlanHashes skip.
• filterNonConventionRunOnlyPlanArtifacts only swallows artifacts inside a matching workspace and subtree; shared-prefix siblings (platform vs platform-other) are covered by tests.
• google/uuid is already a direct dependency, and the only production caller of UpdateProjectStatus handles the new rejection.

Verdict: approve after the High item — the empty-generation path in run() is a real regression for default configs and needs a guard plus a test. The two Mediums read as intentional trade-offs that deserve an explicit maintainer decision and a docs mention rather than blockers.

[Copilot]

Pull request overview

Copilot reviewed 36 out of 37 changed files in this pull request and generated 2 comments.

Files not reviewed (1)

• server/core/db/mocks/mock_database.go: Generated file

[Copilot]

Pull request overview

Copilot reviewed 36 out of 37 changed files in this pull request and generated no new comments.

Files not reviewed (1)

• server/core/db/mocks/mock_database.go: Generated file

[chenrui333]

@jamengual Thanks for the careful review. Addressed on fe7938d3:

• Fixed the High regression: a targeted plan -p <unknown-project> with --silence-no-projects=false now uses the ordinary empty-result update instead of completing an empty generation. The regression test preserves the existing PullStatus and verifies that Plan/Apply are not marked failed.
• Kept unreadable PullStatus and active plan-generation writes fail-closed. Relaxing either would allow an older or incompatible writer to overwrite newer durable state. The recovery paths are now documented: replan after an interrupted generation, or close/reopen after Atlantis processes the close event for unreadable state.
• Added 1-10 ms jitter after Redis CAS conflicts.
• Made the lock-deletion warning database-neutral.
• Apply-result persistence failures now say that apply steps may already have executed and operators should verify infrastructure before retrying.
• Documented that durable state is written before result comments. CHANGELOG.md is explicitly retired, so the operational note lives in runatlantis.io/docs/using-atlantis.md and the PR body instead.

I kept the E2E harness in this PR. It is already integrated with the merged hosted fixture and supplies the final-head acceptance for the custom plan-path and replan lifecycle behavior; splitting it now would separate the implementation from its required proof.

Final head fe7938d3 has 39 passing checks, including tester run 30014671712, e2e-github, and e2e-gitlab.

[pseudomorph]

@chenrui333 - Thanks for working on this.

Couple things:

• Was dropping ensurePlanLoaded for mixed workflows (managed plan step, run-only apply) deliberate? with an external plan store, atlantis apply -p on a re-cloned replica now fails on a missing plan file where main restored it.
• BeginPlanGeneration returns the read error on an unreadable PullStatus record where the old plan path discarded and overwrote it. The next schema change blocks replans until close/reopen. Restore the overwrite behavior for the begin path, or call out why fail-closed is preferred here.
• Per-project plan success status publishes before the DB write. project_command_runner.go#L231 fires as soon as terraform finishes, before completePlanGeneration persists, so a crash in between leaves a green atlantis/plan: <project> while the DB still holds the active generation and rejects apply. you already solved this for apply with the early return at #L227-L229 plus deferred publishing. Would it make sense to add plan to that early return and publish its success statuses after the DB write, same as apply?
• A run-only project with ErroredPolicyCheckStatus gets "plan file is missing; run atlantis plan" when the remedy is approve_policies. statusAllowedForApplyExecution rejects the status so project_command_builder.go#L1424 drops it from planlessProjects, while statusRequiresPlanFileForGenericApply includes it, so validation falls through to the missing plan file error at #L1747.
• Run-only apply validation only runs when the validator happens to implement a second interface p.ApplyPlanValidator is typed as ApplyPlanValidator, and the run-only branch reaches ValidateProjectPlanStatus through the type assertion at project_command_runner.go#L950, so any implementation without that method compiles fine and run-only applies skip PullStatus/staleness checks with no error or log. add ValidateProjectPlanStatus to the ApplyPlanValidator interface so the compiler enforces it.
• plain apply hard-fails when a run-only workflow writes an artifact that collides with the convention name, <workspace>.tfplan or <project>-<workspace>.tfplan per pendingPlanUsesConventionPath, or writes outside the project's subtree so filterNonConventionRunOnlyPlanArtifacts doesn't claim it. Either way the leftover artifact fails validation for every project in the PR. document both naming/placement constraints, since terragrunt setups like Regression (v0.46.0): atlantis apply fails with "plan file is missing" for custom workflows #6642 -out ../../../../atlantis.tfplan are one rename away from tripping them.
• A successful targeted plan reports the combined atlantis/plan status as failed whenever any other project on the pull is mid-plan, because plan_command_runner.go#L608 counts PlanGeneration != "" as errored; it self-corrects when the other plan completes. count in-flight generations as pending instead.
• A test case missing ApplyCommand returns nil, err at e2e/e2e.go#L132 and the caller assigns into the nil result, panicking the whole e2e run instead of failing one case. Return a non-nil result there. apply project statuses also skip the extras check, forbidExtra is hardcoded false at e2e/e2e.go#L225, so a targeted apply that also applied an unrelated project would pass; wire ForbidExtraProjectStatuses through.

[chenrui333]

Addressed all eight review points in the commits below. Current head is dbc6c3ad3832f83291f97b179eaa3177f3f403dc; its two final docs-only commits fix CI spell-check wording and a pre-existing Gitea documentation link that now returns 404.

1. Managed plan + run-only apply restore — 7e539ed8 derives convention-plan ownership from the full workflow and restores external plans before run-only apply. Primary regression: TestDefaultProjectCommandBuilder_TargetedManagedPlanRunOnlyApplyReclonesWithExternalStore.
2. Unreadable PullStatus recovery — 1aca8373 lets BoltDB discard unreadable serialized state and makes Redis replace it through exact-raw-value CAS without swallowing backend errors. Primary regressions: TestBeginPlanGenerationOverwritesCorruptData in both stores and TestBeginPlanGenerationRetriesWhenCorruptRawValueChangesBeforeCAS.
3. Deferred per-project plan success — 7e539ed8 publishes success only after matching CompletePlanGeneration; persistence failure publishes failed statuses and no success comment. Primary regressions: TestPlanCommandRunner_AutoplanPersistsPullStatusBeforePublishingSuccess, TestPlanCommandRunner_AutoplanFinalPersistenceFailureLeavesGenerationNonApplyable, and TestPlanCommandRunner_ManualPlanPersistenceFailureFailsClosed.
4. Run-only policy remediation — 7e539ed8 keeps plan ownership separate from execution eligibility and directs the user to atlantis approve_policies without executing apply. Primary regressions: TestDefaultProjectCommandBuilder_BuildMultiApply_RunOnlyWorkflowWithoutManagedPlan and TestProjectCommandRunner_RunOnlyApplyReportsPolicyApprovalRemediation.
5. Compile-enforced validator interface — 7e539ed8 makes ValidateProjectPlanStatus mandatory on ApplyPlanValidator and removes the optional runtime assertion. Implementations and generated mocks were updated; make go-generate is clean.
6. Custom-artifact constraints — deaeeb1 documents reserved convention filenames, same-workspace/path-safe subtree containment, targeted-vs-generic behavior, and safe Terragrunt examples in custom-workflows.md; markdown lint and website build pass.
7. In-flight aggregate status — 7e539ed8 reports active generations as pending, with actual plan errors taking priority. Primary regression: TestPlanUpdatePlanCommitStatus covers pending, success, all-in-flight, and error-plus-pending cases.
8. E2E nil result and unexpected apply statuses — 5dd612a returns a nonnil E2EResult for missing ApplyCommand and honors ForbidExtraProjectStatuses during apply. Primary regressions: TestPlanApplyLifecycleMissingApplyCommandReturnsResult and TestAssertProjectStatusesForbidsUnexpectedContexts.

The branch is rebased onto origin/main 9d81ac26482788a651e206c5fce5ff420116ccae. Local final-head validation includes make test-all, make check-fmt, make lint, make build-service, go vet ./..., focused race tests, storage corruption/CAS tests at -count=10, E2E module tests/lint, markdown lint, and the website build.

[jamengual]

Review: durable plan state / apply regressions

Reviewed this against main across four areas (plan store + db backends, events command runners, runtime step runners, e2e harness). go build, go vet, golangci-lint are clean and the full unit suite passes under -race — the issues below are liveness, durability and silent-drop behavior, which the test suite structurally can't reach.

Scope note: I've deliberately excluded anything that already exists on main (e.g. DeleteForPull swallowing S3 delete errors, the IsTruncated/nil-continuation-token pagination loops). Everything below is introduced or regressed by this PR.

Cleared first — the two things I most expected to be broken are not:

• No apply-without-plan hole. requiresAtlantisManagedPlanFile (project_command_context_builder.go:338) is true if either the plan workflow has a built-in plan step or the apply workflow has a built-in apply step, evaluated against the fully-merged workflow, so the default workflow always takes the hash-validating path. hasAtlantisManagedApplyStep (project_command_runner.go:1196) mirrors workflow.Apply.Steps exactly, so run-only, mixed run+apply and policy_check shapes all classify consistently. Run-only projects skip file validation but still pass ValidateProjectPlanStatus. Upgrade path is sound — legacy rows have ManagedPlanHash == "" and "" == "" generation comparison.
• No prefix bug in descendant-artifact matching. repoRelDirWithinRoot (project_command_builder.go:1478) behaves correctly: ("a","ab") → false, ("a","ab/c") → false, ("a","a/b") → true, ("a/b","a") → false.

The blocking issues cluster around one design decision and one pattern.

---

CRITICAL

C1. The plan publication claim has no TTL and its waiters spin forever

Two halves that combine badly.

No expiry. server/core/redis/redis.go:81 — the acquire script is a bare SET, no EX:

local current = redis.call("GET", KEYS[1])
if not current then redis.call("SET", KEYS[1], ARGV[1]); return 1 end
if current == ARGV[1] then return 1 end
return 0

Same in server/core/boltdb/boltdb.go:572. No lease, no heartbeat, no owner identity, no deadline.

Unbounded, uncancellable waiters. server/events/db_updater.go:28-39, server/controllers/api_controller.go:1246-1258, and server/events/pull_closed_executor.go:80-89 all have this shape:

for {
    err := c.Database.AcquirePlanPublicationClaim(pull, token)
    if err == nil { return token, nil }
    if !errors.Is(err, db.ErrPlanPublicationBusy) { return "", err }
    time.Sleep(25 * time.Millisecond)
}

No context, no deadline, no attempt cap. Reached from HTTP handlers (api_controller.go:868,922,970,1280), policy_check_command_runner.go:61, plan_command_runner.go:196,381.

And several paths deliberately never release — finishPlanPublicationClaim (api_controller.go:1260-1267), command_runner.go:560,582 ("retaining plan publication claim after ambiguous publication").

Failure scenario: a pod is OOM-killed mid-plan, or GitHub 422s a single commit status. The claim persists forever. Every subsequent plan / policy_check / API call on that PR blocks forever, leaking a goroutine and an HTTP connection each. On BoltDB every 25ms iteration is a db.Update write transaction with fsync against the global mutex — roughly 40 serialized fsyncs/sec per stuck request. Because CleanUpPull has the same loop, every PR-close webhook and every GitHub redelivery of it leaks another permanently spinning goroutine.

Recovery is cmd/plan-claim-recovery, which is BoltDB-only — Redis deployments have no recovery path short of manually DEL-ing the key.

Worth noting this PR also removed the natural self-heal: see H7.

Suggested direction: lease the claim with holder renewal (Redis SET … EX + refresh, stored deadline for Bolt), and bound every wait by the command context plus a max wait, returning 409 rather than spinning.

C2. Commands are silently dropped while the claim is held — and apply holds it for the entire terraform apply

apply_command_runner.go:100   acquire   ← start of Run
apply_command_runner.go:110   finish    ← after execution completes
plan_command_runner.go:116    acquire
plan_command_runner.go:204    finish    ← released before execution

The plan runner releases before executing. The apply runner holds the claim across the whole apply. For that entire duration, every autoplan from a push and every comment command on the PR hits a busy claim and returns with nothing but a log line:

// plan_command_runner.go:114-119
publicationToken, err := p.acquirePlanPublicationClaim(ctx)
if err != nil {
    return
}

No commit status, no PR comment. Same shape at command_runner.go:538-545 and apply_command_runner.go:100-103.

On main these were serialized by workingDirLocker and produced a visible "workspace is currently locked" comment. This replaces a user-visible interlock with a silent drop — from the user's side it is indistinguishable from Atlantis never receiving the webhook.

---

HIGH

H1. A transient VCS status failure permanently wedges the PR

api_controller.go:1077-1081 (same pattern at 751, 769, 829, 836, 882, 946, 981) — finishClaim(err) returns without releasing when updating the pending apply status fails, i.e. before any apply work has happened. There's no retry/backoff around CommitStatusUpdater. This is the most likely real-world trigger for C1. Suggest releasing the claim for pre-execution failures and retaining only where durable state was actually mutated.

H2. Redis DeletePullStatus is a non-atomic two-step delete that can orphan a claim unrecoverably

redis.go:496-507 does deletePull(key) then ForceClearPlanPublicationClaim(pull) as separate round trips. A crash between them leaves a claim key with no pull status — and ForceClear is only ever called from DeletePullStatus, which already ran, so nothing will ever clear it. BoltDB does both in a single db.Update tx (boltdb.go:1053-1065). A single Lua script would fix it; the keys already hash to the same slot.

H3. Project locks are never released when a PR update removes all projects

plan_command_runner.go:427-441 — clearPlansAndPullStatusForNoProjects now only calls beginPlanGeneration(..., nil, true, ...). On main (plan_command_runner.go:383-392) it called deletePlansAndPendingPlanLocks, which discovered pending plans and released their project locks.

deletePlansAndPendingPlanLocks, deletePlanLocksForPendingPlans, deletePlansAndPlanLocks, deletePlans and deletePlanLocks are now dead in production (plan_command_runner.go:794-864; only the internal test at :216 calls them).

Atlantis project locks are cross-PR, so a push that removes the last TF dir now blocks every other PR touching that dir until PR close or explicit unlock — and per C1, CleanUpPull may never run.

H4. applyResultStatusUpdateError false-positives on DB-excluded results

apply_command_runner.go:439-453 requires every errored project result to have a matching ErroredApplyStatus row, but filterProjectResultsForDB (db_updater.go:132-136) drops DirNotExistErr before persistence, and doApply returns DirNotExistErr at project_command_runner.go:1057. Reproduced:

apply result for dir "gone" workspace "default" project "" has errors
but no matching errored apply status was recorded

apply_command_runner.go:302 assigns this to result.Error, so the renderer shows it as the top-level command error. A user who deletes a directory and runs atlantis apply gets this internal message instead of "directory does not exist" — and in a mixed run it suppresses the successful projects' output. errStaleCommandHead is short-circuited at line 433; DirNotExistErr isn't.

H5. Obsolete-generation apply branch leaves per-project checks Pending after infrastructure has already changed

apply_command_runner.go:277-282 is the only post-execution exit that calls neither publishDeferredApplyStatuses nor updatePull nor updateCommitStatus. Apply terminal statuses are now deferred (project_command_runner.go:229-231) while the pending status is still published at command start, so atlantis/apply: dir/workspace stays Pending indefinitely.

Reachable path: a generic apply over project A (applyable) and B (ErroredPolicyCheckStatus with a stale AcceptedPlanGeneration). B escapes isPolicyBlockedApplyResult (apply_command_runner.go:479-487) so it isn't filtered, and ValidateApplyResultsForPlanGeneration (db/plan_generation.go:286-294) rejects ErroredPolicyCheckStatus → ErrPlanGenerationSuperseded. A's terraform apply has already succeeded at that point, but the result is discarded, the DB still says Planned, no comment is posted, the check is stuck pending, and the user can re-apply A.

H6. UI plan discard silently no-ops

locks_controller.go:176-224 — the discard now only happens inside if len(matchingProjects) == 1. On main UpdateProjectStatus(..., DiscardedPlanStatus) was called unconditionally.

When pullStatus == nil, or workspace/dir/name normalization doesn't match, the lock is deleted, the PR is commented "the plan … was discarded", and 200 is returned — but durable status stays Planned. statusAllowedForApplyExecution permits PlannedPlanStatus, and the .tfplan is no longer deleted after apply (H8), so a later atlantis apply applies the plan the user believes was discarded. This should be a 409 rather than a success response.

H7. DeletePullStatus is now dead code

main has two production callers (db_updater.go:85, pull_closed_executor.go:122); this branch has zero. Neither backend has a TTL, so every PR ever opened leaves a permanent pull-status record plus a claim key. redis.go:507 was also the natural self-heal for C1.

H8. All three PlanStore.Remove call sites were removed — plan artifacts are never deleted after apply

main had:

// apply_step_runner.go:76
if err == nil {
    ctx.Log.Info("apply successful, deleting planfile")
    if removeErr := a.PlanStore.Remove(ctx, planPath); removeErr != nil { ... }
}

plus the equivalents at import_step_runner.go:52 and state_rm_step_runner.go:52. All three are gone. The only remaining removal server-wide is project_command_runner.go:1188, for synthetic non-PR API applies. LocalPlanStore.Remove additionally early-returns nil for durable PR-backed plans (plan_store.go:75), and delete_lock_command.go no longer calls WorkingDir.DeletePlan / DeletePlanForProject / DeleteForPull.

Net effect: for PR-backed workflows the .tfplan stays on disk and the S3 object stays in the bucket after a successful apply, import or state rm. Plan files contain resolved variable values and prior state. Only pull_closed_executor.go:131 reaps anything now. If a durable lifecycle boundary is intended to cover this, it'd help to point at it explicitly — I couldn't find one on this branch.

---

MEDIUM

• Snapshot integrity fails open. apply_step_runner.go:45-50,71-73 — both the hash gate and the snapshot are gated on ctx.ExpectedPlanHash != "". When empty, executionPlanPath = planPath and terraform reads the mutable on-disk file directly, which is the exact TOCTOU this PR closes. In practice server.go always wires DefaultApplyPlanValidator, but this should error when RequiresAtlantisManagedPlanFile && ExpectedPlanHash == "" rather than fall through. (The validated path itself is correct: bytes read once → hashed → snapshot written from memory → chmod 0400 → terraform pointed at the snapshot. No re-read window.)
• The 0400 hardening is defeated by its own source file. The snapshot is 0400, but the source plan is created with os.Create (0644 after typical umask) at s3_plan_store.go:290, sitting right next to it.
• Managed plan hash is recorded before the durable save succeeds. plan_step_runner.go:126-138 sets *ctx.GeneratedPlanHash = planHash before PlanStore.Save, and project_command_runner.go:504-517 copies it to ManagedPlanHash unconditionally. On S3 upload failure, durable state can record a hash for an artifact that was never persisted. Bounded by statusAllowedForApplyExecution rejecting ErroredPlanStatus, but the ordering should be inverted. Also hashPlanFile and Save read the file independently, so the digest and the uploaded bytes aren't provably the same object.
• $PLANFILE silently changes shape for custom apply steps. run_step_runner.go:59-62,76 — for managed-plan projects with a custom apply, PLANFILE now points at .atlantis-validated-plan-<rand> (0400, no .tfplan extension) instead of <workspace>.tfplan. Workflows that write to, rename, or extension-match $PLANFILE will break. The docs added here cover custom artifact naming but never mention this.
• The snapshot only binds scripts that actually read $PLANFILE. The original <workspace>.tfplan remains in the project dir, so a custom apply step that hardcodes the filename or globs *.tfplan bypasses the hash binding entirely. Worth making a documented hard requirement, since the security property degrades silently. The snapshot is also only removed by defer — a crash leaves a read-only .atlantis-validated-plan-* behind in the working dir.
• Retain-claim is applied to too broad an error set. The policy is defensible for genuinely ambiguous terminal writes, but it currently catches DiscardReviews failures (plan_command_runner.go:272-278 — GitHub 422s on already-dismissed reviews, previously log-and-continue), failure to set the pending status before any work runs (plan_command_runner.go:352-358, apply_command_runner.go:232-235), and HidePrevCommandComments (pull_updater.go:36, cosmetic and routinely flaky). Any one of these wedges the PR per C1.
• DeleteLock returns 200 while permanently retaining the claim. locks_controller.go:242-245 — a failed courtesy PR comment (archived/locked/deleted PR, comment API 5xx) sets retainClaim = true. The lock is already deleted and the discard already committed, so retention buys nothing, and the operator gets a 200 with no signal that manual recovery is now required.
• claimTokens ...string makes a forgotten fence token a compile-time no-op. db.go:34-40 across six interface methods; PlanPublicationClaimToken (plan_generation.go:88) returns ("", nil) for zero args and ValidatePlanPublicationClaim("", "") returns nil. Any caller that omits the token mutates durable state unfenced whenever no claim exists. A required parameter with an explicit NoClaim sentinel for legacy callers would be safer.
• The "no status" sentinel is a real status. plan_generation.go:180,186,191,284,290 use models.ProjectPlanStatus(0), which is ErroredPlanStatus (models.go:858). Currently write-only, so latent — but an unknown-status project would report as errored if these were ever consumed.
• UpdateDiscardResultsForPlanGeneration overwrites terminal apply history. boltdb.go:1013 / redis.go:944 set DiscardedPlanStatus unconditionally, and ValidateDiscardResultsForPlanGeneration (plan_generation.go:258-266) accepts AppliedPlanStatus as a valid prior state with command.Unlock as a valid command — so an unlock after a successful apply rewrites Applied → Discarded and loses the fact the apply happened.
• Database.UpdateProjectStatus is now dead. Its only production caller on main was locks_controller.DeleteLock, which this PR replaced. It's also the only mutation method that doesn't validate the publication claim.
• dir: . run-only projects swallow repo-wide artifacts. filterNonConventionRunOnlyPlanArtifacts (project_command_builder.go:1458) — reachable damage is narrow and fails safe as "plan file is missing for dir X", but the diagnostic points at the wrong cause.
• Potential nil deref on noProjectBeginResult. plan_command_runner.go:373-387 — the guard is len(projectCmds)==0 && !cmd.IsForSpecificProject(), but projectCmds was reassigned by partitionProjectCmds (line 339) after noProjectBeginResult was set against the pre-partition slice. Cheap to make defensive.
• completePlanGenerationResults misattributes missing results. plan_command_runner.go:569-580 synthesizes "plan skipped because an earlier execution-order group failed" for any project context lacking a result, including cancellation via cancellationTracker.

---

On the failing e2e-github job

Not caused by this PR. Evidence:

• The server received 23 opened vs 28 closed pull-request events. The 5 missing opened events correspond exactly to the 5 failing cases.
• Pulls 23105 (builtin-autoplan-apply) and 23114 (custom-plan-path-apply) have zero server log lines — not even Handling GitHub Pull Request 'opened' event. 23126 (builtin-replan-apply) received only the closed event.
• standalone-with-workspace got its opened event 62s late; the plan then succeeded 3s after the harness had already given up.
• output-long-line failed on project is currently locked by an unapplied plan from pull #23123 — another concurrent CI run's PR.
• The server processed pulls 23102–23125, far more than this run created, because .github/workflows/test.yml:18-20 keys concurrency on head_ref, so concurrent e2e runs each register their own webhook on the shared runatlantis/atlantis-tests repo and every server receives every run's events.
• main is failing the same way — run 32060968318, both e2e-github and e2e-gitlab, on standalone / multi-project-single / multi-project-fanout.

I also checked the obvious code-side suspect: the silent claim-acquire return in C2 would produce exactly this "plan never started" symptom. It's ruled out — there are zero claim-related lines in the server log for the whole run, and the affected pulls produced no log lines at all.

However: three of the new lifecycle cases never got a plan started, so they verified nothing this run. The new apply/replan paths are still unproven end-to-end.

Two harness observations that would help here, both independent of this PR's logic:

• A repo-global concurrency group on the e2e jobs (group: e2e-github-atlantis-tests, cancel-in-progress: false) would remove webhook fan-out, shared-ngrok contention, cross-PR lock contention and cross-server commit-status clobbering in one change.
• GitHub doesn't retry a failed webhook delivery, and the harness has no recovery. Posting atlantis plan as a comment when the aggregate status is still empty after ~10 polls would turn a permanently lost delivery into a ~20s delay. The machinery already exists in postAtlantisCommandAndWaitForExpectedState.

Also worth flagging: cancel-in-progress: true means a superseded job is SIGKILLed and the defer vcsClient.DeleteAtlantisHook at e2e/main.go:129 never runs (the error is nolint:errcheck'd too). GitHub caps webhooks at 20 per repo, after which CreateAtlantisWebhook fails and log.Fatalf at main.go:64 would break all future e2e runs.

---

Summary

The blocking set is dominated by one design decision: an exclusive, TTL-less publication claim on a critical path, whose waiters spin uncancellably and whose only recovery is a BoltDB-only offline CLI. C1, C2, H1, H2, H6, H7 and the retain-claim breadth are all facets of it. Giving the claim a lease with heartbeat, bounding every wait by the command context, and releasing on pre-execution failures would collapse most of them.

The second theme is independent: cleanup call sites removed without replacement — PlanStore.Remove ×3 (H8), deletePlansAndPendingPlanLocks (H3), DeletePullStatus (H7), and the artifact reaping in delete_lock_command.go. Each is a regression against main, and each leaks either plan files on disk and in S3, cross-PR project locks, or unbounded DB rows.

The direction of the PR — binding plan generations to durable artifacts and hash-validating what apply consumes — is sound, and the validated-snapshot path itself is well built. It's the claim lifecycle and the removed cleanup paths that need work before this is safe to merge.

[jamengual]

Follow-up: two pre-existing S3 issues that this PR newly exposes

Deliberately separated from my review above, which covered only code this PR changes. The two below are older code that this PR doesn't touch — but the new plan-generation machinery is what makes them reachable, so they're worth deciding on here rather than filing away.

---

1. PlanStore.Save is unfenced, so a superseded generation can overwrite the canonical object

s3_plan_store.go:123 — Save has the same signature as on main and writes the deterministic canonical key with no conditional put. There's no IfMatch, IfNoneMatch or ETag check anywhere in the store, and no comparison against the current generation. Generation is carried as metadata ("plan-generation"), not as a write precondition.

The function was always unfenced. What's new is that plan generations now exist on top of it:

1. Generation G1 for project P is in flight.
2. A new plan starts G2, superseding G1 in the DB (db/plan_generation.go:307).
3. G2 completes, writes canonical key K plus the immutable K.atlantis-managed/G2/h2, and records ManagedPlanHash = h2.
4. The slower G1 worker finishes and overwrites K with h1 bytes. Its CompletePlanGeneration is correctly rejected as superseded — but the S3 write already landed.

Managed-plan projects mostly survive this, because Load prefers the immutable key and DefaultApplyPlanValidator re-hashes. Two paths don't:

• RestorePlans (s3_plan_store.go:394) filters on strings.HasSuffix(key, ".tfplan"), which excludes the immutable …​.atlantis-managed/<gen>/<hash> objects. It restores the canonical object only — i.e. the one that just got clobbered.
• Custom-plan-file / non-managed projects take the !ctx.RequiresAtlantisManagedPlanFile branch at s3_plan_store.go:139, which writes the canonical key with no digest binding at all.

A generation-conditional write (or an ETag precondition) on the canonical key would close it.

---

2. Non-PR API plans share a single S3 key, and the new comments assert the opposite

s3Key (s3_plan_store.go:537) interpolates strconv.Itoa(ctx.Pull.Num), which is pre-existing. isSyntheticNonPRAPI (s3_plan_store.go:304) is new:

func isSyntheticNonPRAPI(ctx command.ProjectContext) bool {
	return ctx.API && ctx.Pull.Num <= 0
}

For these contexts Pull.Num is 0, so every non-PR API plan for a given repo/workspace/dir lands on the same key:

<prefix>/<owner>/<repo>/0/<workspace>/<repoRelDir>/<planfile>.tfplan

These contexts also bypass digest and generation binding (s3_plan_store.go:152, :237), and they're the only contexts Remove still deletes (:322). So two concurrent POST /api/plan calls against the same repo+dir+workspace can overwrite or delete each other's artifact. The head-commit metadata check only helps when the refs differ.

The part that's new and worth fixing regardless of whether you fix the key: the comments added in this PR state the opposite of what the code does.

// s3_plan_store.go:317
// Remove deletes legacy/hashless artifacts and uniquely scoped synthetic
// non-PR API artifacts. ...

// plan_store.go:71
// Remove deletes legacy/hashless local plans and uniquely scoped synthetic
// non-PR API plans. ...

"Uniquely scoped" is precisely what these keys are not. Since Remove's safety argument rests on that claim, the comment is load-bearing and currently misleading. A per-request nonce in the key for synthetic contexts would make it true; otherwise the comments should be corrected.

---

Neither of these blocks the PR on its own — flagging them because the generation work is what turns them from dormant into reachable, and #2 is a two-line comment fix either way.

[jamengual]

Heads up: the #6642 regression fix is extracted into #6781

@chenrui333 — following up on my review above. Since you're short on time on this one, I've pulled the #6642 regression fix out of this PR into #6781 so it can ship and be backported to 0.46.x on its own. No action needed from you; I wanted you to know before you spend more time here.

Why split rather than push on this branch. Tracing #6642: apply_plan_validator.go arrived in #6605 and first shipped in v0.46.0, matching the reporter's bisect. The break is the unconditional ValidateProjectPlan call in doApply, which stats the convention <workspace>.tfplan for every project — including workflows whose apply is pure run steps with a custom plan path. There's a second break right below it: the hashFile call on that same nonexistent path when ExpectedPlanHash is empty.

Fixing that needs your requiresAtlantisManagedPlanFile / hasAtlantisManagedApplyStep predicates and roughly 40 lines around them. It does not need plan generations, the publication claim, S3 digest binding, validated snapshots, or the recovery CLI. Measured against main, this PR is 5,142 production insertions (plus 12,713 lines of tests/mocks, which is why it reads as 20k) — so the regression fix is well under 2% of it.

Your gating logic is unchanged in #6781. Reviewers checked it across run-only, mixed run+apply and policy_check workflow shapes and it holds, including the legacy upgrade path where ManagedPlanHash == "". The predicates are yours; I only moved them.

One change worth flagging: I made the gate fail closed.

func requiresManagedPlanFileForApply(ctx command.ProjectContext) bool {
	return ctx.RequiresAtlantisManagedPlanFile || hasAtlantisManagedApplyStep(ctx.Steps)
}

Gating on the context field alone means any ProjectContext built outside the builder — targeted applies, API paths, tests — gets Go's zero value false and silently skips plan-file validation. TestProjectCommandRunner_ApplyRejectsPlanDeletedAfterBuilderValidation caught exactly that. Making the executing steps authoritative removes the possibility of bypass by omission. Worth carrying back into this branch, and the same shape applies to apply_step_runner.go:45-50, where an empty ExpectedPlanHash currently disables the snapshot rather than erroring.

The rest of this PR is still valuable and I'd like to see it land in stages. The blocking items from my review are unchanged, and the claim lifecycle (no TTL, uncancellable spin loops, BoltDB-only offline recovery) is the piece that needs a design decision before more is built on it. There's also a set of cleanup call sites removed without replacement — PlanStore.Remove ×3, deletePlansAndPendingPlanLocks, DeletePullStatus, and the reaping in delete_lock_command.go — which are regressions against main independent of the feature work.

Suggested order, smallest first: hash binding + validated snapshot (a real security win that stands alone), then generations, then the claim once it has a lease. Happy to take any of those on, or to hand them back if you get time — just say which.

Also note the e2e-github failures on this PR are not yours: the server never received the opened webhook for the affected PRs (23 opened vs 28 closed events, and the failing pulls have no server log lines at all), and main is red the same way.

[chenrui333]

gonna break down the big pr into small ones

[chenrui333]

The monolith is being superseded by these reviewable extraction PRs:

• fix: only require a managed plan file for managed apply workflows #6781 — merged minimal Regression (v0.46.0): atlantis apply fails with "plan file is missing" for custom workflows #6642 custom-plan-path fix; preserved, not duplicated.
• fix: persist plan results before publishing success #6848 — A: persist successful plan/autoplan results before publishing success. Addresses Regression: Autoplan -> Apply fails #6641.
• fix: apply the verified managed plan snapshot #6849 — B: validate and execute an exact managed-plan snapshot; custom run-only workflows unchanged.
• fix: fence managed plans with durable generation identity #6855 — C: durable generations, accepted artifact identity, backend parity, and generation-aware cleanup/API compatibility.
• fix: fence pull-status writes with expiring leases #6857 — D1: expiring publication-lease storage, typed write ownership, atomic cleanup and backend recovery parity.
• fix: retain durable ownership of unresolved applies #6858 — D2: durable apply execution admission and ambiguous-execution protection.
• fix: coordinate plan publication with bounded leases #6859 — D3: bounded/cancellable short publication sections, command/API integration and exact-owner recovery. The PR body explains the lease state machine.

Stack: main → B → C → D1 → D2 → D3, with A independent and its two prerequisites explicitly included in C until A lands. All six branches include current main 172e1ece8d48b96e1dc5bd6698c278770ca7ff61. Every current head passes hosted Tests and GitHub/GitLab E2E; other image/analysis checks are tracked separately.

The old immortal claim, broad retention, cleanup removals, custom PLANFILE redirection, and broad E2E harness rewrite were not carried forward. Generation-specific regression tests and the main cleanup lifecycle are preserved in the corresponding slices. Optional E remains separate: current synthetic API requests use negative identifiers, so the literal shared-zero premise is stale; cross-replica identifier isolation is not claimed or expanded into this stack.

Please review A and B first, then the stack in order. This PR remains open and draft as the history/umbrella record while the split behavior and scope-accounting dispositions receive review. Nothing has been merged, and this branch has not been force-pushed.