external_stores (S3) plan restore is skipped on apply when the local pull dir already exists without a plan, breaking multi-replica/HA StatefulSet deployments

[shaarif-lab]

Overview of the Issue

Running Atlantis with replicaCount > 1 (e.g. a Kubernetes StatefulSet behind a Service with no session affinity), --enable-external-stores with an S3 plan_store, and --locking-db-type=redis for shared locking, atlantis apply intermittently fails with:

plan file is missing for dir "<dir>" workspace "default" project "<project>" with status "planned"; run `atlantis plan`

— even though the plan was successfully generated and uploaded to S3 moments earlier (confirmed via planstore/s3_plan_store.go logging uploaded plan to s3://...), and PullStatus (backed by the shared Redis DB) correctly reports the project's status as planned.

Digging into server/events/project_command_builder.go, buildAllProjectCommandsByPlan only attempts to restore plans from the external store when WorkingDir.GetPullDir(...) returns an os.IsNotExist error — i.e. only when the per-PR working directory is completely absent on the replica serving the apply request:

pullDir, err := p.WorkingDir.GetPullDir(ctx.Pull.BaseRepo, ctx.Pull)
if err != nil {
    if !os.IsNotExist(err) {
        return nil, err
    }
    // ... restore-from-store logic only runs here ...
}
plans, err := p.PendingPlanFinder.Find(pullDir)

If a different event earlier touched this PR on the same replica and created the directory — even one that failed for an unrelated reason (in our case, an autoplan that failed with "project is currently locked by an unapplied plan from pull #X", after which Atlantis's deletePlansAndPlanLocks deletes the .tfplan file but leaves the cloned working directory in place) — then a later apply on that same replica sees the directory already exists, never attempts the S3 restore, finds zero pending plans locally, and throws the "plan file is missing" error despite a valid plan sitting in the external store.

We also observed a worse variant: a targeted apply (atlantis apply -d <dir>) goes through a different code path that does restore a plan file regardless of directory existence, but the restored plan can be inconsistent with that replica's own terraform init state (different provider lock resolution across pods), causing terraform apply to fail with Error: Inconsistent dependency lock file instead of a clear "stale plan" message.

With no session affinity between the VCS webhook delivery and the StatefulSet's pods, this can affect any multi-replica deployment relying on external_stores for HA — it's not an edge case, just a matter of which pod a given comment happens to route to.

Reproduction Steps

1. Run Atlantis with 2+ replicas (e.g. Kubernetes StatefulSet, no sticky routing), --enable-external-stores (S3), --locking-db-type=redis.
2. Open a PR that autoplans a project whose dir/workspace is already locked by an unapplied plan from a different PR. The autoplan fails with "project is currently locked...", and Atlantis deletes the (failed) plan file — but the per-PR working directory it created remains on whichever pod served the request.
3. Resolve the other PR's lock (atlantis unlock / merge it).
4. Comment atlantis plan on the PR from step 2. If routed to a different pod than step 2 used, it clones fresh, plans successfully, and uploads to S3.
5. Comment atlantis apply. If routed back to the pod from step 2, it fails with "plan file is missing ... run atlantis plan" despite the valid S3 plan and planned status.

Logs

Logs (redacted)

{"level":"info","msg":"Running autoplan..."}
{"level":"info","msg":"creating dir '/atlantis-data/repos/<org>/<repo>/<pull>/default'"}
{"level":"info","msg":"Updating GitHub Check status for 'atlantis/plan: <project>' to 'failure'"}
{"level":"info","msg":"Failure running plan operation: This project is currently locked by an unapplied plan from pull #<other-pull>. ..."}
{"level":"info","msg":"deleting plans because there were errors and automerge requires all plans succeed"}

... (later, plan succeeds on a different pod and uploads) ...
{"level":"info","caller":"planstore/s3_plan_store.go:144","msg":"uploaded plan to s3://<bucket>/<prefix>/<org>/<repo>/<pull>/default/<dir>/<project>-default.tfplan"}

... (apply routed back to the first pod) ...
{"level":"error","caller":"events/instrumented_project_command_builder.go:81","msg":"Error building apply commands: plan file is missing for dir \"<dir>\" workspace \"default\" project \"<project>\" with status \"planned\"; run `atlantis plan`"}
{"level":"error","caller":"events/pull_updater.go:22","msg":"plan file is missing for dir \"<dir>\" workspace \"default\" project \"<project>\" with status \"planned\"; run `atlantis plan`"}

Environment details

• Atlantis version: v0.47.1
• Deployment method: Helm chart (runatlantis/helm-charts, chart atlantis-6.11.0), Kubernetes StatefulSet on EKS, replicaCount: 2–3, no session affinity on the Service in front of it
• Atlantis flags: --allow-commands=all --enable-drift-detection --enable-external-stores --locking-db-type=redis
• external_stores.plan_store: type: s3 (bucket/region redacted)
• Redis: single-node/serverless, TLS enabled, ACL-based auth
• Server-side repo config (relevant excerpt):

  repos:
    - id: /.*/
      apply_requirements: [mergeable]
      workflow: default
  external_stores:
    plan_store:
      type: s3
      s3:
        bucket: <redacted>
        region: <redacted>
        prefix: <redacted>/

• Repo-level atlantis.yaml: automerge: true, parallel_plan: true, parallel_apply: true

Additional Context

• Related (closed): missing plan file does not trigger error on apply #3487 — a different trigger (mid-flow reclone) with a similar underlying symptom: local working-directory state getting out of sync with what plan validation assumes, without a clear error pointing at the real cause.
• Possible fix direction: key the restore-from-store decision off whether PendingPlanFinder.Find (or the specific project's entry in PullStatus) finds a matching plan, rather than solely off WorkingDir.GetPullDir's existence check — i.e. attempt restore per-project whenever PullStatus says planned but no local .tfplan is found, regardless of whether the parent pull directory happens to already exist for an unrelated reason.

[dosubot]

Hi @shaarif-lab! I'm Dosu and I'm helping the runatlantis team.

Your analysis is accurate. The code confirms the root cause and your proposed fix direction.

The root cause in code

buildAllProjectCommandsByPlan gates the entire S3 restore path on os.IsNotExist from GetPullDir. If the pull directory exists for any reason — including a prior autoplan that failed due to a cross-PR lock — the restore branch is skipped, PendingPlanFinder.Find returns zero plans from the (planless) local directory, and validatePullStatusHasPlanFiles emits the "plan file is missing" error against the Redis-backed PullStatus.

The mechanism that triggers this is deletePlansAndPlanLocks, which removes only the .tfplan files but leaves the cloned working directory intact. So on a subsequent apply routed to the same pod, the directory exists, the S3 restore is bypassed, and no local plan is found.

Related open PRs

There are PRs addressing overlapping HA/multi-replica plan-restore gaps:

1. PR #6790 — fixes a related symptom where a leftover local .tfplan causes a hash mismatch when PlanStore.Load overwrites it from S3. The fix computes ExpectedPlanHash after PlanStore.Load rather than before.

2. PR #6750 — introduces a Redis ownership lease + internal forwarding model for active-active HA. In external-store mode, it explicitly handles "apply-all lists stored workspaces, clones them before restoration, and restores their plans for discovery." However, without fixing the directory-existence gate, the same bypass could still occur.

3. PR #6657 — a large rework of plan generation lifecycle that touches buildAllProjectCommandsByPlan and the plan store restore flow, though it may not specifically target this directory-exists bypass.

Your proposed fix direction is correct

Keying the restore decision off whether PendingPlanFinder.Find actually finds a matching local plan — or off PullStatus showing planned without a corresponding local .tfplan — rather than solely off GetPullDir's existence check, would address this. Concretely, the fix would attempt S3 restore whenever PullStatus shows projects with statusRequiresPlanFileForGenericApply status but no matching .tfplan is found locally, regardless of whether pullDir exists.

Coordinating with the authors of #6750 and #6790 would help ensure consistency with the ownership-lease flow and post-load hashing fix.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[SaiPisey2]

Confirmed. buildAllProjectCommandsByPlan only restores from the external store inside the os.IsNotExist branch, so a pull dir that exists but holds no plan files never triggers a restore — which is exactly the multi-replica case, and also happens after deletePlansAndPlanLocks clears the .tfplan on a replica that keeps the directory.

I have a fix that also attempts the restore when the directory is there but PendingPlanFinder comes back empty, with a test that reproduces the plan file is missing ... with status "planned" error you hit. Will open a PR.