feat: pluggable storage backends

[pseudomorph]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Pluggable storage backends

Context

Atlantis stores several kinds of data, and each kind currently hardcodes its own backend selection:

• Coordination data (project locks, the global apply/command lock, pull status) goes through the db.Database interface. --locking-db-type picks BoltDB or Redis for all of it at once, and the Redis connection is configured through dedicated --redis-* flags.
• Plan artifacts go through the planstore.PlanStore interface. --enable-external-stores plus the external_stores.plan_store block in the server-side repo config picks local filesystem or S3.
• Job output lives in process memory with no backend at all.

Two problems fall out of this. Connection configuration is tied to a single use case: the Redis flags exist only to serve locking, so pointing a second function (say, plans) at the same Redis requires new flags. And each new function invents its own selection mechanism: locking uses a flag, plans use a yaml block plus an enable flag, and a future jobs store would add a third pattern.

Decision

Split the two concepts and give each a home:

• A backend declares how to reach a storage service: connection parameters, credentials, endpoint. Backends are configured once per type (boltdb, filesystem, redis, s3; memory is a future candidate) and know nothing about what data they hold. boltdb and filesystem live under the server's --data-dir and work without configuration. One instance per type keeps the config flat; if separate clusters of one type are ever needed, a named-instance form can be added without breaking this schema.
• A store names a function that persists data: an interface scoped to one use case, plus one driver per backend type it supports. Each store binds to a backend type in config through its own schema, since stores don't share the same options; both take only backend today. On s3, each store appends its own key segment (plans/) under the backend's prefix, so stores sharing the backend cannot collide. Stores are coordination and plans today; jobs, logs, and remote work executors are future candidates.

Naming

coordination covers everything behind today's db.Database: project locks, the global apply/command lock, and pull status. These share transactional key-value semantics and always move between backends together. Pull status is state rather than a lock, so it may split into its own store later; nothing in this design blocks that.

plans covers plan artifacts — the .tfplan files a store must survive a server restart with. It replaces the external_stores.plan_store block from #6312, which is deleted rather than deprecated because it never reached a release (latest tag v0.46.0 predates it).

Backend types always use the concrete technology name — boltdb, filesystem, redis, s3, later mysql or memory — so the yaml states what actually holds the data; an earlier draft had a local umbrella type whose meaning varied per store, dropped for exactly that ambiguity.

Configuration

Backends and bindings live in the server-side repo config:

backends:
  redis:
    host: redis.internal
    port: 6379
    tls_enabled: true
  s3:
    bucket: atlantis-data
    region: us-east-1
    prefix: some/team

stores:
  coordination:
    backend: redis
  plans:
    backend: s3

store	local	redis	s3
coordination	yes (BoltDB file)	yes	no
plans	yes (workdir files, no restore)	planned	yes
jobs	yes (in memory)	planned	no

Package layout

server/core/backends/          # connection layer, function-agnostic
    backends.go                # Kind, Backend handles, lazy Registry
    boltdb.go                  # bolt file open
    redis.go                   # Redis dial (TLS, cluster selection, ping)
    s3.go                      # S3 client construction
    resolve.go                 # legacy-flag synthesis (removed with the flags)
server/core/coordination/      # store: locks, apply lock, pull status
    coordination.go            # LockStore/PullStatusStore/CommandLockStore + Store + NewStore(backend)
    locking.go                 # Locker client (was core/locking)
    apply_locking.go           # ApplyLocker client (was core/locking)
    pull_status_updater.go     # pull-status write policy (was events.DBUpdater)
    boltdb/                    # driver (was core/boltdb)
    redis/                     # driver (was core/redis)
server/core/planstore/         # store: plan artifacts
    plan_store.go              # PlanStore interface
    factory.go                 # New(backend) factory
    s3_plan_store.go           # s3 driver

This change absorbs the old server/core/db (interface, now coordination.Store), server/core/boltdb, server/core/redis, and server/core/locking packages into server/core/coordination. The locking package merged in rather than staying a consumer because its entire content (Locker client, NoOpLocker, ApplyLocker) wraps the coordination store; the lock-key parsing the drivers needed from it moved to models.ParseLockKey, which also removes the driver→service import.

The convention for a new store is a package under server/core/<store>/ owning its interface, its service API, a New*(backend) factory that switches on backend kind, and one driver file or subpackage per supported kind. The convention for a new backend type is a Kind constant, a config struct in raw/valid, and a client constructor in backends — no store code changes.

Constructed backends are typed handles behind the small backends.Backend interface — BoltDBBackend{DB}, FilesystemBackend{}, RedisBackend{Client}, S3Backend{Client, Bucket, Prefix} — mirroring the per-type config structs, so a handle cannot carry another type's fields. Store factories type-switch on the concrete handle and take the client, not the config: opening the bolt file, dialing redis (TLS, cluster selection, ping), and building the S3 client happen once in the registry, and store drivers only add their data layout on top (bucket creation and key migration for boltdb, key schema for redis). The registry constructs backends lazily on first Get, so a configured-but-unbound backend is never dialed.

Related consolidation

Landing with this change: coordination.Store splits into LockStore, PullStatusStore, and CommandLockStore so consumers depend on the slice they touch; events.DBUpdater becomes coordination.PullStatusUpdater (pull-identity freshness helpers move to models, result-classification types DirNotExistErr/ErrStaleCommandHead to command); and NoOpLocker is deleted — per-project repo_locking: false and --disable-repo-locking early-return inside DefaultProjectLocker.TryLock instead of swapping in a fake locker. That last change means the locks UI and unlock paths under --disable-repo-locking now query the real store: pre-existing locks become visible and deletable rather than silently ignored, and an unreachable redis surfaces as an error where the no-op never could fail.

Legacy flags

--locking-db-type and --redis-* predate this design in released versions, so backends.Resolve keeps them working: --locking-db-type=redis synthesizes a backends.redis block from the flags (a real backends.redis block wins over flag values) and binds coordination to it, while an explicit stores.coordination binding overrides the flag entirely. An unknown --locking-db-type value now errors at startup; previously it silently left the database nil. The flags are deprecated from this change on.

Follow-ups

Remove --locking-db-type, the --redis-* flags, and backends.Resolve after a deprecation window; drop the runtime plan-store aliases; migrate jobs/logs onto stores as needed.

Follow-up considerations before this grows past the first two stores, roughly in the order they'd start to hurt:

1. Concurrent startup migrations. Both coordination drivers run key migrations inside their constructors and assume a single writer; two replicas booting at once (the HA motivation for this work) can race the migration. A per-store migration lock or version marker should exist before more stores copy the pattern.
2. Secrets in checked-in config. backends.redis.password sits in the server-side repo config yaml. Env-var interpolation in the backends block should land before a mysql backend brings real credentials.
3. Driver conformance tests. Each store interface should have one behavioral suite run against every driver (boltdb and redis passing the same LockStore/PullStatusStore tests). A memory backend gives the suite a cheap third target and doubles as the future jobs-store default.
4. Registry-owned lifecycle. The registry constructs backends but the server closes the coordination store directly; moving Close ordering and per-backend health checks into the registry generalizes Ping into a readiness endpoint.
5. Driver-count growth. Drivers are hand-written per store×backend pair — fine at today's size; if both sets grow, shared capability interfaces (KV, object) implemented once per backend cap the matrix.
6. Store-op observability. One instrumentation wrapper applied in the store factories (op latency and error count tagged by store and backend) beats per-driver metrics plumbing later.
7. Keyspace ownership within a shared backend. Plan objects live directly under the s3 prefix and DeleteForPull deletes the pull subtree without filename filtering, so a second s3 store must take its own key segment or plans must filter to *.tfplan.

Consequences

Adding a Redis plan store or a MySQL coordination store becomes one driver plus one factory case; the connection config already exists. Users pointing several stores at one service declare the connection once. The config surface moves from flags to the server-side repo config, which fits its deployment model (checked-in, reviewed) but means secrets like redis.password land in yaml — the existing --redis-password flag remains usable via synthesis until env-var interpolation for backend config is added. Coordination data still moves between backends as one unit; splitting pull status out remains open.

[jamengual]

This design is the right foundation, and I want to add some research to help turn it into a full HA plan: a sweep of prior issues/PRs on HA caveats, an inventory of what state actually breaks with multiple replicas, and what we can learn from Lyft's fork. The goal is to make explicit what each store must eventually absorb, so the conventions being set here (backend/store split, driver-per-kind, conformance suites) are shaped by where this is going.

What history tells us

• Highly available cluster with multiple nodes #1571 is the canonical HA thread. The key maintainer insight from it still holds: "The locker isn't the hard part. It's the reliance on the filesystem for storing plans and for knowing which PRs are in progress." Users who shared data-dir over EFS/Azure Files crashed both nodes, because BoltDB is single-process (exclusive file lock — this also breaks Fargate/K8s rolling deploys where old and new instances briefly overlap).
• External backend for Locking. #894 was the original "stateless Atlantis" proposal (KV-store abstraction for locks), Proposal: Write plans to S3 #265 the original S3-plans proposal (2018, closed by feat: Plan store object storage #6312). Multiple instances? #1155 documented the classic failure: plan on pod A, apply on pod B → stat .../repos/...: no such file or directory.
• External Locking DB: Redis #2491 added Redis locking; every Redis capability since arrived as a new flag (Adding Redis TLS Option #2510 TLS, Adding Redis DB option #2527 DB number, feat: Redis cluster support. #6295 cluster; Improve Redis configuration options to allow the use of non default user or connection string #3610, Support redis cluster and sentinel mode #5445, Explicit Valkey support #6634 still open) — direct evidence for the backend-declaration model here.
• The "Enhanced locking" series (feat: Enhanced locking #2 - Backward Compatibility Adapter #5836–feat: Enhanced locking #0 - Consolidated Documentation #5845) tried a big-bang locking rewrite and stalled. The incremental store-by-store migration this issue proposes is the approach that history suggests can actually land.
• Bug: Jobs are specific to the instance that ran the job and are not shared across multiple instances #6022 and how to store jobs data in atlantis #6137 confirm jobs/output as the biggest user-visible HA gap today: with Redis locking, locks work across instances but job URLs 404 on the replica that didn't run the command, and the UI differs per replica behind an LB.
• Automerge with multiple Atlantis instances causes premature PR merge #6302/Merge strategy when multiple servers are used in the same repository #5773: automerge decides from a replica-local project view, so with two instances the first one whose applies finish merges the PR while the other still has unapplied plans.
• fix: coalesce duplicate autoplan webhooks for the same pull request head #6677: duplicate GitHub webhook deliveries race each other; the proposed in-memory dedup at the controller only works within one replica — cross-replica dedup is a textbook coordination-store record (SETNX-style).
• Race condition in BoltDB UnlockByPull implementation #5924: BoltDB UnlockByPull does read-then-delete in separate transactions. Worth noting for follow-up 3 (driver conformance): the conformance suite should pin down atomicity guarantees per operation, not just behavior, or this class of race multiplies across drivers.

What each store must eventually absorb (state inventory)

Beyond today's coordination and plans:

State	Where it lives today	HA impact	Store home
Working-dir + pull-level command locks (TryLock/TryLockPull/HasCommandLock)	in-process map (server/events/working_dir_locker.go)	two replicas both pass TryLockPull; comment command on B proceeds while A plans (loses the "wait until previous command is complete" UX)	coordination (distributed lock, TTL + fencing)
Job output buffers, receiver channels, pull→job index	in-process (server/jobs/project_command_output_handler.go)	#6022/#6137; websocket streaming is push-based from local memory	jobs (persistence) + pub/sub for live tails
Drift cache, remediation results, full-detection serialization	drift.InMemoryStorage, sync.Map in api_controller.go	detect on A, GET /api/drift or remediate on B → empty/refused	new drift store — the drift.Storage interface is already a clean seam
Cancellation tracker	in-process map	cancel on B can't stop a run on A	coordination (per-pull flag)
Webhook dedup/ordering	none (delivery IDs only logged)	redelivery to a different replica re-executes; consecutive-commit autoplans race across replicas	coordination
Scheduled work (stats publisher, token rotation; future cron per #5899)	every replica runs everything	duplicate runs; N-fold scheduled drift detection if #5899 lands	coordination (per-job lease — lighter than the leader election #1428 rejected as disproportionate)
Plan-hash fencing (ExpectedPlanHash)	computed from the local file at build time, never persisted	TOCTOU guard is single-replica only	coordination (persist per-generation, extending #6657's record)

Also: apply with no arguments discovers projects by scanning the pull dir on disk (buildAllProjectCommandsByPlan → PendingPlanFinder). On a replica without the clone that scan finds nothing — enumeration has to move to pull status / plan-store listing for any cross-replica apply.

The working-directory problem

The plans store alone doesn't make apply location-independent, because the default apply stage runs a single apply step with no init. Apply needs, on local disk: the .tfplan, an init-ed .terraform/ (backend + providers) with .terraform.lock.hcl, and the git tree (var-files, run steps, terragrunt caches, pre-workflow-hook-generated atlantis.yaml). A stored .tfplan blob covers one of those. Options, roughly:

• A. Pull ownership / sticky routing — any replica accepts the webhook, resolves an owner via a coordination-store lease (owner:<repo>/<pull>, TTL), forwards internally. Cheapest path; keeps everything else working; effectively per-pull active-passive. This is where the "remote work executors" future candidate fits.
• B. Shared filesystem — superficially attractive, historically catastrophic (Highly available cluster with multiple nodes #1571, Multiple atlantis instances - Installation using Helm chart #3795): the git-level mutexes (gitLocks, pendingMerges) are process-local sync.Maps, so two replicas would corrupt clones. Would move every git lock into the hot path of the coordination store. I'd treat this as explicitly unsupported.
• C. Artifact bundling — tar the project dir (plan, lockfile, generated files) into the plans store; unpack + init at apply. Captures hook/terragrunt-generated files, but big blobs and secrets-at-rest concerns.
• D. Rebuild-on-apply — clone at PR head (staleness already guarded by the head/base validators), terraform init (lockfile pins providers, plugin cache makes it cheap), fetch .tfplan from the plans store, verify a persisted hash, apply. Small blobs, deterministic; caveat is anything generated at plan time by hooks/run steps silently differs at apply time — either documented as unsupported in HA mode or covered by C as opt-in bundle mode.

D as default + C as opt-in for terragrunt/hook-heavy workflows, with A as the interim story, seems like the pragmatic sequence.

Worth stating in the design: #6657's PlanGeneration mechanism (Begin/Complete with Redis compare-and-swap, v0.46-compatible encoding) is the proof-of-pattern for durable fencing inside the coordination store. It covers atomic pull-status writes and cross-replica apply authorization — it does not yet cover execution-level mutual exclusion (two replicas can still both run terraform), the artifact itself, or working-dir locks. It's precisely a coordination-store record waiting for a plans store to point at.

Lyft fork (Neptune) learnings

Their fork (detached at v0.17.3, GitHub-only, Temporal-based — a design reference, not a merge source) is instructive because they hit every problem above:

• Storage abstraction: server/neptune/storage/client.go + a persistence config block is essentially a working prototype of this issue's shape — one default store (s3/local) with per-datatype key prefixes and a tiny Get/Set/Delete interface. A Lyft engineer even tried to upstream the jobs piece: PR feat: Add support for s3 persistence to log streaming jobs  #2157 (S3 persistence for log-streaming jobs, closed unmerged) — worth re-reading before designing the jobs store.
• Two-tier job store: in-memory while streaming, flush to blob store on close, read-through on miss. Solves persistence without touching the live streaming path; live tails stay host-local because the job URL points at the specific worker host. That's the incremental pattern for our jobs store; cross-replica live tails then need either that URL-routing trick, sticky sessions, or store-backed pub/sub.
• Persist only what must survive: they store tiny JSON metadata (last deployed revision), never planfiles or in-flight logs. Their answer to plan artifacts was host affinity — plan and apply pinned to one host per execution, artifacts ephemeral. That's option A above, and it's an argument that affinity can substitute for artifact externalization if we'd rather sequence it that way.
• Locks eliminated structurally: one durable workflow per root with an internal queue replaced shared lock storage entirely. Not transferable without Temporal, but it shows the endgame alternative to ever-finer distributed locks.

Suggested additions to the follow-up list

1. Coordination-store distributed working-dir/command locks (TTL + fencing token) — the prerequisite for any cross-replica correctness beyond what fix: preserve custom apply workflows and autoplan state #6657 covers.
2. Persist the plan hash + head commit + generation ID together at CompletePlanGeneration time — turns the existing local TOCTOU guard into a cross-replica one and gives apply-side rehydration something to verify against.
3. Jobs store with the two-tier pattern; treat live streaming (pub/sub or URL affinity) as a separate follow-up from persistence so it can land incrementally.
4. Webhook dedup keys (VCS delivery ID, SETNX + TTL) in the coordination store.
5. Startup validation refusing multi-writer BoltDB (detecting the Highly available cluster with multiple nodes #1571 shared-EFS foot-gun) once replicas are supportable at all.
6. An explicit "HA mode" doc stating the supported sequence: active-passive (Redis + leader lease, accept clone/job loss on failover — fix: preserve custom apply workflows and autoplan state #6657's persisted generation already makes the failed-over apply fail safe) before active-active.

Happy to help break any of these into issues/PRs.