Skip to content

README.md

Latest commit

 

History

History
208 lines (167 loc) · 8.83 KB

README.md

File metadata and controls

208 lines (167 loc) · 8.83 KB
Raw

Bedrock

Elixir CI Coverage Status Hex.pm

Bedrock is a distributed key-value store that runs inside your Elixir application. It takes the transaction architecture that made FoundationDB legendary — strictly serializable ACID transactions across the entire key-space, over an unbundled sequencer / log / storage design — and rebuilds it natively on the BEAM, with one modern twist: durable history lives in object storage.

Local disks hold only a few seconds of write-ahead log. Everything committed flows into immutable chunks in S3-compatible storage (or the local filesystem), and the WAL continuously trims itself behind that durable floor. There is no external database to operate and no fleet to manage: add a dependency, join your supervision tree, and every node of your cluster shares one transactional key-space.

def deps do
  [
    {:bedrock, "~> 0.6"}
  ]
end

A taste

Define a cluster and a repo, add them to your supervision tree, and you have a database:

defmodule MyApp.Cluster do
  use Bedrock.Cluster, otp_app: :my_app, name: "my_app"
end

defmodule MyApp.Repo do
  use Bedrock.Repo, cluster: MyApp.Cluster
end

Transactions read from a consistent snapshot, see their own writes, and either commit atomically or not at all — across any keys, on any node:

alias MyApp.Repo

Repo.transact(fn ->
  balance = Repo.get("alice/balance")
  Repo.put("alice/balance", debit(balance, 10))
  Repo.put("bob/balance", credit(balance, 10))
  {:ok, :transferred}
end)

Familiar FoundationDB layers come along for the ride: Bedrock.Directory and Bedrock.Keyspace for organizing data, key selectors, range reads, and atomic operations (add, min, max, append_if_fits, compare_and_clear, ...) for building higher-level abstractions without read-write conflicts.

Try it live: the class-scheduling tutorial (adapted from FoundationDB's classic) builds a working enrollment system in a notebook — no setup beyond clicking the badge.

Run in Livebook

Why Bedrock

The FoundationDB model, kept. A single sequencer hands out global versions (a Lamport clock), commit proxies batch transactions, resolvers detect MVCC conflicts over key ranges, logs make commits durable, and materializers serve reads — each a separate, independently recoverable process. Optimistic concurrency means no locks and no deadlocks; conflicts are detected at commit time. The result is strict serializability: the strongest isolation guarantee a database can offer. FoundationDB's architecture overview, read/write path, and SIGMOD '21 paper are excellent background on the design; Bedrock's own deep dives cover how it maps onto the BEAM and where it diverges.

Durability, modernized. A commit is acknowledged only after every required log has appended it to its write-ahead log and fsynced. From there, a known-committed-version watermark gates everything downstream: nothing enters object storage or a materializer's disk unless it is known to be committed, so recovery never has to undo derived state. Every five seconds, each log slices its recent transactions per shard and writes them to object storage — and because every replica sees the same versions and computes the same cuts, replicas produce byte-identical chunks, so "that file already exists" counts as success. Once chunks are confirmed, the WAL trims itself: a log's disk footprint tracks a few seconds of traffic, not the cluster's lifetime.

Restarts that don't scale with history. Recovery replays only the untrimmed WAL tail — seconds of transactions — never the whole history. Surviving materializers are handed back their shards and resume streaming from where they stopped; only a genuinely lost shard rebuilds from chunks. A typical restart converges in under a second regardless of how old the cluster is.

Pure BEAM. No ports, no NIFs to a storage engine, no sidecar processes. Distribution rides on Erlang distribution; supervision, recovery, and backpressure are OTP all the way down. Bedrock also leans on a quiet BEAM superpower: large binaries are shared between processes by reference, never copied. Keys, values, and whole encoded transactions stay binaries as they flow from commit proxy to log to shard streams, so the hot path hands around pointers rather than payloads — a design that is naturally multi-core and cache friendly. If you can run an Elixir node, you can run Bedrock.

How a write becomes durable

  1. A commit proxy pushes the transaction to every required log; the client is acknowledged only after each log has fsynced its WAL.
  2. The sequencer tracks the highest version confirmed on every log — the known committed version — and everything downstream waits for it.
  3. On five-second version boundaries, each log's shards flush everything at or below the cut to object storage as immutable, replica-identical chunks.
  4. Confirmed chunks raise the durable floor; the log recycles every WAL segment that falls entirely below it.
  5. Materializers stream each shard continuously — chunks for history, the in-memory buffer for recent data — and serve reads without ever touching the WAL.

The Durability Foundation guide tells this story end to end, including what happens on restart.

Object storage

The local filesystem is the default backend. To point durable history at any S3-compatible store (AWS S3, MinIO, ...):

config :bedrock, Bedrock.ObjectStorage,
  backend: :s3,
  s3: [
    bucket: "bedrock",
    access_key_id: "...",
    secret_access_key: "...",
    region: "us-east-1"
  ]

See the S3 Object Storage guide for conditional-write semantics and the full option set.

Production configuration

Bedrock validates a durability profile at startup — replication factor, log count, and persistent paths for coordinator, log, and materializer roles. The default mode is :strict, which fails fast when requirements are not met. For local development or a single-node rollout, relax it explicitly:

config :bedrock, MyApp.Cluster,
  durability_mode: :relaxed

Details in the Durability Profile guide.

Documentation

The guides are organized in three tiers — quick reads for orientation, guides for tasks, deep dives for internals:

The full index lives in guides/ai-start-here.md and the glossary.

Status

Bedrock is pre-1.0 and under active development. The transaction system, the durability pipeline, and the S3 backend are exercised by unit, MinIO-backed integration, and distributed durability test suites — but the API may still shift between minor versions. We'd love early feedback.

Development and testing

mix test                    # default suite
mix test --include s3       # + MinIO-backed S3 integration tests

S3-tagged tests need a local MinIO binary (MIX_ENV=test mix minio_server.download --arch darwin-arm64 --version latest) and are skipped automatically when it isn't available. The distributed durability suite is documented in guides/distributed-durability-tests.md.

License

Bedrock is released under the MIT License.