Storage servers solve a fundamental problem in distributed databases: how to serve fast, consistent reads while maintaining strong transactional guarantees. They sit between the authoritative Transaction Log and the applications that need to read data, creating a layer that optimizes for read performance without sacrificing consistency.
Location: lib/bedrock/data_plane/storage.ex
When Bedrock commits a transaction, that transaction is immediately durable in the log servers. But serving reads directly from logs would be prohibitively slow—logs are optimized for append-only writes, not random key lookups. The solution is storage servers (called materializers in the codebase) that maintain read-optimized copies of the data, updated asynchronously from a per-shard stream produced by each log's Demux.
This creates a classic consistency challenge: how do you serve fast reads from local caches while ensuring that transactions see a consistent view of the world? Storage servers solve this fundamental tension between performance and correctness through Multi-Version Concurrency Control (MVCC), eventually consistent handling, and pluggable architecture that can adapt to different workloads while maintaining strict ACID guarantees.
Storage servers solve the consistency problem through multi-version concurrency control. Every piece of data in Bedrock exists at multiple points in time. When a key gets updated by different transactions, storage servers keep all the historical versions rather than overwriting the old value. This enables "time travel"—a transaction can ask for the value of a key as it existed at any point after the minimum read version.
This multi-version approach is what makes Optimistic Concurrency Control (OCC) possible. When the system needs to detect conflict between transactions, it can look at exactly which versions each transaction read and determine whether they interfered with each other. Without version history, this conflict detection would be impossible.
Version management also solves garbage collection elegantly. Storage servers can safely delete old versions once they know that no future transaction will need them, based on tracking the minimum read version still in use across the cluster.
Storage servers maintain an eventually consistent relationship with the transaction log. Committed transactions arrive asynchronously over each server's shard stream: the log's Demux slices every transaction by shard, and each storage server streams exactly its own shard's slice — object-storage chunks for history, the ShardServer's in-memory buffer for recent data, one continuous stream from any starting position. Every stream reply also carries version currency ("nothing for you, but you are current through v"), so a server whose shard is idle keeps advancing without ever polling. There is still always a window where a transaction has been committed but not yet reflected in all storage servers.
Bedrock handles this carefully through version leasing. The Gateway ensures that transactions only read at versions that are guaranteed to be available on all storage servers they'll access. If a transaction tries to read at version 100, the system first confirms that all relevant storage servers have applied transactions up to at least version 100.
This coordination enables the best of both worlds: writes achieve immediate durability through the log, while reads get fast local access through storage servers. The version-based consistency model ensures that despite the asynchronous updates, every transaction sees a coherent snapshot of the data.
As data grows, storage servers scale horizontally through key range partitioning. Each storage server owns specific ranges of keys and only maintains data for those ranges. From a performance perspective, each storage server can optimize its storage layout and caching strategies for its specific key ranges. Hot keys can be identified and cached more aggressively, and the storage engine can be tuned for the access patterns of its particular data.
Operationally, range partitioning enables dynamic load balancing. If one key range becomes a hotspot, it can be split and redistributed across multiple storage servers. The Director manages these range assignments and can adapt them during recovery or rebalancing operations.
Storage servers implement an abstract interface that separates the storage logic from the engine implementation. The interface is minimal—essentially versioned key-value reads, transaction application, and recovery coordination. But this simplicity enables radical implementation differences. Some storage engines might prioritize ultra-low latency using pure in-memory storage, while others might optimize for cost using cloud object storage.
This pluggability enables experimentation and gradual migration. A cluster could run proven disk-based storage engines alongside experimental new technologies, gradually shifting load as confidence in the new engines grows.
The relationship between storage servers and the durable stream becomes crucial during recovery. A storage server can be completely rebuilt from its shard's snapshot and chunk history in object storage, which means it is not a point of failure for data durability—that responsibility belongs to the logs and the chunk pipeline behind them.
Storage servers apply transactions eagerly for read currency but only persist to disk up to the known committed version, so their disk can never hold a version a recovery would discard. When a recovery rolls the cluster back, the rollback is a pure in-memory pointer discard—no disk surgery. A server that has been offline simply resumes its shard stream from its own applied position; the stream serves any starting point, so a stale server just has more stream to drink.
Storage servers integrate with the transaction system at several key points. Transaction Builder are their primary consumers, using "horse racing" to query multiple storage replicas in parallel and take the first successful response. The storage system also supports conflict detection indirectly by maintaining the version history that Resolvers need. Version leasing creates another integration point with the Gateway, ensuring that transactions only read at versions that are guaranteed to be available across all storage servers they'll access.
For the complete transaction flow, see Transaction Processing Deep Dive.
- Olivine: The materializer engine implementation
- Log System: Hosts the Demux whose per-shard streams feed storage updates
- Transaction Builder: Primary consumer of storage read operations with horse racing performance optimization
- Gateway: Coordinates read version leasing to ensure Storage server data availability
- Director: Control plane component that manages storage recovery and key range assignment