Transaction Building and Committing Process in Bedrock

This document explains the complete process of building and committing a transaction in the Bedrock distributed key-value store, from reading a key to committing a modification of that key. This is the definitive technical reference for transaction processing in Bedrock.

Overview

Bedrock implements a distributed ACID transaction system based on FoundationDB's architecture. The transaction process involves multiple specialized components working together to provide strict serialization while maintaining high performance through optimistic concurrency control (MVCC).

Navigation: This document provides the complete technical implementation details. For quick reference, start with Transaction Overview. For architectural context, see Architecture Deep Dive. For component-specific details, see individual Component Documentation.

Key Components

Client: Application code that initiates and executes transactions
Gateway: Client interface that manages transaction coordination
Transaction Builder: Per-transaction process that accumulates reads/writes and manages transaction state
Sequencer: Assigns global version numbers for reads and commits (Lamport clock)
Commit Proxy: Batches transactions for efficient processing and conflict resolution
Resolver: Implements MVCC conflict detection across key ranges
Log System: Provides durable transaction storage with strict ordering
Storage (Olivine): Serves versioned key-value data and applies committed transactions

💡 Deep Dive Available: Click on any component name above to access detailed technical documentation including APIs, implementation details, performance characteristics, and code references.

Complete Transaction Sequence Diagram

sequenceDiagram
    participant Client
    participant Gateway
    participant TransactionBuilder as Transaction Builder
    participant Sequencer
    participant CommitProxy as Commit Proxy
    participant Resolver
    participant Log
    participant Storage
    
    Note over Client, Storage: Phase 1: Transaction Initiation
    Client->>Gateway: begin_transaction()
    Gateway->>TransactionBuilder: start_link()
    activate TransactionBuilder
    TransactionBuilder-->>Gateway: {:ok, transaction_pid}
    Gateway-->>Client: {:ok, transaction_pid}
    
    Note over Client, Storage: Phase 2: Read Operations
    Client->>TransactionBuilder: fetch(key)
    TransactionBuilder->>TransactionBuilder: check local writes (read-your-writes)
    alt Read version not yet obtained
        TransactionBuilder->>Sequencer: next_read_version()
        Sequencer-->>TransactionBuilder: {:ok, read_version}
    end

    TransactionBuilder->>Storage: fetch(key, read_version)
    Storage-->>TransactionBuilder: {:ok, value}
    TransactionBuilder->>TransactionBuilder: track read in transaction state
    TransactionBuilder-->>Client: {:ok, value}
    
    Note over Client, Storage: Phase 3: Write Operations (Local Accumulation)
    Client->>TransactionBuilder: put(key, new_value)
    TransactionBuilder->>TransactionBuilder: accumulate write locally
    TransactionBuilder-->>Client: :ok
    
    Note over Client, Storage: Phase 4: Commit Phase (Multi-Step Process)
    Client->>TransactionBuilder: commit()
    
    rect rgb(255, 245, 235)
        Note over TransactionBuilder, CommitProxy: Step 4.1: Prepare and Route to Commit Proxy
        TransactionBuilder->>TransactionBuilder: prepare_transaction_for_commit()
        TransactionBuilder->>CommitProxy: commit(transaction)
        
        Note over CommitProxy, Sequencer: Step 4.2: Batch Formation and Version Assignment
        CommitProxy->>CommitProxy: add_transaction_to_batch()
        CommitProxy->>Sequencer: next_commit_version()
        Sequencer-->>CommitProxy: {:ok, last_commit_version, commit_version, known_committed_version}
        
        Note over CommitProxy, Resolver: Step 4.3: Conflict Resolution
        CommitProxy->>CommitProxy: prepare_for_resolution()
        CommitProxy->>Resolver: resolve_transactions(batch, versions)
        Resolver->>Resolver: check read-write and write-write conflicts
        Resolver-->>CommitProxy: {:ok, aborted_indices}
        
        Note over CommitProxy, Client: Step 4.4: Handle Aborted Transactions
        CommitProxy->>CommitProxy: split_transactions_by_abort_status()
        CommitProxy->>Client: reply({:error, :aborted}) [for aborted transactions]
        
        Note over CommitProxy, Log: Step 4.5: Prepare for Logging
        CommitProxy->>CommitProxy: group_successful_transactions_by_tag()
        CommitProxy->>CommitProxy: build_log_transactions_by_coverage()
        
        Note over CommitProxy, Log: Step 4.6: Durable Log Persistence
        par Push to all logs in parallel
            CommitProxy->>Log: push(encoded_transaction, last_commit_version, known_committed_version)
            Log->>Log: append when predecessor reaches WAL tip
            Log->>Log: fsync connected predecessor chain
            Log-->>CommitProxy: :ok
        end
        
        Note over CommitProxy, Sequencer: Step 4.7: Notify Sequencer of Success
        CommitProxy->>Sequencer: report_successful_commit(commit_version)
        Sequencer->>Sequencer: update committed version tracking
        
        Note over CommitProxy, Client: Step 4.8: Notify Successful Clients
        CommitProxy-->>TransactionBuilder: {:ok, commit_version}
    end
    
    Note over Client, Storage: Phase 5: Transaction Completion
    TransactionBuilder-->>Client: {:ok, commit_version}
    deactivate TransactionBuilder
    
    Note over Log, Storage: Background: Storage Streams from the Log's Demux
    Storage->>Log: get_shard_server(shard_id) — one-time discovery
    Log-->>Storage: {:ok, shard_server}
    Storage->>Log: ShardServer.pull(from_version) — chunks + buffer
    Log-->>Storage: {:ok, [slices], %{high_water, kcv}}
    Storage->>Storage: apply slices to local storage

Component Deep Dives

For detailed technical documentation on any component, see the Components Documentation directory:

Gateway Deep Dive - Client interface, version leasing, worker advertisement
Transaction Builder Deep Dive - Per-transaction processes, read-your-writes, storage coordination
Sequencer Deep Dive - Version assignment, Lamport clock, global ordering
Commit Proxy Deep Dive - Transaction batching, finalization pipeline, client coordination
Resolver Deep Dive - MVCC conflict detection, version history, range processing
Log System Deep Dive - Durable storage, replication, recovery coordination
Shale Deep Dive - Disk-based log implementation, WAL architecture
Storage Deep Dive - Multi-version storage, MVCC reads, log integration

Transaction Format

Bedrock uses a sophisticated tagged binary format for transaction encoding that replaced the simple map structure. This format provides several key advantages:

Binary Structure

Tagged sections: Self-describing sections with type, size, and embedded CRC validation
Order independence: Sections can appear in any order for better extensibility
Efficient operations: Extract specific sections without full decode
Space optimization: Empty sections are omitted, opcodes are size-optimized

Section Types

MUTATIONS (0x01): Always present, contains {:set, key, value} and {:clear_range, start, end} operations
READ_CONFLICTS (0x02): Present when transaction performed reads, includes read version
WRITE_CONFLICTS (0x03): Present when write conflicts exist
COMMIT_VERSION (0x04): Added by commit proxy after version assignment

Usage Throughout System

The flexible design allows each component to work with only needed sections:

Transaction Builder → Commit Proxy: Full transaction with mutations, conflicts, and read version
Commit Proxy → Resolver: Conflicts and versions for conflict detection (mutations not needed)
Commit Proxy → Logs: Mutations and commit version for storage (conflicts not needed)

This approach improves efficiency and reduces data transfer overhead between components.

Binary Format: Transactions use Transaction encoding with tagged binary sections for efficient processing. See the deep dive for technical details.

Detailed Phase Breakdown

Phase 1: Transaction Initiation

Purpose: Establish a transaction context and obtain a consistent read version.

Process:

Client calls Bedrock.Repo.transact/1
Gateway creates a new Transaction Builder process via start_link/1
Transaction Builder initializes with gateway reference and transaction system layout
Client receives transaction builder PID for subsequent operations

Key Code Locations:

Gateway creation: lib/bedrock/cluster/gateway.ex:19
Transaction Builder startup: lib/bedrock/cluster/gateway/transaction_builder.ex:22

Phase 2: Read Operations

Purpose: Read data at a consistent snapshot version while tracking read keys for conflict detection.

Process:

Client calls fetch/2 on the transaction builder
Transaction builder checks local writes first (read-your-writes consistency)
If not found locally and no read version exists:
- Request read version from Sequencer via next_read_version/1
Fetch data from Storage servers at the read version
Storage performs "horse race" across replicas for performance
Transaction builder tracks the read key and value
Return value to client

Key Code Locations:

Fetching logic: lib/bedrock/cluster/gateway/transaction_builder/fetching.ex:10
Read version management: lib/bedrock/cluster/gateway/transaction_builder/read_versions.ex:11
Storage fetch: lib/bedrock/data_plane/storage.ex:33

Read-Your-Writes Consistency: The transaction builder maintains local writes in memory, ensuring that reads within the same transaction immediately see previous writes without network calls.

Phase 3: Write Operations

Purpose: Accumulate write operations locally without network traffic until commit time.

Process:

Client calls put/3 on the transaction builder
Transaction builder accumulates writes in local memory
No network operations occur during writes
Writes are immediately visible to subsequent reads within the same transaction

Key Code Locations:

Write accumulation: lib/bedrock/cluster/gateway/transaction_builder/putting.ex
Local write storage: lib/bedrock/cluster/gateway/transaction_builder/state.ex:16

Optimization: This batching approach minimizes network traffic and allows for optimistic concurrency control.

Phase 4: Commit Phase

This is the most complex phase involving multiple distributed components working together.

Step 4.1: Prepare and Route to Commit Proxy

Process:

Transaction builder calls do_commit/1
Prepare transaction using Transaction binary format:
- mutations: List of {:set, key, value} or {:clear_range, start, end} operations
- read_conflicts: {read_version, [read_conflict_ranges]} or {nil, []} for write-only transactions
- write_conflicts: List of write conflict ranges for all mutations
- Uses tagged binary sections with CRC validation for efficient processing
Select a Commit Proxy randomly from available commit proxies
Send encoded transaction to selected Commit Proxy

Key Code Locations:

Commit preparation: lib/bedrock/cluster/gateway/transaction_builder/committing.ex:8
Transaction format: lib/bedrock/data_plane/bedrock_transaction.ex

Step 4.2: Batch Formation and Version Assignment

Purpose: Improve throughput by batching multiple transactions and assign global commit version.

Process:

Commit Proxy adds transaction to current batch
When batch reaches finalization criteria (size or timeout):
- Request commit version from Sequencer via next_commit_version/1
- Sequencer returns last_commit_version, commit_version, and the known_committed_version (KCV)
- The {last, current} pair defines the Lamport predecessor chain; numeric gaps are valid, but every log appends only the connected prefix
- KCV is an independent monotonic watermark carried on every log push and accumulated with max, so downstream durability machinery (Demux chunk cuts, storage eviction) can gate on it even when a future transaction is parked

Key Code Locations:

Batching logic: lib/bedrock/data_plane/commit_proxy/batching.ex
Server handling: lib/bedrock/data_plane/commit_proxy/server.ex:110

Step 4.3: Conflict Resolution

Purpose: Detect and resolve transaction conflicts using Multi-Version Concurrency Control (MVCC).

Process:

Validation: Validate transaction format using Transaction validation
- Verify binary format integrity with CRC checks
- Ensure transaction summaries conform to expected format
- Handle validation errors with appropriate telemetry
Transform transactions into conflict resolution format
Distribute transactions to appropriate Resolvers based on key ranges
Each Resolver checks for:
- Read-Write conflicts: Transaction read a key that was written by a later-committed transaction
- Write-Write conflicts: Two transactions wrote to the same key
- Within-batch conflicts: Transactions in the same batch conflict with each other
Timeout handling: Transactions waiting for version ordering may timeout (default 30 seconds)
Return list of aborted transaction indices or timeout errors

Key Code Locations:

Conflict resolution: lib/bedrock/data_plane/commit_proxy/finalization.ex:257
Resolver implementation: lib/bedrock/data_plane/resolver.ex

MVCC Details: Conflicts are detected by comparing transaction read/write sets against the version history maintained by Resolvers.

Step 4.4: Handle Aborted Transactions

Purpose: Immediately notify clients of aborted transactions to minimize latency.

Process:

Split transactions into aborted and successful sets
Send {:error, :aborted} responses to aborted transaction clients
Continue processing successful transactions

Key Code Locations:

Transaction splitting: lib/bedrock/data_plane/commit_proxy/finalization.ex:421

Step 4.5: Prepare for Logging

Purpose: Organize successful transactions by storage team tags for efficient log distribution.

Process:

Group writes by storage team tags (key ranges)
Build transaction shards for each tag
Ensure all keys are covered by storage teams (coverage validation)

Key Code Locations:

Tag grouping: lib/bedrock/data_plane/commit_proxy/finalization.ex:522
Coverage validation: lib/bedrock/data_plane/commit_proxy/finalization.ex:602

Step 4.6: Durable Log Persistence

Purpose: Achieve durability by persisting transactions to multiple log servers.

Process:

Build transaction for each log based on tag coverage
Encode transactions for each log server
Push transactions to ALL log servers in parallel
Each log parks future predecessor links and drains the connected prefix in chain order
Wait for acknowledgment from ALL log servers (ack sent only after that transaction's WAL append + fsync; Demux is asynchronous)
If any log fails, trigger recovery (fail-fast approach)

Key Code Locations:

Log push coordination: lib/bedrock/data_plane/commit_proxy/finalization.ex:744
Individual log push: lib/bedrock/data_plane/log.ex:56

Durability Guarantee: ALL logs must acknowledge only after WAL append + fsync before transaction is considered committed.

Step 4.7: Notify Sequencer of Success

Purpose: Update the sequencer's committed version tracking for future conflict resolution.

Process:

Call report_successful_commit/2 on Sequencer
Sequencer updates its internal committed version tracking
This information is used for future read version assignments

Key Code Locations:

Sequencer notification: lib/bedrock/data_plane/commit_proxy/finalization.ex:829

Step 4.8: Notify Successful Clients

Purpose: Inform clients that their transactions have been successfully committed.

Process:

Send {:ok, commit_version} to all successful transaction clients
Clients can use the commit_version for debugging and monitoring

Key Code Locations:

Success notification: lib/bedrock/data_plane/commit_proxy/finalization.ex:856

Phase 5: Transaction Completion

Purpose: Clean up transaction resources and return final result to client application.

Process:

Client receives final transaction result
Transaction Builder process terminates
Resources are cleaned up
Client application continues execution

Background Operations

Storage Updates from the Demux

Purpose: Eventually consistent application of committed transactions to storage servers.

Process:

Each storage server streams its shard's slices from a log's Demux ShardServer — object-storage chunks for history, the in-memory buffer for recent data, one continuous stream
Slices are applied in version order; empty "current through v" replies advance the server's version when its shard is idle
Storage maintains multiple versions for MVCC reads, applying eagerly but persisting to disk only up to the known committed version
Old versions leave memory through window advancement based on version-time lag

Key Code Locations:

Storage streaming: lib/bedrock/data_plane/materializer/olivine/streaming.ex
Shard serving: lib/bedrock/data_plane/demux/shard_server.ex
Log pull (recovery-only): lib/bedrock/data_plane/log.ex

Error Handling and Recovery

Transaction Conflicts

Clients receive {:error, :aborted} for conflicted transactions
Applications should retry with exponential backoff
Conflicts are natural in optimistic concurrency control

Validation Errors

Format Validation: Transaction binary format validation with CRC checks
Transaction Summary Validation: Ensures transaction summaries conform to expected {read_info | nil, write_keys} format
Waiting List Validation: Validates transactions before adding to resolver waiting queues
All validation failures include detailed telemetry for debugging

Timeout Handling

Waiting List Timeout: Transactions waiting for version ordering timeout after 30 seconds (default)
WaitingList Management: Automatic cleanup of expired transactions with appropriate error responses

System Failures

Log Server Failures: Trigger commit proxy recovery (fail-fast)
Storage Server Failures: Reads continue from replicas
Commit Proxy Failures: Director detects and starts new commit proxies
Network Partitions: Raft consensus ensures consistency

Version Management

Version Too Old: Storage no longer has the requested version
Version Too New: Read version exceeds current committed version

Performance Characteristics

Optimizations

Batching: Multiple transactions processed together
Pipelining: Read versions assigned while commits process
Local Caching: Transaction builders cache storage server choices
Horse Racing: Parallel queries to multiple storage replicas
Tag-Based Sharding: Efficient distribution of writes across logs

Latency Sources

Network Round Trips: Client ↔ Gateway ↔ Data Plane components
Conflict Resolution: Resolver processing time
Log Durability: Disk I/O for transaction persistence
Version Assignment: Sequencer coordination

Throughput Factors

Batch Size: Larger batches improve throughput but increase latency
Conflict Rate: High conflicts reduce effective throughput
Key Distribution: Hot keys can become bottlenecks
Storage Parallelism: More storage servers improve read throughput

Transaction Guarantees (ACID)

Atomicity

All writes in a transaction commit together or none do
Partial commits are impossible due to conflict resolution + logging

Consistency

All transactions see a consistent view at their read version
Invariants are maintained through conflict detection

Isolation

Strict serialization: transactions appear to execute sequentially
Read-your-writes consistency within transactions
No dirty reads, phantom reads, or write skew

Durability

Committed transactions survive system failures
ALL log servers must WAL-fsync acknowledge before commit confirmation
Storage servers eventually reflect all committed transactions

Client Usage Examples

Based on the BedrockEx test harness, here are practical examples of how applications use Bedrock transactions:

Simple Key-Value Operations

# Basic put operation
def hello do
  Repo.transact(fn ->
    Repo.put("hello", "world")
    {:ok, :ok}
  end)
end

# Basic get operation  
def hello2 do
  Repo.transact(fn ->
    {:ok, Repo.get("hello")}
  end)
end

Complex Business Logic: Money Transfer

def move_money(amount, account1, account2) do
  Repo.transact(fn ->
    with :ok <- check_sufficient_balance_for_transfer(account1, amount),
         {:ok, new_balance1} <- adjust_balance(account1, -amount),
         {:ok, new_balance2} <- adjust_balance(account2, amount) do
      {:ok, {new_balance1, new_balance2}}
    end
  end)
end

def check_sufficient_balance_for_transfer(account, amount) do
  with {:ok, balance} <- fetch_balance(account) do
    if can_withdraw?(amount, balance) do
      :ok
    else
      {:error, "Insufficient funds"}
    end
  end
end

def fetch_balance(account) do
  case Repo.fetch(key_for_account_balance(account)) do
    {:ok, balance} -> {:ok, balance}
    _ -> {:error, "Account not found"}
  end
end

def adjust_balance(account, amount) do
  with {:ok, balance} <- fetch_balance(account) do
    new_balance = balance + amount
    Repo.put(key_for_account_balance(account), new_balance)
    {:ok, new_balance}
  end
end

def key_for_account_balance(account), do: {"balances", account}

Batch Operations

def setup_accounts do
  Repo.transact(fn ->
    Repo.put(key_for_account_balance("1"), 100)
    Repo.put(key_for_account_balance("2"), 500)
    {:ok, :ok}
  end)
end

# High-volume transaction example
def rando do
  1..10_000
  |> Enum.each(fn _ ->
    Repo.transact(fn ->
      1..5
      |> Enum.each(fn _ ->
        key = :crypto.strong_rand_bytes(5) |> Base.encode32(case: :lower)
        value = :crypto.strong_rand_bytes(5) |> Base.encode32(case: :upper)
        Repo.put(key, value)
      end)
      {:ok, :ok}
    end)
  end)
end

Repository Configuration

defmodule BedrockEx.Repo do
  use Bedrock.Repo,
    cluster: BedrockEx.Cluster,
    key_codecs: [
      default: Bedrock.KeyCodec.TupleKeyCodec  # Supports structured keys like {"balances", "account1"}
    ],
    value_codecs: [
      default: Bedrock.ValueCodec.BertValueCodec  # Elixir term serialization
    ]
end

# Note: Transaction binary format handles the low-level encoding/decoding
# of mutations and conflict ranges transparently to client applications

Key Transaction Patterns

1. Read-Modify-Write Pattern

The money transfer example demonstrates the classic read-modify-write pattern:

Read current balance (fetch_balance)
Validate business rules (check_sufficient_balance)
Modify data (adjust_balance)
All within a single transaction for atomicity

2. Read-Your-Writes Consistency

Within a transaction, all reads immediately see previous writes:

Repo.transact(fn ->
  Repo.put("key", "value1")
  {:ok, "value1"} = Repo.get("key")  # Sees the write immediately
  Repo.put("key", "value2")
  {:ok, "value2"} = Repo.get("key")  # Sees the updated value
  {:ok, :ok}
end)

3. Structured Keys

Using tuple keys for hierarchical data organization:

key_for_account_balance(account) -> {"balances", account}
# This creates keys like {"balances", "123"} which can be efficiently 
# range-queried and distributed across storage teams

4. Error Handling

Transactions can return errors that cause rollback:

case Repo.transact(fn ->
  case some_operation() do
    {:ok, result} -> {:ok, result}
    {:error, reason} -> {:error, reason}  # Transaction rolls back
  end
end) do
  {:ok, result} -> handle_success(result)
  {:error, :aborted} -> handle_conflict()  # Retry logic here
  {:error, reason} -> handle_error(reason)
end

Conclusion

The Bedrock transaction system provides a sophisticated implementation of distributed ACID transactions with strong consistency guarantees. The multi-phase commit process, while complex, enables high performance through batching, pipelining, and optimistic concurrency control while maintaining strict serialization semantics.

The architecture separates concerns cleanly:

Control Plane: Manages cluster coordination and recovery
Data Plane: Handles transaction processing and data storage
Client Interface: Provides simple transaction semantics

This separation allows for independent scaling and optimization of each component while maintaining system-wide consistency and availability.

From the client perspective, the system provides intuitive transaction semantics that hide the underlying distributed complexity while delivering strong ACID guarantees. The examples from BedrockEx demonstrate how applications can build complex business logic on top of Bedrock's transactional foundation.

Uh oh!

FilesExpand file tree

transactions.md

Latest commit

History

transactions.md

File metadata and controls

Transaction Building and Committing Process in Bedrock

Overview

Key Components

Complete Transaction Sequence Diagram

Component Deep Dives

Transaction Format

Binary Structure

Section Types

Usage Throughout System

Detailed Phase Breakdown

Phase 1: Transaction Initiation

Phase 2: Read Operations

Phase 3: Write Operations

Phase 4: Commit Phase

Step 4.1: Prepare and Route to Commit Proxy

Step 4.2: Batch Formation and Version Assignment

Step 4.3: Conflict Resolution

Step 4.4: Handle Aborted Transactions

Step 4.5: Prepare for Logging

Step 4.6: Durable Log Persistence

Step 4.7: Notify Sequencer of Success

Step 4.8: Notify Successful Clients

Phase 5: Transaction Completion

Background Operations

Storage Updates from the Demux

Error Handling and Recovery

Transaction Conflicts

Validation Errors

Timeout Handling

System Failures

Version Management

Performance Characteristics

Optimizations

Latency Sources

Throughput Factors

Transaction Guarantees (ACID)

Atomicity

Consistency

Isolation

Durability

Client Usage Examples

Simple Key-Value Operations

Complex Business Logic: Money Transfer

Batch Operations

Repository Configuration

Key Transaction Patterns

1. Read-Modify-Write Pattern

2. Read-Your-Writes Consistency

3. Structured Keys

4. Error Handling

Conclusion