Proposal: Oxia Shard Replica Consistency Validation

[mattisonchao]

• Author(s): @mattisonchao
• Proposal time: 2026-01-12
• Implemented: NO
• Released: NO
• Repository: oxia-db/oxia
• Discussion Link: TBD

TL;DR

This proposal introduces a mechanism to validate strict data consistency across all replicas (Leader and Followers) of an Oxia shard. By introducing a new ConsistencyCheck WAL entry, we create a deterministic synchronisation point where all replicas capture an atomic snapshot of their LSM state and WAL chain CRC. This feature enables operators to detect silent data corruption, "split-brain" divergence due to non-deterministic application logic (e.g., memory counters), and storage anomalies.

Background knowledge

Oxia operates on the Replicated State Machine (RSM) model. In this model, high availability and consistency are achieved by replicating a write-ahead log (WAL) to multiple servers.

• State Machine Approach: Given an initial state $S_0$ and a deterministic sequence of log entries $L_1...L_n$, every replica must arrive at the exact same final state $S_n$.
• LSM-Tree (Pebble): Oxia uses an embedded LSM tree for state storage. These engines support "Snapshots," which provide a lightweight, point-in-time view of the database without blocking new writes.
• WAL (Write-Ahead Log): The immutable sequence of operations. In Oxia, this is the source of truth for the replication protocol. we could rely on the designed chain CRC to validate if the entry is exactly the same.

Motivation

While the RSM model guarantees consistency in theory, in practice, replicas can diverge due to software or hardware faults. We have observed issues in production where:

1. Non-Deterministic Apply Logic: Logic bugs in the Apply() phase (where the log entry updates the DB) rely on volatile inputs—such as in-memory counters not backed by the WAL, system time, or map iteration order. This causes the in-memory state of the Leader to differ from Followers.
2. Silent Storage Corruption: Rare hardware bit-rot or filesystem issues affecting the WAL or SSTables that do not trigger immediate I/O errors but alter the retrieved data.

Currently, Oxia lacks a tool to "prove" that Replica A and Replica B contain the exact same data at a specific log index.

(todo...)