Bound `ChainState` partition growth by archiving pre-checkpoint history to object storage

[ndr-ds]

Problem

Each microchain's ChainStateView is persisted as a single storage partition (RootKey::ChainState(chain_id); on ScyllaDB this is one partition of kv.table_linera, PRIMARY KEY (root_key, k)). Every view leaf (each MapView entry, LogView element, collection sub-entry) is one row in that partition, and several protocol-maintained sub-structures grow without bound and are never pruned:

• received_log: LogView<ChainAndHeight> (linera-chain/src/chain.rs) — +1 row per received cross-chain message bundle, append-only.
• block_hashes: CustomMapView<BlockHeight, CryptoHash> (post-Merge confirmed_log and preprocessed_blocks into block_hashes #6163; previously confirmed_log) — +1 row per block, insert-only.

On busy chains this drives partitions past ScyllaDB's large-partition warning thresholds (compaction_rows_count_warning_threshold = 100_000 rows; compaction_large_partition_warning_threshold_mb = 1000), causing sustained "Writing large partition" warnings, ~13x LCS write amplification on repeatedly-rewritten partitions, and degraded cache/compaction behavior. The ChainState partitions are the only partition type in the schema with unbounded per-partition growth.

Measured data (production, testnet_conway)

Per-structure row counts for one large ChainState partition (a retired prediction-market chain), measured 2026-06-09/10 directly on a validator's ScyllaDB via single-partition clustering-range COUNT(*) (validator image = testnet_conway commit 84adeb0, whose view key layout is 0x01 + u32-LE field index per level):

structure	rows	share	needed after a checkpoint at tip?
received_log	8,332,095	55%	no
application KV state (execution_state.users)	4,336,259	29%	yes (live state, app-owned)
confirmed_log (= block_hashes on main)	2,402,433	16%	no
system execution state + misc	~30	~0%	yes
total	~15.07M

So ~71% of the partition (~10.7M rows) is protocol history that no node needs once a checkpoint exists above it. In a 24h sample, warning volume per GKE validator ranged from ~100 events (one validator with few but very large partitions) to ~22k events; on the OVH validator single partitions have reached 2–3 GB, crossing the byte-size threshold as well. (The application-owned 29% is tracked separately as an application data-model concern; it is out of scope here.)

Why checkpoints make this tractable

Checkpoint bootstrap already removes the need for pre-checkpoint history during sync:

• bootstrap_chain_from_checkpoint (linera-core/src/client/mod.rs) lets a node reach the producer's exact state hash by downloading only the checkpoint certificate, its CheckpointExecutionState blob(s), and the chain's referenced blobs (OracleResponse::Checkpoint, Ensure checkpoint-bootstrapped nodes have all referenced blobs. #6364). Verified end-to-end by test_checkpoint_bootstrap across memory / storage-service / RocksDB.
• Checkpoint with messages #6399 (merged) lifted the no-messages restriction. Two event-related preconditions remain: a chain that has published events cannot checkpoint (prepare_checkpoint rejects on non-empty previous_event_blocks), and a chain that has consumed events cannot either (check_checkpoint_preconditions rejects on non-empty next_expected_events).

Because a node syncing past a checkpoint at height H never reads pre-H received_log / block_hashes entries, those rows can be archived to object storage (GCS/S3) and deleted from primary storage, keeping each ChainState partition bounded by (current execution state + history since the last checkpoint) — regardless of chain age. Restores from the archive should be rare and may be slow; the application/operator layer owns the archive, not the database.

Proposal (sketch — needs discussion)

1. Auto-checkpointing policy: checkpoint each chain periodically (e.g. every N months), skipping chains with no activity since their last checkpoint. The most recent blocks always stay in primary storage; only history below the latest checkpoint is ever archived.
2. Range-prune the historical sub-structures below H (selective row deletion inside the ChainState partition — the partition itself obviously stays, it holds the live state):
   • block_hashes[height < H]: keys are big-endian heights, so a range delete is expressible today via CustomMapView::remove per height or a bounded scan.
   • received_log[index < N] (N = first index at height ≥ H): requires a new LogView prefix/range-delete primitive — today LogView only has push/clear (all-or-nothing), and its BCS-u32 little-endian index keys are not lexicographically range-scannable. The stored_count/front bookkeeping must be updated accordingly.
3. Archive before delete: export the pruned rows (and optionally the corresponding pre-H block/certificate/event partitions, which are separate root_keys and cleaner to handle) to object storage in a durable format.
4. Keep durable: the checkpoint certificate at H and its CheckpointExecutionState blobs (mirror them to the archive bucket as well), and all blobs referenced by used_blobs — blobs are never dropped under this proposal.

Open questions

• Bootstrap authoritativeness: latest_checkpoint_height is self-advertised, and clients silently fall back to full-history sync when a checkpoint cert can't be served. Deleting history requires a guarantee that every (re)joining node can bootstrap from a durable checkpoint. What makes checkpoint availability authoritative?
• Event-stream preconditions: chains that publish or consume event streams still cannot checkpoint (see above). Many real application chains do both; do these preconditions need lifting (analogous to Checkpoint with messages #6399, with snapshot reconciliation for stream state) before the proposal is broadly applicable?
• Where the archival/GC job runs (per-validator vs coordinated), the archive format, and how a restore is triggered/served.
• Should the redundant height→hash duplication between block_hashes (inside ChainState) and the separate RootKey::BlockByHeight(chain_id) index be collapsed as part of this work?
• Checkpoint cost: dump_content re-reads and re-writes the entire execution state; for chains with millions of live app-KV rows this is significant and untested at that scale. Does auto-checkpointing need chunked/incremental dumping?

[ma2bd]

block_hashes is precisely the fix to allow removals from the confirmed logs later. received_log may need the same treatment.

We may want to "bucket" entries in those things at some points to spare DB keys at some point.