[Feature] Sparse Delta Column Group (SDCG) for lake PK column-mode partial update

Adds an experimental, DEFAULT-OFF sparse overlay format for shared-data (lake)
primary-key column-mode partial updates, plus an adaptive write-mode selector.

What it adds (all opt-in; zero behavior change at default config):
- Sparse `.spcols` delta column group: a partial update writes only the K updated
  cells (source_rowid + values) instead of a row-complete dense `.cols`, cutting
  write amplification for wide/low-coverage updates on large segments. Gated by
  `enable_sparse_dcg` (default false).
- partial_update_mode=auto: a per-load + per-segment cost model that picks the
  cheapest write mode (row / dense / sparse; packed / masked-dense for flexible)
  from update width, coverage, segment size and a per-table read-amp budget.
  Opt-in per load; default mode stays `row`.
- Flexible (per-row heterogeneous column sets) partial update via a hidden
  `__cset__` slot; packed sparse + masked-dense emit paths.
- Read path: LayeredOverlayColumnIterator merges sparse overlays on read with
  per-column presence gating; sound zone-map pruning for overlay columns.
- Background convergence: DCG-depth-aware lake PK compaction folds sparse chains;
  compaction-conflict replay (incl. racing-DELETE reconciliation) preserves
  overlays raced by a compaction. Gated by
  `enable_sdcg_compaction_conflict_replay` (default false).
- Observability: sdcg_write_mode_* and sdcg_compaction_conflict_* metrics.

Safety: inert at default config. `enable_sparse_dcg`,
`enable_sdcg_compaction_conflict_replay`, and `sdcg_enable_per_column_zone_map`
all default false; `partial_update_mode` defaults to `row`. With defaults, no
new code path executes and dense/row behavior is byte-identical. All proto
changes are additive (optional/repeated, no reused ordinals). FE flexible/auto
planning is reached only when the user opts into the corresponding mode.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NrM3YZ3WD5d7UPxp7D8tTd
Signed-off-by: meegoo <meegoo.sr@gmail.com>

Author: meegoo

be/src/common/config.h

@@ -461,6 +461,102 @@ CONF_mDouble(update_compaction_ratio_threshold, "0.5");
 // This config controls max memory that we can use for partial update.
 CONF_mInt64(partial_update_memory_limit_per_worker, "2147483648"); // 2GB
 
+// === SDCG (Sparse Delta Column Group) ===
+// Master gate for the sparse delta column group write path (lake / shared-data only).
+// When false the behavior is byte-identical to today: column-mode partial update
+// always writes dense `.cols`, and no `.spcols` files are produced or referenced.
+CONF_mBool(enable_sparse_dcg, "false");
+// Density decision threshold: when (K updated rows / M source-segment rows) is at
+// least this ratio, take the dense `.cols` path instead of writing a sparse `.spcols`.
+CONF_mDouble(sdcg_dense_threshold, "0.3");
+// Absolute cap on K (updated rows in one source segment) for the sparse path: at or
+// above this many rows, fall back to dense even if the density ratio is below
+// sdcg_dense_threshold (large K makes per-ordinal random seeks costlier than a full
+// sequential scan; lake/object-storage favors the conservative cap).
+CONF_mInt64(sdcg_sparse_max_rows, "50000");
+// Cost-model lower gate on the source-segment row count M: only take the sparse `.spcols` path when the
+// base segment is at least this large. Sparse's per-file framing + 8B/row source_rowid + read-amp only
+// pay off when dense's whole-column rewrite (M rows × col width) is genuinely expensive, which scales
+// with M. On small segments (or K spread thin across many small segments, large R) dense is faster and
+// reads flatter even though sparse writes fewer bytes. M below this floor -> dense. Validated: small
+// segment (M~31k) sparse LOST, large segment (M>=100k/1M) sparse won 2-7x. 0 disables the gate.
+CONF_mInt64(sdcg_sparse_min_segment_rows, "65536");
+// SAFETY-VALVE cap on sparse overlay-chain depth. Convergence of the chain is normally done by
+// BACKGROUND lake PK compaction (it reads input segments through the DCG overlay and emits fresh dense
+// segments, dropping the sparse chain). Only if a chain reaches this HIGH cap before compaction
+// converges it does the writer fall back to a synchronous in-place dense rewrite -- a rare safety net,
+// no longer a per-batch convergence mechanism, so it does not inflate publish-time p95. (Was 16, which
+// fired a depth-16 dense rewrite on the load's critical path every ~16 partial-update batches.)
+CONF_mInt32(sdcg_promotion_hard_count, "256");
+// WIDE-chain aggressive-fold trigger. A WIDE sparse overlay chain (per-layer value bytes/row >=
+// SDCG_WIDE_BYTES_PER_ROW) read-amplifies ~30ms/layer (a wide value column is re-decoded per layer per
+// read), vs a narrow chain's ~negligible per-layer cost. So background convergence folds a WIDE chain at
+// this MUCH lower depth than the default SDCG_COMPACTION_TRIGGER_DEPTH (10) used for narrow chains, to
+// keep wide reads near depth-1. The compaction SCORE is renormalized so a wide chain at this trigger
+// still crosses the FE min_score on its own. Set >= 10 to disable the wide fast-fold (treat wide like
+// narrow). Folding still reuses the SAME validated background fold path (no write-path change).
+CONF_mInt64(sdcg_compaction_trigger_depth_wide, "2");
+// UNIFIED single user knob — per-column allowed READ-AMP BUDGET R_max (= max sparse overlay layers a
+// read must merge). Collapses the read-amp control into ONE number, mapped internally to the dense/sparse
+// gate + the fold trigger:
+//   <0 (default -1) = AUTO: per-column-WIDTH defaults (wide cols fold near depth-1 via
+//                    sdcg_compaction_trigger_depth_wide, narrow at the default depth) — the validated behavior.
+//   0  = force DENSE (no sparse overlay ever): write-time merge via column-dense (supersede) / row;
+//        zero sparse read-amp. (Fixed-column path in this first version; flexible-path budget TBD.)
+//   N>0 = allow sparse, fold the chain when it would exceed N layers (uniform trigger = N+1), overriding
+//        the per-width auto defaults. Larger N = more write throughput, more read-amp.
+// Design: handbook/plans/active/2026-06-01-partial-update-sdcg-design.md §5.2.4.
+CONF_mInt64(sdcg_read_amp_budget, "-1");
+// partial_update_mode=auto level-1 (per-load) column-vs-ROW routing by UPDATE WIDTH. ROW (whole-row
+// rewrite) wins over the column overlays when many columns change at once: sparse pays per-column
+// machinery (a roaring bitmap + column file + apply-time DCG merge PER updated column) and dense pays
+// M_seg*W_upd, both of which exceed row's bulk rewrite once the update is wide. So auto routes ROW when
+// the update touches a large fraction of value columns (>= sdcg_auto_row_width_frac) OR a large absolute
+// count (>= sdcg_auto_row_width_min_cols); otherwise COLUMN, whose per-segment dense/sparse format is
+// chosen at apply. Read-strictness no longer forces ROW (dense is read-flat too) -- it is handled at
+// level-2. Calibration knobs (mutable so they can be tuned on a live cluster). See project_sdcg_auto_mode.
+CONF_mDouble(sdcg_auto_row_width_frac, "0.5");
+CONF_mInt32(sdcg_auto_row_width_min_cols, "8");
+// partial_update_mode=auto level-2 (per-segment) sustained-load penalty: sparse write throughput decays
+// as the overlay chain deepens (chains outrun compaction), so scale the sparse/packed WRITE cost by
+// (1 + sdcg_auto_sparse_depth_penalty * chain_depth). This makes the cost model migrate sparse->dense as
+// depth grows under sustained load, instead of staying on sparse until the hard depth valve trips.
+// DEFAULT 0 (disabled): cluster tests showed a nonzero penalty makes auto OSCILLATE sparse<->dense with no
+// hysteresis, producing mixed overlay chains that read AND write worse than any single mode, while pure
+// sparse self-limits (compaction plateaus its depth). Kept as a mutable knob for future calibration, off
+// by default. Mutable for live calibration.
+CONF_mDouble(sdcg_auto_sparse_depth_penalty, "0");
+// DEPRECATED (no longer drives the write path): the old in-place-promotion threshold as a fraction of
+// the base segment row count M. Kept for config compatibility only; convergence is now
+// background-compaction-driven, not cum_K/M-threshold-driven.
+CONF_mDouble(sdcg_promotion_threshold, "0.3");
+// Per-column segment-level zone-map gate refinement. Historically, when a segment has ANY delta
+// column group, segment-level zone-map pruning is disabled for ALL non-key columns, because a DCG
+// rewrites some columns and the base segment's zone maps are stale for them. That is overly broad:
+// a column NOT present in any DCG still has fully valid base zone maps and can be pruned safely.
+// When true, segment-level zone-map pruning is restored per-column for columns absent from every
+// DCG (and for columns whose DCG layer stack contains no SPARSE overlay -- the dense `.cols` file's
+// own zone map already governs page-level pruning, see SegmentZoneMapPruner). Columns covered by a
+// SPARSE overlay are never segment-pruned (the overlay carries newer values). This refinement also
+// benefits stock dense-DCG tables (flag enable_sparse_dcg off), so it is gated separately. DEFAULT
+// false: this is the ONE SDCG read-path refinement that would otherwise change behavior for existing
+// dense-DCG tables even with enable_sparse_dcg off; keeping it off preserves the historical
+// all-or-nothing pruning so SDCG is fully inert by default (GA opt-in posture). Set true to opt into
+// the per-column pruning refinement (correct + reversible). See project_sdcg_gap_audit.
+CONF_mBool(sdcg_enable_per_column_zone_map, "false");
+// SDCG compaction-conflict REPLAY (A-family, PK-keyed). When a full lake PK compaction is about to be
+// discarded because concurrent column-mode partial updates wrote racing SPARSE_PERCOL `.spcols` overlays
+// (version > compact_version) on its input segments, instead KEEP the compaction output and re-apply
+// those racing column values onto the new output segments keyed by PK: read the base PK at each racing
+// source_rowid, look the PK up in the (already-updated-to-output) primary index to get its new
+// (output_rssid, output_rowid), and emit an equivalent `.spcols` overlay on the output segment. This
+// gives the full compaction a forward-progress guarantee under sustained ingest (it no longer starves on
+// a hot segment) without blocking writes. Bounded to the strictly-replayable case: pure homogeneous
+// SPARSE_PERCOL races with a uniform updated-column set and NO racing delete (delvec advance); any
+// dense/flexible/inline/IDG race, a non-uniform column set, or a racing delvec still falls back to the
+// historical discard. Requires enable_sparse_dcg. EXPERIMENTAL, default OFF.
+CONF_mBool(enable_sdcg_compaction_conflict_replay, "false");
+
 CONF_mInt32(repair_compaction_interval_seconds, "600"); // 10 min
 CONF_Int32(manual_compaction_threads, "4");
 

be/src/common/config_compaction_fwd.h

@@ -77,6 +77,40 @@ CONF_mInt32(update_compaction_delvec_file_io_amp_ratio, "2");
 // This config defines the maximum percentage of data allowed per compaction
 CONF_mDouble(update_compaction_ratio_threshold, "0.5");
 
+// WIDE-chain aggressive-fold trigger. A WIDE sparse overlay chain (per-layer value bytes/row >=
+// SDCG_WIDE_BYTES_PER_ROW) read-amplifies ~30ms/layer (a wide value column is re-decoded per layer per
+// read), vs a narrow chain's ~negligible per-layer cost. So background convergence folds a WIDE chain at
+// this MUCH lower depth than the default SDCG_COMPACTION_TRIGGER_DEPTH (10) used for narrow chains, to
+// keep wide reads near depth-1. The compaction SCORE is renormalized so a wide chain at this trigger
+// still crosses the FE min_score on its own. Set >= 10 to disable the wide fast-fold (treat wide like
+// narrow). Folding still reuses the SAME validated background fold path (no write-path change).
+CONF_mInt64(sdcg_compaction_trigger_depth_wide, "2");
+
+// UNIFIED single user knob — per-column allowed READ-AMP BUDGET R_max (= max sparse overlay layers a
+// read must merge). Collapses the read-amp control into ONE number, mapped internally to the dense/sparse
+// gate + the fold trigger:
+//   <0 (default -1) = AUTO: per-column-WIDTH defaults (wide cols fold near depth-1 via
+//                    sdcg_compaction_trigger_depth_wide, narrow at the default depth) — the validated behavior.
+//   0  = force DENSE (no sparse overlay ever): write-time merge via column-dense (supersede) / row;
+//        zero sparse read-amp. (Fixed-column path in this first version; flexible-path budget TBD.)
+//   N>0 = allow sparse, fold the chain when it would exceed N layers (uniform trigger = N+1), overriding
+//        the per-width auto defaults. Larger N = more write throughput, more read-amp.
+// Design: handbook/plans/active/2026-06-01-partial-update-sdcg-design.md §5.2.4.
+CONF_mInt64(sdcg_read_amp_budget, "-1");
+
+// SDCG compaction-conflict REPLAY (A-family, PK-keyed). When a full lake PK compaction is about to be
+// discarded because concurrent column-mode partial updates wrote racing SPARSE_PERCOL `.spcols` overlays
+// (version > compact_version) on its input segments, instead KEEP the compaction output and re-apply
+// those racing column values onto the new output segments keyed by PK: read the base PK at each racing
+// source_rowid, look the PK up in the (already-updated-to-output) primary index to get its new
+// (output_rssid, output_rowid), and emit an equivalent `.spcols` overlay on the output segment. This
+// gives the full compaction a forward-progress guarantee under sustained ingest (it no longer starves on
+// a hot segment) without blocking writes. Bounded to the strictly-replayable case: pure homogeneous
+// SPARSE_PERCOL races with a uniform updated-column set and NO racing delete (delvec advance); any
+// dense/flexible/inline/IDG race, a non-uniform column set, or a racing delvec still falls back to the
+// historical discard. Requires enable_sparse_dcg. EXPERIMENTAL, default OFF.
+CONF_mBool(enable_sdcg_compaction_conflict_replay, "false");
+
 CONF_mInt32(repair_compaction_interval_seconds, "600"); // 10 min
 
 // if compaction of a tablet failed, this tablet should not be chosen to

be/src/common/config_fwd_headers_manifest.json

@@ -716,6 +716,8 @@
         "lake_compaction_max_rowset_size",
         "lake_enable_compaction_async_write",
         "lake_pk_compaction_max_input_rowsets",
+        "sdcg_compaction_trigger_depth_wide",
+        "sdcg_read_amp_budget",
         "lake_pk_compaction_min_input_segments",
         "enable_lake_pk_compaction_score_gate",
         "lake_pk_compaction_min_level_score",
@@ -724,6 +726,7 @@
         "lake_pk_compaction_delvec_benefit_weight",
         "lake_pk_compaction_size_overflow_ratio",
         "enable_light_pk_compaction_publish",
+        "enable_sdcg_compaction_conflict_replay",
         "enable_lake_compaction_use_partial_segments",
         "enable_lake_compaction_range_split",
         "lake_compaction_chunk_size"
@@ -739,6 +742,17 @@
         "transaction_apply_worker_idle_time_ms",
         "update_compaction_per_tablet_min_interval_seconds",
         "partial_update_memory_limit_per_worker",
+        "enable_sparse_dcg",
+        "sdcg_dense_threshold",
+        "sdcg_sparse_max_rows",
+        "sdcg_sparse_min_segment_rows",
+        "sdcg_promotion_hard_count",
+        "sdcg_read_amp_budget",
+        "sdcg_auto_row_width_frac",
+        "sdcg_auto_row_width_min_cols",
+        "sdcg_auto_sparse_depth_penalty",
+        "sdcg_promotion_threshold",
+        "sdcg_enable_per_column_zone_map",
         "enable_pk_index_eager_build",
         "pk_index_eager_build_threshold_bytes",
         "pk_index_parallel_compaction_threadpool_max_threads",