Design spike: close during-snapshot p99.9 (291ms) via per-key incremental delta snapshots

[ELares]

Design spike: close the during-snapshot p99.9 (291ms -> ms-class) via per-key incremental delta snapshots

Owner go/no-go. Recommendation: GO Phase 0 + Phase 1; gate Phase 2/3 on a measurement.

1. Root-cause verdict (CONFIRMED)

The 291ms during-snapshot p99.9 is a fundamental shared memory-bandwidth floor, not a scheduling bug. The ic-persist-<shard> thread reads EVERY frozen entry once, deep-clones it (to_kvobj, store/src/kvobj.rs:2912), and encodes it (encode_kvobj, repl/src/kvcodec.rs:156) into the body Vec. On an all-cores-busy thread-per-core datapath (ADR-0002) this full-keyspace read saturates the memory controller and lengthens every concurrent op's tail. The write+fsync is once-per-shard and negligible.

Proof it is bandwidth, not CPU/scheduling — the five failed levers (all c7g):

• yield-between-chunks (perf: yield the serving shard between snapshot chunks (bounded BGSAVE tail, #518 durable-load) #571): 3.5s (yielding doesn't touch bandwidth)
• time-proportional throttle (perf: save_backpressure_percent throttle -- cheap stopgap for the concurrent-snapshot p99.9 (#576 PR-A) #577/perf: save_backpressure_percent throttle for the concurrent-snapshot p99.9 (closes #577) #578): 3.5s -> 6.8s@25pct -> 16.75s@10pct (throttle stretches the wall-time => LONGER contention window => WORSE)
• dedicated-core pin (perf: close the durable-snapshot tail residual (303ms -> ms-class) via persist-thread core/bandwidth isolation #589/perf: pin the ic-persist thread to a dedicated core so the snapshot encode stops stealing a serving core (part of #589) #590): 1125ms, 3.9x WORSE (slower encode => longer window; can't relocate the memory controller)
• per-slot Arc-COW (perf: per-slot Arc copy-on-write snapshot isolation, uncontended datapath during a save (#576 core) #588, SHIPPED): removed the O(N) serving-core copy, 3.5s -> 291ms; residual = the irreducible read.

This corrects the "just yield like Dragonfly PR #4910" framing: that fixes CPU starvation; IronCache already tried the yield family and it does nothing here. The only lever that moves a bandwidth floor is reading fewer bytes.

Phase 0 must include a c7g perf stat DRAM-bandwidth/LLC-miss capture to confirm this inference directly (it's currently derived from the pin/throttle A/Bs, not measured).

2. Recommended lever

Incremental (dirty-tracked delta) snapshots at PER-KEY granularity. Only this reduces the persist READ footprint.

• Compression (ADR-0021): shrinks WRITE bytes, not READ. Floor untouched. Stack later.
• DF-style versioned-iterate (SNAPSHOT.md, never shipped): still re-reads the whole keyspace; wrong tool; much larger change ([DECISION]: Epoch-based reclamation (crossbeam-epoch) vs custom drain-list framework #33 epoch GC interlock).
• NUMA/non-temporal: ~10-40% of residual, cheap, doesn't remove the floor — fold into Phase 0.
• Cheap win the 5 attempts missed: each entry is copied TWICE (to_kvobj then encode_kvobj). Encode directly from the frozen Entry blob for the string-family common case => ~half the persist-thread memory traffic. Ship in Phase 0.

CRITICAL correction to the research: the proposed per-SLOT dirty bit does NOT work. DEFAULT_SLOTS_PER_DB = 256 and slot_index = scan_hash(key) & (slots-1) spread keys uniformly across 256 slots; at 10% SET / zipf essentially ALL 256 slots go dirty per window => ~0% read reduction. Granularity must be per-key (per-shard dirty-key set) or per-bucket.

3. Design

• Hook: add a per-shard dirty-key set in the single write funnel slot_table_mut (store/src/lib.rs:786), mirroring the reserved OnWrite attach point (touch_watch, lib.rs:800). Single-writer per shard => no atomics.
• Delta content: ShardDumpBuilder (persist/src/lib.rs:261) skips entries whose key isn't in the dirty set captured at the epoch cut; emits TOMBSTONEs for deletes. Base = full dump + clear set (inside the saving window). Delta = dirty keys + tombstones.
• Format (FORMAT_VERSION 1 -> 2): keep MAGIC=ICSS, per-shard .icss + dump.manifest as sole commit point (tmp->fsync->rename->dir-fsync). Manifest gains a per-shard CHAIN {base, [deltas], base_epoch, chain_epochs}; each delta is self-describing (base_epoch, own epoch, record type {PUT,TOMBSTONE}, body CRC32). v1 fail-closes on v2 (harden: format-version-mismatched snapshots must fail loudly, not boot silently empty #530).
• Warm-start: load base, replay deltas in manifest order (later wins; TOMBSTONE removes). Deterministic (ADR-0003: manifest-order = fixed integer sequence, no clock/RNG).
• Crash-safety: manifest-last stays the ONLY commit point; crash mid-delta loads the last committed prefix; torn delta caught by CRC => truncate chain to last good link. Never reference a delta whose file+CRC wasn't fsync'd first. Preserve the unsafe impl Send for FrozenSlot contract and the saving-gated freq-bump.
• Compaction: max_deltas_per_base (default ~8, tunable); on overflow the next save is a full base and the chain resets. Bounds warm-start replay + disk.
• forbid-unsafe preserved: dirty-set lives in ironcache-store; persist/repl stay #![forbid(unsafe_code)].

4. Effort / risk / phasing — XL, decomposed

• Phase 0 (Small, ship now, GO-independent): perf-counter confirm; halve the double deep-copy; MADV_SEQUENTIAL/non-temporal reads. Likely 291ms -> ~150-180ms, near-zero risk, no format bump.
• Phase 1 (Medium, INERT/default-off): per-key dirty set behind a flag; v2 delta format + manifest chaining; produce deltas but loader still base-only. Ships dark; measures dirty-key-bytes / total-bytes = the go/no-go for Phase 2.
• Phase 2 (Medium): enable delta warm-start; v1->v2 + v2->v2 forward-load CI; crash-injection ([TASK]: Seeded fault-injection and corruption scenarios #100).
• Phase 3 (Small-Med): compaction + base-rewrite + chain metrics.

Sharpest hazards: (1) dirty-key fraction may be large at scale => payoff evaporates (Phase 1 measures it first); (2) dirty-set insert on the HOT write path (may need roaring bitmap over scan_hash) — must pass headtohead qps gate; (3) base save still hits the floor => bimodal tail; (4) delta-chain crash-safety — needs crash-injection, not just unit tests.

5. Acceptance gate

• Perf: SNAPSHOT=1 SNAPSHOT_INTERVAL_SECS=2 MAXMEMORY=512mb scripts/bench/tail.sh on c7g (owner-gated; macOS can't measure). Report delta-save p99.9 AND base-save p99.9 separately (bimodal). Success = delta-save p99.9 materially below 291ms toward ms-class; base-save ~291ms and documented. Record the dirty-key-bytes fraction from Phase 1.
• No regression: scripts/bench/headtohead.sh qps flat (perf-gate budget protocol per [TASK]: Dash-style extendible-hashing table (clear uniform win on memory AND O(1) eviction) #285).
• Correctness: extend snapshot-forward-load CI (mixed-version-compat.yml): v1 base under v2, v2 base+chain round-trip, v1 fail-close on v2. Crash-recovery ([TASK]: Seeded fault-injection and corruption scenarios #100): kill mid-delta => last committed prefix, torn delta => CRC-truncate. cargo test --workspace --all-features green (mind the load-dependent raft deadlock; use the timeout stopgap).

6. Honest ceiling

Reaches "much better than 291ms" at low write ratios (plausibly 30-80ms delta-save), NOT par with Redis ~15ms, and the tail stays bimodal because base saves re-hit the floor. Redis's 15ms is a fork+COW artifact (child on another core pays the read); IronCache is forkless + all-cores-busy by design, so there is no free core/bandwidth to offload to. Dragonfly doesn't get par for free either (PR #4910: forkless serialize dropped 250K->8K qps, recovered to 70-80K). The payoff is contingent on a SMALL dirty-key set, which is unmeasured. So: ship Phase 0 + Phase 1 unconditionally; gate the XL Phase 2/3 on the measured dirty fraction; if it's large, document the residual as a principled forkless thread-per-core tradeoff rather than spend XL on faith.

Refs: store/src/lib.rs:786 (write funnel), :3099 (begin_save), :3050 (FrozenSlot), persist/src/lib.rs:190/261, format.rs:45/51 (MAGIC/FORMAT_VERSION), kvobj.rs:2912, kvcodec.rs:156; docs/design/SNAPSHOT.md, docs/bench/OPTIMIZATION_LOG.md; issues #571 #576 #577 #578 #585 #586 #588 #589 #590 #33 #67 #100 #530.

[ELares]

Part of the Dragonfly par-or-better epic #665 (loss #3). Verified in-code: the persist push_entry (crates/ironcache-persist/src/lib.rs) does entry.to_kvobj() (deep clone) THEN encode_kvobj(&kv) -- the double copy Phase 0(b) removes.

[ELares]

Phase 0(b) SHIPPED (#677 merged): persist per-entry copy halved

The persist path's double copy is gone: ShardDumpBuilder::push_entry now uses ironcache_repl::encode_entry_into, which encodes string entries DIRECTLY from the frozen blob (no to_kvobj deep-clone intermediate) + a reused scratch buffer. Byte-identical (17-case parity test + two-lens adversarial SHIP + full CI). forbid-unsafe preserved, no format bump.

REMAINING for the tail quantification + the Phase-1 go/no-go (bundled, one c7g session with a debugged harness):

• Phase 0 payoff: the tail.sh SNAPSHOT=1 A/B extraction failed this run (empty p999; box auto-torn-down before inspection). Redo with full log capture + robust extraction (the p999_us line column shifts with/without a competitor; handle both).
• Phase 0(a)/(c): the perf-counter DRAM-bandwidth confirm + MADV_SEQUENTIAL/non-temporal reads (the other two cheap Phase-0 items, not yet built).
• Phase 1: the per-key dirty-set + v2 delta format (dark) to measure dirty-key-bytes/total = the gate for the XL Phase 2/3.

Phase 0(b) stands on its own as a safe allocation reduction; the quantification does not block it.

[ELares]

Phase 1a SHIPPED (PR #695): the per-shard dirty-key set

The foundation for the delta format is in: ShardStore now carries a per-shard set of mutated (db, key) pairs, fired from the existing observe_put/observe_remove write-funnel chokepoints (before their repl_active gate, so tracking is independent of replication).

Coverage is the complete per-key mutation set: create/overwrite, every in-place edit + TTL change (rmw/rmw_mut), delete/expiry/eviction, and FLUSHDB/FLUSHALL (per-key removes). SWAPDB creates/destroys no entry so it bypasses the funnel; an adversarial review caught this as a MEDIUM (a delta would resurrect pre-swap contents), so swap_db now dirties both dbs explicitly (present keys re-encode, moved-out keys tombstone), gated so a default-off server keeps SWAPDB O(1). Two documented LOW exceptions remain for Phase 2 to confirm: remove_keys_where (aborted-migration purge, deliberately observer-silent, its keys must be proven never-in-a-base) and the currently-dead read-only-rmw Mutated-TTL arm.

INERT: default-off, nothing consumes the set yet. The Option IS the fast-path gate: Option<HashSet> is niche-optimized on the table pointer (verified against the repo's hashbrown: same 40 bytes as HashSet; off-path codegen is one load + compare-branch), so no new bool, no lockstep field, no measurable hot-path cost when off. API for the Phase-1b consumer: enable_dirty_tracking / take_dirty_keys (drain + reset at the epoch cut) / dirty_key_count (the numerator for the dirty-fraction gate) / dirty_tracking_active.

Verification: 6 unit tests (INERT-by-default, complete capture incl. tombstone, window reset, disable, SWAPDB present+tombstone, SWAPDB inert-when-off); ironcache-store 93/93, ironcache 370/370, clippy --all-features -D warnings clean; two-lens adversarial review both SHIP.

Next (Phase 1b): the v2 delta format + manifest chaining that CONSUMES this set at the epoch cut (produce deltas dark, loader still base-only), which then yields the dirty-key-bytes/total measurement = the go/no-go gate for the XL Phase 2/3.

[ELares]

Phase 1b (part 1): the dirty-fraction measurement -- the go/no-go, measured (PR #696)

Rather than build the v2 delta format on faith to get the number the owner gated Phase 2/3 on, Phase 1a's dirty-key set made it directly observable. New reproducible bin dirty_fraction (ironcache-bench) populates a base keyspace, enables tracking at the epoch cut, applies W zipfian writes through the real store, and reports distinct-dirty / N (exactly what a delta re-reads).

Result (1M keys, theta=0.99 = the production skew):

W (writes/interval)	dirty fraction	reuse
50,000	2.2%	2.3x
100,000	3.8%	2.6x
250,000	8.0%	3.1x
500,000	13.6%	3.7x
1,000,000	22.4%	4.5x
2,000,000	35.2%	5.7x
5,000,000	58.4%	8.6x

Skew HELPS: at W=1M the reuse (writes per distinct key) is 4.5x at theta=0.99 vs 1.8x at theta=0.5, i.e. more skew -> fewer distinct dirty keys. Real caches are skewed, so the fraction sits on the favorable (lower) end.

Read -- conditional GO. W = write_qps x snapshot_interval. In the frequent-snapshot / moderate-write regime -- the exact regime where the p99.9 tail bites repeatedly -- W is small and the dirty set is single-digit-percent, so a delta save re-reads a small slice of the keyspace = a real reduction of the persist read that sets the 291ms floor. It degrades gracefully to a base save as writes-per-interval approaches the keyspace (high write rate x long interval), which the planned periodic-base compaction (Phase 3, max_deltas_per_base) already bounds.

So the delta approach IS worth building for the frequent-snapshot tail, with the honest caveat that its payoff shrinks for infrequent snapshots under heavy writes (there, the tail is also rarer, so it matters less). This clears the owner's "don't spend XL on faith" gate with a measured number.

Next: the v2 delta FORMAT slice (FORMAT_VERSION 1->2, per-shard base + manifest delta chain, PUT/TOMBSTONE, CRC32, manifest-last commit) that consumes take_dirty_keys at the epoch cut and produces deltas dark, then the real c7g tail.sh confirmation (delta-save p99.9 vs base-save p99.9, bimodal).