[UMBRELLA] Persistent Store Support in MC/DC

[VladRodionov]

Summary

MC/DC currently assumes ephemeral cache semantics, enabling online training, dictionary GC, and probabilistic migration. This model does not work for persistent Redis use cases such as semantic caching or LLM memory. We must introduce a robust persistent-mode subsystem for dictionary lifecycle management.

---

Requirements

1. Dictionary Pinning / Persistence

• Dictionaries referenced by any persistent values must never be GCed
• Maintain refcounts or probabilistic retirement protection
• Ensure replica safety (replication must receive dicts before values)

2. Offline Dictionary Training Pipeline

• Support dictionary creation from RDB snapshots
• Allow manual mcdc dictionary install command
• Disable online training when in persistent mode

3. Safe Dictionary Lookup

• On decode failure, return detailed error (not silent deletion)
• Detect missing dictionary IDs early
• Provide command to inspect installed dictionaries

4. Controlled Migration Strategy

• Optional on-read migration with throttle
• Background low-rate scanner for recompression
• Ability to freeze dictionary version for stability

5. Replication Safety

• Introduce special MC/DC keys (e.g., mcdc.dict.<id>) for dictionary propagation
• Ensure dicts replicate before any dependent values
• Protect against missing-dictionary state on replicas

6. Namespace Modes

Every MC/DC namespace must explicitly declare:

• mode = cache (default, online training allowed)
• mode = persistent (online training disabled, offline only)

---

Deliverables

• Persistent-mode configuration per namespace
• Dictionary pinning + safe GC rules
• Offline dictionary upload + validation command
• Replication-safe dictionary distribution
• Background recompression support with throttling
• Enhanced decode error handling
• Commands for dictionary lifecycle management
• Documentation for persistent vs cache modes

---

Motivation

Redis is now widely used for:

• AI feature pipelines
• Semantic caching
• LLM memory and personalization
• Vector/embedding stores
• Durable inference caches

These workloads require durable, correct, future-proof compression. Online training alone cannot guarantee correctness for persistent datasets. MC/DC must support both models explicitly.

---