Skip to content

Commit 75381ff

Browse files
ar2rsawseenclaude
ar2rsawseen
and
claude
committed
docs(ui): scenario picker — all four cutover topologies, spelled out
The four supported scenarios, now selectable on the Guide tab (persisted) with per-scenario checklists, and as a decision matrix in the RUNBOOK: 1. Two clusters, no mirroring (plain switch) -> bound UNSET; top-up chases new old-cluster data until the final drain. 2. Two clusters, mirror old->new (old primary) -> bound SET at the tee flip; detect + one-click apply; badge on every pod; sync parity during validation; GDPR/merge re-apply before sign-off. 3. Two clusters, mirror new->old (rollback net) -> same machinery as 2; the detector is direction-agnostic (ClickHouse starts cold either way). 4. Single cluster, in-place upgrade -> bound UNSET; the upgrade froze mongo drill collections; live-parallel path with backpressure protecting the production ClickHouse. The picker states the one decision that changes configuration — tee or no tee — because a wrong bound setting is the one mistake the tool cannot detect afterwards (unset with a tee = undetectable duplicates; set without one = orphaned new data). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent a5b5e78 commit 75381ff

2 files changed

Lines changed: 76 additions & 0 deletions

File tree

‎docs/RUNBOOK.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,24 @@ incident.
8484
drill → optionally `bench/seed-failures.ts` for a full failure-scenario drill
8585
(breaker, DLQ, monitor, retry-failed).
8686

87+
## Choose your scenario first
88+
89+
The one decision that changes the configuration is whether a TEE mirrors
90+
the same requests into both stacks. Everything else is shared machinery.
91+
92+
| # | Topology | LEDGER_CD_UPPER_BOUND | Ingestion switch | New data arriving in old Mongo | Sign-off |
93+
|---|---|---|---|---|---|
94+
| 1 | Two clusters, **no mirroring** (plain switch) | **UNSET** | Before the migration (cutover-first) or after the bulk (bulk-before-cutover + final drain) | **Migrated** — top-up passes chase it until the drain finds nothing | Verify + audits, DLQ = 0 |
95+
| 2 | Two clusters, **mirror old → new** (old primary) | **SET** = tee flip | At customer sign-off | **Never migrated past the bound** — it is the tee's copy (different _id/cd; duplicates would be undetectable) | Verify + audits for pre-bound; dashboard comparison + sync parity for post-bound |
96+
| 3 | Two clusters, **mirror new → old** (new primary, old = rollback net) | **SET** = the moment new became primary | Already happened at the flip | Same as 2 — post-flip old-side docs are mirror copies | Same as 2 |
97+
| 4 | **Single cluster, in-place upgrade** (drill mongo → ClickHouse in background) | **UNSET** | The upgrade itself is the switch; old drill collections freeze | Transition tail drained by top-up; no tee → nothing to duplicate | Verify + audits, DLQ = 0 (live-parallel path; backpressure protects prod CH) |
98+
99+
Scenario is also selectable on the dashboard's **Migration Guide** tab —
100+
it renders the per-scenario checklist and states the bound requirement.
101+
For 2 and 3: use **Detect boundary** + **Apply this bound to the run**
102+
(one click covers all pods), verify the `bounded · cd < …` badge on every
103+
pod, and keep re-running sync parity during the validation window.
104+
87105
## Tee-mirror cutover (customer keeps the old architecture until sign-off)
88106

89107
For customers who require approval before switching: the old arch stays

‎src/http/ledger-viz-route.ts‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -322,6 +322,12 @@ const PAGE = `<!doctype html>
322322
323323
<!-- ══════════════════ GUIDE ══════════════════ -->
324324
<div class="pane" id="pane-guide">
325+
<div class="card">
326+
<h2>Which cutover is this? <span class="hint">— pick once; the settings below are per-scenario, and the wrong bound setting is the one mistake this tool cannot detect afterwards</span></h2>
327+
<div id="scenario-picker" style="display:flex;gap:8px;flex-wrap:wrap;margin-bottom:10px"></div>
328+
<div id="scenario-steps"></div>
329+
</div>
330+
325331
<div class="card">
326332
<h2>Preflight checks <span class="hint">— run anytime; read-only</span></h2>
327333
<button class="btn primary" id="btn-preflight" onclick="runPreflight(this)">Run preflight checks</button>
@@ -805,6 +811,58 @@ async function tick() {
805811
let dlqOffset = 0;
806812
function dlqPage(delta) { dlqOffset = Math.max(0, dlqOffset + delta); slowTick(); }
807813
814+
var SCENARIOS = [
815+
{ id: 'switch', name: '1 \u00b7 Two clusters, no mirroring',
816+
bound: false,
817+
html: '<p><b>Plain ingestion switch</b> (cutover-first, or bulk-before-cutover with a final drain).</p>' +
818+
'<ul>' +
819+
'<li><b>LEDGER_CD_UPPER_BOUND: LEAVE UNSET.</b> The migration must take everything, including data still arriving in the old cluster \u2014 top-up passes chase it until the final drain finds nothing new.</li>' +
820+
'<li>Cutover-first: switch SDK ingestion to the new cluster, then run the migration (old drill data is frozen). Bulk-before-cutover: run the bulk first, switch ingestion, then let the final top-up pass drain the tail.</li>' +
821+
'<li>Ignore the Tee boundary card \u2014 it is for mirrored setups only. Applying a bound here would ORPHAN newly arrived data.</li>' +
822+
'<li>Sign-off: Verify + Audit vs source + content audit, DLQ pending = 0.</li>' +
823+
'</ul>' },
824+
{ id: 'tee-old', name: '2 \u00b7 Mirror old \u2192 new',
825+
bound: true,
826+
html: '<p><b>Old cluster stays primary; nginx tees the same requests to the new stack</b>, which re-ingests them with its own logic (same event = different _id/cd on each side \u2014 nothing downstream can dedup across that seam).</p>' +
827+
'<ul>' +
828+
'<li>Flip the tee inside a ~60s old-ingestion pause (SDKs queue and retry \u2014 lossless). The pause creates a sharp boundary.</li>' +
829+
'<li>Run <b>Detect boundary</b> (Overview tab) BEFORE the migration maps anything: a green gap = exact seam. Click <b>Apply this bound to the run</b> \u2014 one click covers every pod \u2014 or set <code>LEDGER_CD_UPPER_BOUND</code> in the deployment config.</li>' +
830+
'<li><b>Every pod must show the</b> <code>bounded \u00b7 cd &lt; \u2026</code> <b>badge.</b> A pod without it migrates post-flip data = undetectable duplicates.</li>' +
831+
'<li>During validation, re-run the boundary card\u2019s sync parity regularly: nginx mirror drops silently when the secondary is down; flagged hours are bounded backfill windows.</li>' +
832+
'<li>GDPR erasures / user merges done on the old side during validation must be re-applied on the new stack before sign-off.</li>' +
833+
'</ul>' },
834+
{ id: 'tee-new', name: '3 \u00b7 Mirror new \u2192 old',
835+
bound: true,
836+
html: '<p><b>New cluster is already primary; nginx mirrors back to the old stack</b> as the customer\u2019s rollback safety net during validation.</p>' +
837+
'<ul>' +
838+
'<li>Everything from scenario 2 applies unchanged \u2014 detection, bound, badge, sync parity. ClickHouse is the store that started cold in both directions, so the detector does not care which side is primary.</li>' +
839+
'<li>The bound = the moment the new cluster became primary. Old-cluster docs after it are the mirror\u2019s copies \u2014 never migrate them.</li>' +
840+
'<li>Post-flip data quality is validated by comparing dashboards between stacks, not by this tool: this migration owns only the pre-flip history.</li>' +
841+
'</ul>' },
842+
{ id: 'inplace', name: '4 \u00b7 Single cluster, in-place upgrade',
843+
bound: false,
844+
html: '<p><b>One cluster upgraded to the new architecture</b>; old drill_events collections stay in the same MongoDB and are migrated into ClickHouse in the background while live traffic already flows into the same table.</p>' +
845+
'<ul>' +
846+
'<li><b>LEDGER_CD_UPPER_BOUND: LEAVE UNSET.</b> The upgrade itself froze the Mongo drill collections (new events go to ClickHouse natively); if a transition tail trickled in, top-up drains it. There is no tee, so there is nothing to duplicate.</li>' +
847+
'<li>Live rows and migrated rows share the table safely: live cd is post-upgrade, migrated cd is historical \u2014 verify and the audits are built on that boundary (this is the tested live-parallel path).</li>' +
848+
'<li>Backpressure matters most here \u2014 migration ATTACHes into the same ClickHouse serving production; the built-in throttle yields before parts pressure reaches live inserts.</li>' +
849+
'<li>Sign-off: Verify + Audit vs source + content audit, DLQ pending = 0. Retention TTL deletions on old data show as drift, not defects.</li>' +
850+
'</ul>' },
851+
];
852+
function renderScenario() {
853+
var sel = localStorage.getItem('mig-scenario') || '';
854+
document.getElementById('scenario-picker').innerHTML = SCENARIOS.map(function(s) {
855+
return '<button class="btn' + (sel === s.id ? ' primary' : '') + '" onclick="pickScenario(\\'' + s.id + '\\')">' + s.name + '</button>';
856+
}).join('');
857+
var cur = SCENARIOS.find(function(s) { return s.id === sel; });
858+
document.getElementById('scenario-steps').innerHTML = cur
859+
? '<div style="border-left:3px solid var(--' + (cur.bound ? 'blue' : 'green') + ');padding-left:12px">' + cur.html +
860+
'<p><b>Bound setting for this scenario: ' + (cur.bound ? 'REQUIRED \u2014 <code>LEDGER_CD_UPPER_BOUND</code> set (or applied from the boundary card)' : 'must stay UNSET') + '</b></p></div>'
861+
: '<div class="empty">Select the scenario \u2014 it decides whether the cd bound must be set (mirrored setups) or must stay unset (everything else).</div>';
862+
}
863+
function pickScenario(id) { localStorage.setItem('mig-scenario', id); renderScenario(); }
864+
renderScenario();
865+
808866
async function applyBound(btn, ms) {
809867
if (!armed.get(btn)) {
810868
armed.set(btn, true);

0 commit comments

Comments
 (0)