Skip to content

Commit f8421b6

Browse files
andre-salvaticlaude
andcommitted
docs: commit a worked data-divergence investigation as the skill's example
The data-divergence skill was the only one of the three shipping no example. This adds one: a full investigation of prod's batch vs SDP silver tables, re-run against live prod rather than transcribed, and structured the way the skill prescribes — layer counts, key-level diff, row_commit_version to pin the blast radius, a Delta-history timeline, proven-vs-inferred, costed fixes, and an appendix of the queries. It is kept for its findings, not its format. The reported complaint was one shifted date column; three further divergences turned up, including a live gap in prod.report.order_agg (2026-07-24 missing after a failed run, never backfilled because the incremental MERGE is scoped to seed_date). Also folds in the README rephrasing of the skill bullets. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent a90c7d1 commit f8421b6

4 files changed

Lines changed: 394 additions & 8 deletions

File tree

.claude/skills/data-divergence/SKILL.md

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
name: data-divergence
3-
description: Investigate why two datasets that should agree don't — two pipelines writing the same logical table, a rollup vs the detail it aggregates, a dashboard vs its source, one environment vs another. Use when row counts, totals, or date ranges disagree and the question is what happened rather than just what differs. Covers localizing the first layer that diverges, diffing by grain and by key, reconciling across an aggregation boundary, reading Delta history and row_commit_version, why append-only tables diverge permanently, and what a fix actually costs.
3+
description: Investigate why two datasets that should agree don't — two pipelines writing the same logical table, a rollup vs the detail it aggregates, a dashboard vs its source, one environment vs another. Use when row counts, totals, or date ranges disagree and the question is what happened rather than just what differs. Covers localizing the first layer that diverges, diffing by grain and by key, reconciling across an aggregation boundary, reading Delta history and row_commit_version, why append-only tables diverge permanently, and what a fix actually costs. See examples/ for a worked investigation.
44
---
55

66
# Investigating a data divergence
@@ -178,3 +178,19 @@ broken column **and** discards every frozen value it was carrying — you may be
178178
in one column for a divergence in another. Say which columns move, in both directions, before
179179
recommending it. Check the run state of the other path first too: resetting a table whose writer is
180180
currently failing leaves it empty.
181+
182+
## The worked example
183+
184+
`examples/2026-08-06-prod-batch-vs-sdp.md` is a full investigation of this repo's own `prod` catalog,
185+
written to the shape above: layer counts, a key-level diff, `row_commit_version` to pin the blast
186+
radius, a Delta-history timeline, then proven-vs-inferred and costed fix options.
187+
188+
It is the only place this skill names real tables, and it is worth reading for what it found rather
189+
than for the procedure. The reported complaint was one shifted date column; three further divergences
190+
turned up, including a **live gap in a production gold table** that nobody had reported — a failed
191+
daily run whose date-scoped incremental MERGE meant no later run ever backfilled it, while the
192+
full-overwrite layer above it self-healed and hid the failure. Two of the four also came from a
193+
single append-only commit freezing a column nobody intended to freeze.
194+
195+
Structure a report the same way, and keep the appendix of queries: the next investigation starts by
196+
editing them rather than by rewriting them.

0 commit comments

Comments
 (0)