daily: wire cells refresh (raw h3 → pyramid → cells.db → D1)

Adds the cells chain to `daily.yml` before the D1 import, and registers
`cells` in `api/d1-import.dvc` (cmd + `cells.db` dep). DVX-gated: the heavy
pyramid rebuild only runs on days the crash data changed; `d1-import` then
exact-diffs cells.db into D1 (~tens of writes/day). `d1-diff.py` now logs
the per-table delta so the first runs confirm that volume.

The R2 pyramid push for the filtered/parquet path (4b) is deferred — the
daily job has no R2 creds yet; the D1 default-view path (4a) needs none.

Not yet exercised in CI — must be validated via a scoped `workflow_dispatch`
run before the cron relies on it (a cells-stage failure would otherwise
block the site deploy).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: Ryan Williams

.github/workflows/daily.yml

@@ -154,10 +154,23 @@ jobs:
       - name: Post to Slack
         if: ${{ !inputs.targets }}
         run: $DVX njsp/data/slack_post.dvc
+      # Cells refresh for the map. DVX-gated: the pyramid rebuild only runs on
+      # days the crash data actually changed. `cells.db` (the D1 rollup source)
+      # feeds the d1-import below; the R2 pyramid push for the filtered/parquet
+      # path is deferred (needs R2 creds — see 4b).
+      - name: Rebuild cells raw H3 index
+        if: ${{ !inputs.targets }}
+        run: $DVX data/cells/raw/h3_r15.dvc
+      - name: Rebuild cells pyramid
+        if: ${{ !inputs.targets }}
+        run: $DVX data/cells/pyramid.dvc
+      - name: Rebuild cells rollup DB (D1 source)
+        if: ${{ !inputs.targets }}
+        run: $DVX data/cells/cells.db.dvc
       - name: Install api dependencies
         if: ${{ !inputs.targets }}
         run: cd api && pnpm install
-      - name: Sync NJSP crashes.db → D1
+      - name: Sync crash DBs (NJSP, cmymc, cells) → D1
         if: ${{ !inputs.targets }}
         run: $DVX api/d1-import.dvc
       - name: Regenerate homepage OG mosaic (→ S3)

api/d1-import.dvc

@@ -1,8 +1,9 @@
 meta:
   computation:
-    cmd: bash scripts/d1-import.sh --inplace njsp-crashes cmymc
+    cmd: bash scripts/d1-import.sh --inplace njsp-crashes cmymc cells
     deps:
       /www/public/njsp/crashes.db: 6ade2d0f110f6012132763f54b8f2a0e
       /www/public/njdot/cmymc.db: 9a59b8fefd7f0559c935d147c4622572
+      /data/cells/cells.db: a8d179291a9528aaa5924304341fca28
     git_deps:
-      scripts/d1-import.sh: 80bbb27c725d2f243dcb79d6f52406f378301d1b
+      scripts/d1-import.sh: 0020fb0125c04a7fd7f9c6573893e2cd1721ceb5

api/scripts/d1-diff.py

@@ -70,6 +70,10 @@ def main():
 
     conn.close()
 
+    # One-line delta summary → stderr, so the import log shows the actual
+    # daily write volume per table (expected ~tens for the cells rollup).
+    print(f"  d1-diff {table}: {len(delete_pks)} delete + {len(upsert_rows)} upsert row(s)", file=sys.stderr)
+
     out_del = Path(args.out_delete)
     out_ins = Path(args.out_upsert)
     out_del.parent.mkdir(parents=True, exist_ok=True)