Commit 205e432
authored
pyramid: two-phase iterative build (finest from source, then rollups) (#138)
* spec: two-phase iterative hex pyramid build to bound build-time memory
Replace the single 7-branch UNION ALL COPY with one finest-from-source
COPY followed by iterative parent rollups from the previous resolution.
Bounds working memory by output cardinality per step instead of by full
source size, so global high-cardinality datasets (GBIF, irrecoverable
carbon) build without OOM.
Also adjusts .gitignore to track committed specs under
docs/superpowers/specs/ while continuing to ignore the rest of
docs/superpowers/ (matches the precedent set by the existing 2026-04
specs).
* plan: implementation steps for two-phase iterative hex pyramid build
* tiles: bump layout version to v3-iterative for two-phase pyramid
* pyramid: two-phase iterative build (finest from source, then rollups)
* pyramid: rename AVG internal weight column; fix docstring; cover multi-col SUM
- Rename AVG-mode internal column count → __pyramid_weight to avoid collision
when a user names their value column "count"
- Fix register_hex_tiles docstring: "Other aggs" bullet now correctly states
aggregation happens at every resolution including finest (not passed-through raw)
- Add test_sum_mode_with_multiple_value_columns covering multi-value-column SUM
* test: end-to-end mathematical invariants for two-phase pyramid
* test: AVG weighted-mean fixture must span two h5 cells
Original fixture placed all five points within ~100m, so they collapsed
to a single h5 cell at the finest level. Phase 2 then saw one child per
parent, where SUM(v*w)/SUM(w) and an incorrect AVG(v) both yield the
same answer. The test would have passed even with the unweighted bug
it was meant to catch.
Spread the two clusters to (37.0, -122.0) and (37.2, -122.0) — distinct
h5 cells, same h4 parent — and add a sanity assertion that the finest
level really has two distinct h5 children before checking parent AVG.1 parent 9cc4462 commit 205e432
7 files changed
Lines changed: 1273 additions & 167 deletions
File tree
- docs/superpowers
- plans
- specs
- tests
- tiles
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | | - | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
12 | 14 | | |
13 | 15 | | |
14 | 16 | | |
| |||
0 commit comments