You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: lead region-subset guidance with paraphrase-robust hN IN subquery form (#325)
Regression pass (qwen, dev) after #322 showed the model paraphrases the
SEMI JOIN mask example into `SEMI JOIN read_parquet(...) s ... WHERE s.STUSPS=...`,
which fails with 'Referenced table s not found' — DuckDB SEMI JOIN does not bring
the joined table's columns into outer SELECT/WHERE scope. It cost ~5 retries
before the model converged on the hN IN (SELECT hN FROM <mask> WHERE <attr>) form.
Lead the region-subset section with that IN-subquery form (attribute filter lives
inside the subquery, so it can't be misplaced), and keep SEMI JOIN as a secondary
alternative with an explicit 'columns are not in outer scope — filter inside the
mask CTE' warning.
Copy file name to clipboardExpand all lines: h3-guide.md
+18-4Lines changed: 18 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -166,21 +166,35 @@ Some datasets carry NULL in their finest pre-computed parent column for very lar
166
166
167
167
**`h0` is the res-0 partition key — a coarse *storage* key, never a spatial or boundary filter.** Each res-0 base cell spans ~4.35 **million** km² (larger than any US state), so `WHERE h0 = …` or `WHERE h0 IN (…)` selects whole base cells, not the region. Florida sits inside a single base cell, so `WHERE h0 = <fl_cell>` renders the entire continental US; California spans two base cells, so `WHERE h0 IN (<ca_cell_1>, <ca_cell_2>)` renders both, far larger than the state. Resolving a region to its base cells (`SELECT DISTINCT h0 … WHERE STUSPS='CA'`) and filtering the value dataset by that `h0` set **alone** is always wrong — it clips to nothing finer than the base cells.
168
168
169
-
To clip a value dataset (carbon, land cover, biomass — anything under a global `hex/h0=*/`) to a named region, **join it to the region's hex mask on the finest shared resolution and filter by the mask's attribute.** The census state/county hexes are ordinary catalog datasets — find their exact path with `get_stac_details` like any other hex dataset. Build the mask (already filtered to the region) as a CTE, then `SEMI JOIN` it onto the raw `read_parquet(...)`*before* the `GROUP BY` (the MASK BEFORE AGGREGATE rule — this also prunes the value dataset's `h0` partitions):
169
+
To clip a value dataset (carbon, land cover, biomass — anything under a global `hex/h0=*/`) to a named region, **filter it by the region's hex mask at the finest resolution the two share, keyed on the mask's attribute.** The census state/county hexes are ordinary catalog datasets — find their exact path with `get_stac_details` like any other hex dataset.
170
+
171
+
The most robust form is an `IN` subquery: the attribute filter lives *inside* the subquery, so it can never be misplaced. Pair the `h8 IN (…)` boundary filter with a coarse `h0 IN (…)` prefilter to prune partitions:
172
+
173
+
```sql
174
+
SELECTc.h8, SUM(c.carbon) AS carbon
175
+
FROM read_parquet('<value_hex>', hive_partitioning = true) c
176
+
WHEREc.h0IN (SELECT DISTINCT h0 FROM read_parquet('<census_state_hex>', hive_partitioning = true) WHERE STUSPS ='CA')
177
+
ANDc.h8IN (SELECT DISTINCT h8 FROM read_parquet('<census_state_hex>', hive_partitioning = true) WHERE STUSPS ='CA')
178
+
GROUP BYc.h8;
179
+
```
180
+
181
+
Here `h8 IN (…)` is the real boundary (the finest shared resolution); `h0 IN (…)` only prunes which partition files are scanned — see the closing note.
182
+
183
+
A `SEMI JOIN` to a **pre-filtered mask CTE** is equivalent and also prunes partitions (the MASK BEFORE AGGREGATE rule). Filter the attribute *inside* the CTE and join with `USING` — **never reference the mask's columns in the outer query.** DuckDB `SEMI JOIN` tests row existence only; it does NOT bring the joined table's columns into the outer `SELECT`/`WHERE` scope, so `SEMI JOIN <mask> s … WHERE s.STUSPS='CA'` fails with `Binder Error: Referenced table "s" not found`:
170
184
171
185
```sql
172
186
WITH ca AS (
173
187
SELECT h8, h0
174
188
FROM read_parquet('<census_state_hex>', hive_partitioning = true)
175
-
WHERE STUSPS ='CA'
189
+
WHERE STUSPS ='CA'-- attribute filter lives HERE, inside the CTE
176
190
)
177
191
SELECTc.h8, SUM(c.carbon) AS carbon
178
192
FROM read_parquet('<value_hex>', hive_partitioning = true) c
179
-
SEMI JOIN ca USING (h8, h0)
193
+
SEMI JOIN ca USING (h8, h0)-- do NOT reference ca's columns in SELECT/WHERE
180
194
GROUP BYc.h8;
181
195
```
182
196
183
-
Restricting `h0` is legitimate only as a **partition-pruning prefilter paired with a real boundary filter** — the mask join above, or an attribute filter on the value dataset itself (see *Scoping by name or feature id* in query-optimization.md). On its own, `h0` narrows which files are scanned; it never clips to the region.
197
+
Restricting `h0` is legitimate only as a **partition-pruning prefilter paired with a real boundary filter** — the `h8 IN`/mask join above, or an attribute filter on the value dataset itself (see *Scoping by name or feature id* in query-optimization.md). On its own, `h0` narrows which files are scanned; it never clips to the region.
0 commit comments