Small-tails sweep: NULL-value statics + schema-less qualifiers — corpus 505 -> 511 (TASK-55)

backlog/tasks/task-55 - Specializer-small-tails-sweep-—-NULL-value-statics-schema-qualified-relations.md

@@ -0,0 +1,43 @@
+---
+id: TASK-55
+title: >-
+  Specializer small-tails sweep — NULL-value statics + schema-qualified
+  relations
+status: Done
+assignee: []
+created_date: '2026-07-27 22:10'
+updated_date: '2026-07-27 22:23'
+labels: []
+milestone: m-7
+dependencies:
+  - TASK-53
+priority: medium
+type: feature
+ordinal: 49000
+---
+
+## Description
+
+<!-- SECTION:DESCRIPTION:BEGIN -->
+Post-wave-B census tails that are conservative rejects rather than semantic gaps:
+
+1. NULL values in static join tables (9 cases): the original static-map design rejected any NULL in a value column. Real fitted-encoding tables have NULLs. Design: split each NULLABLE value column into (validity i1, payload) pairs at materialization — zero IR changes; the frontend's static_lane combines validity into the null lane (AND with the LEFT-miss flag). NULL KEYS keep the existing drop-the-row rule (never equi-match).
+
+2. Schema-qualified relations (5 cases): SELECT test.tbl.col FROM test.tbl / FROM s1.t1, s2.t1. The engine is schema-less; accept a single qualifier when the table part matches the registered bare name (driving, joined, and comma-joined statics — registered names may themselves be qualified), and bind 3-part column refs (schema.table.col). This AMENDS the wave-5 main.-only pin: DuckDB's schema-existence errors are unknowable to a schema-less registry; document as a contract choice in known-limitations.md + twin test updates in the same commit.
+
+Deferred with named rejects: COLUMNS(* REPLACE ...) and fn(COLUMNS(*)) expression forms, try_trim_null (a corpus-local macro), UBIGINT key payloads.
+<!-- SECTION:DESCRIPTION:END -->
+
+## Acceptance Criteria
+<!-- AC:BEGIN -->
+- [x] #1 Static tables with NULL VALUE columns serve: NULL flows through join lanes (INNER and LEFT, incl. residuals); NULL keys keep dropping; both backends
+- [x] #2 Schema-qualified driving/joined/comma-joined relations + 3-part column refs bind by suffix match; known-limitations.md §5 + twin suite updated in the same commit
+- [x] #3 Corpus replay: zero FAILs, match count reported (expect ~+12 to ~517)
+- [x] #4 Gate green both backends, clippy clean
+<!-- AC:END -->
+
+## Final Summary
+
+<!-- SECTION:FINAL_SUMMARY:BEGIN -->
+Small-tails sweep shipped: corpus 505 -> 511 of 678, zero FAILs. (1) NULL values in static join tables serve — declared-nullable value columns flatten to (validity i1, payload) pairs in the map layout with zero IR changes; the probe's typed miss-defaults make validity=false free on LEFT misses, and StaticCol ANDs validity into the null lane on both backends (INNER/LEFT/residual all oracle-checked); NULL keys keep the drop rule; the materializer maps NULL -> (false, typed default) with the non-nullable guard kept as a safety net. (2) Schema qualifiers are registry-noise: s1.t1 resolves by table-part suffix match for driving/joined/comma-joined relations, 3-part schema.table.col refs bind; amends the wave-5 main.-only rule, documented as a §5 contract choice (DuckDB's schema-existence errors are unknowable to a schema-less registry); ambiguity still errors. known-limitations.md + twin suite updated in the same commit (NULL-value reject row flipped to served). 3 of the 9 NULL-static corpus cases uncovered second blockers (dup keys / self-joins — stage-B constituency grows again). Gates: 155 Rust + 599 py green, interp identical minus backend-identity guards, clippy clean on touched files. PR #42 (stacked on #41).
+<!-- SECTION:FINAL_SUMMARY:END -->

docs/known-limitations.md

@@ -30,7 +30,7 @@ rejected, permanently by design:
 |---|---|---|
 | Regex patterns must be constants (`regexp_matches(s, pattern_col)` rejects) | `unsupported: non-constant regex pattern (compiled at prepare in v0)` | Regexes compile at prepare; DuckDB compiles per row. Per-row compilation is the opposite of specialization. |
 | Replacement strings / regex options / extract group indexes must be constants | `non-constant regexp_replace replacement` etc. | Same. |
-| Static (join) tables must be provided at build time, with unique keys and no NULL values | `duplicate map key`, `has a NULL in value column` | Joins are frozen hash maps baked into the function. Duplicate keys mean 1:N join multiplicity — a designed extension (stage B, TASK-50 notes) that is deliberately not built yet. |
+| Static (join) tables must be provided at build time, with unique keys | `duplicate map key` | Joins are frozen hash maps baked into the function. Duplicate keys mean 1:N join multiplicity — a designed extension (stage B, TASK-50 notes) that is deliberately not built yet. NULL *values* serve since TASK-55 (they flow through as NULL); NULL *keys* drop the row, matching equi-join semantics. |
 | The dynamic table cannot be joined to itself | `joining the dynamic table to itself` | The batch is the probe side; using it as a build side too needs stage-B machinery. |
 | Exactly one row table drives the query | `the specializer takes exactly one row table`, `must be the dynamic table` | The serving contract is rows-in → rows-out for one entity stream. |
 
@@ -83,6 +83,7 @@ wrong answer or require semantics we can't reproduce exactly:
 | `^` operator | It IS pow in DuckDB, but sqlparser's precedence differs from DuckDB's (`2*x^y` would parse as `(2*x)^y`). Mapping it computes the wrong tree silently. Use `pow()`. |
 | prefix `~`, `#`, `NOT GLOB` | Same class: precedence/parse divergences that would silently mis-associate. `xor()` covers bit-xor; `NOT (x GLOB p)` works. |
 | Regex reject list: `\B`, `\Q…\E`, `(?<name>…)`, duplicate group names, bounds > 1000, stacked quantifiers (`a*+`), `\u` escapes, negated Perl classes inside `[...]` | The RE2↔rust-regex differential battery (98 entries) proved these are the constructs where the engines disagree or DuckDB itself is broken (`\B` crashes DuckDB at runtime on non-ASCII). Everything else is byte-identical. |
+| Fuzzer-found regex rejects (TASK-54): `\1`–`\9` backrefs outside classes, the full stacked-quantifier grammar (`{2}*`, `?*`, `a???` — one lazy `?` is the only legal follower), nested repetition products > 1000, whitespace inside `{m, n}` bounds, class set-op lookalikes (`--`/`&&`/`~~`), non-POSIX `[` inside classes, Perl-class range endpoints (`[a-\d]`), capturing `(x){0}`, anchor-only multi-anchor patterns (DuckDB is SELF-inconsistent on these — its row path disagrees with its own constant fold) | The standing differential fuzzer (`tests/test_duckdb_regexp_fuzz.py`, in the normal gate) found these 12 classes in its first 3k-case deep run — each one a silent-wrong-answer risk in rust-regex — then re-swept to ZERO divergences over 40k cases across 8 seeds. Pins: `pins-waveB/fuzzer-task54.json`. |
 | `SIMILAR TO ... ESCAPE` | Not implemented in DuckDB itself. |
 | `* EXCLUDE (t.key)` on a USING join | DuckDB UNMERGES the coalesced column (it reappears at the right table's position) — measured, not modeled. Unqualified EXCLUDE works. |
 | `BETWEEN`/`IN` mixing non-numeric string literals with numbers | DuckDB converts at EXECUTION time (an empty input succeeds!); a bind-time conversion was measured to be over-eager. Numeric literals convert fine. |
@@ -115,6 +116,12 @@ These are served, but with a consciously chosen surface — know them:
   principle; the engine is NUL-transparent (the ASCII-kernel behavior).
 - **`%`-by-zero NaN bit pattern is platform-libm** — pinned as
   engine==oracle bit agreement per platform, not a constant.
+- **Schema qualifiers are registry-noise** (TASK-55): the engine's table
+  registry is schema-less, so `s1.t1` (and 3-part `s1.t1.col` refs)
+  resolve when the table part matches a registered bare name. DuckDB's
+  schema-existence errors (`schema "x" does not exist`) are not
+  reproduced — a schema-less registry cannot know which schemas would
+  exist. Ambiguous matches still error.
 
 ## 6. How to read a rejection
 
@@ -128,3 +135,18 @@ whose message starts with a classification:
 
 If a message you hit isn't in this document or the tests, that's a bug in
 our bookkeeping — file it.
+
+## 7. How this document stays honest
+
+Three mechanisms, all in the normal test gate:
+
+1. **The corpus replay** (678 statements mined from DuckDB's test suite):
+   every statement must match bit-for-bit, reject cleanly, or be a named
+   divergence — a wrong answer anywhere fails the gate.
+2. **The executable twin** (`tests/test_known_limitations.py`): every
+   limitation in this document is asserted; lifting one breaks a test.
+3. **The standing differential fuzzer** (`tests/test_duckdb_regexp_fuzz.py`):
+   randomized DuckDB-vs-engine sweeps of the regex surface on every run
+   (seed/size overridable for deep runs) — new divergences fail with the
+   reproducing seed and SQL, and their fix lands as a reject-list entry
+   plus a row in this document.

src/duckdb/mod.rs

@@ -98,27 +98,53 @@ fn materialize_map(
             });
         }
         let mut vals = Vec::with_capacity(val_tys.len());
-        for (name, ty) in spec.val_cols.iter().zip(val_tys) {
+        let mut vt = val_tys.iter();
+        for (name, &nullable) in spec.val_cols.iter().zip(&spec.val_nullable) {
             let v = get(name)?;
-            if v.is_none() {
-                return Err(build_err(format!(
-                    "static table '{}' has a NULL in value column '{name}' — joins to NULL \
-                     values are not supported",
-                    spec.table
-                )));
-            }
-            vals.push(match ty {
-                Ty::I1 => ScalarVal::I1(v.extract()?),
-                Ty::I64 => ScalarVal::I64(v.extract().map_err(|_| {
-                    build_err(format!(
-                        "unsupported: static table '{}' value column '{name}' value \
-                         outside BIGINT range (UBIGINT/HUGEINT payloads)",
+            let convert = |v: &pyo3::Bound<'_, PyAny>, ty: Ty| -> PyResult<ScalarVal> {
+                Ok(match ty {
+                    Ty::I1 => ScalarVal::I1(v.extract()?),
+                    Ty::I64 => ScalarVal::I64(v.extract().map_err(|_| {
+                        build_err(format!(
+                            "unsupported: static table '{}' value column '{name}' value \
+                             outside BIGINT range (UBIGINT/HUGEINT payloads)",
+                            spec.table
+                        ))
+                    })?),
+                    Ty::F64 => ScalarVal::F64(v.extract()?),
+                    Ty::Str => ScalarVal::Str(v.extract()?),
+                })
+            };
+            if nullable {
+                // (validity, payload) pair per the flattened map layout
+                // (TASK-55): NULL -> (false, typed default).
+                let _validity_ty = vt.next();
+                let ty = *vt.next().expect("payload type follows validity");
+                if v.is_none() {
+                    vals.push(ScalarVal::I1(false));
+                    vals.push(match ty {
+                        Ty::I1 => ScalarVal::I1(false),
+                        Ty::I64 => ScalarVal::I64(0),
+                        Ty::F64 => ScalarVal::F64(0.0),
+                        Ty::Str => ScalarVal::Str(String::new()),
+                    });
+                } else {
+                    vals.push(ScalarVal::I1(true));
+                    vals.push(convert(&v, ty)?);
+                }
+            } else {
+                let ty = *vt.next().expect("one type per non-nullable column");
+                if v.is_none() {
+                    // Declared non-nullable yet NULL in the data — the
+                    // original guard stays as a safety net.
+                    return Err(build_err(format!(
+                        "static table '{}' has a NULL in value column '{name}' — declared \
+                         non-nullable",
                         spec.table
-                    ))
-                })?),
-                Ty::F64 => ScalarVal::F64(v.extract()?),
-                Ty::Str => ScalarVal::Str(v.extract()?),
-            });
+                    )));
+                }
+                vals.push(convert(&v, ty)?);
+            }
         }
         entries.push((keys, vals));
     }

src/specializer/frontend.rs

@@ -249,14 +249,15 @@ fn bind_from<'a>(
     let dyn_name = match &table.relation {
         TableFactor::Table { name, alias, .. } => {
             let n = name.to_string();
-            // `main.tbl` resolves to bare `tbl` (DuckDB's default schema);
-            // every OTHER qualifier fails in DuckDB itself with a
-            // schema-does-not-exist Catalog Error, so it stays unsupported
-            // here — never a silent bare-name fallback (wave-5 pins).
-            let bare = n
-                .strip_prefix("main.")
-                .or_else(|| n.strip_prefix("MAIN."))
-                .unwrap_or(&n);
+            // The engine's registry is SCHEMA-LESS: a single schema
+            // qualifier is accepted when the table part matches the
+            // registered bare name (TASK-55, amends the wave-5 main.-only
+            // rule — DuckDB's schema-existence errors are unknowable to a
+            // schema-less registry; documented in known-limitations.md §5).
+            let bare = match n.rsplit_once('.') {
+                Some((_, t)) => t,
+                None => &n,
+            };
             if !bare.eq_ignore_ascii_case(this_name) {
                 return Err(unsup(format!(
                     "table '{n}' as the driving relation (must be the dynamic table '{this_name}')"
@@ -568,9 +569,15 @@ fn bind_from<'a>(
 }
 
 fn resolve_static(statics: &[StaticTable], raw_name: &str) -> Result<usize, PrepareError> {
+    // Schema-less registry (TASK-55): an exact registered-name match wins;
+    // otherwise a single-qualifier SQL name (`s1.t1`) matches a registered
+    // bare `t1`. Ambiguity stays an error.
+    let bare = raw_name.rsplit_once('.').map(|(_, t)| t);
     let mut table_idx = None;
     for (i, st) in statics.iter().enumerate() {
-        if st.name.eq_ignore_ascii_case(raw_name) {
+        let hit = st.name.eq_ignore_ascii_case(raw_name)
+            || bare.is_some_and(|b| st.name.eq_ignore_ascii_case(b));
+        if hit {
             if table_idx.is_some() {
                 return Err(PrepareError::Bind(format!(
                     "ambiguous static table '{raw_name}'"
@@ -1502,6 +1509,9 @@ impl Binder<'_> {
             SqlExpr::Identifier(ident) => self.column(&ident.value),
             SqlExpr::CompoundIdentifier(parts) => match parts.as_slice() {
                 [table, col] => self.qualified(&table.value, &col.value),
+                // `schema.table.col` — the schema part is registry-noise
+                // (TASK-55; structs would be a 4th meaning, not modeled).
+                [_, table, col] => self.qualified(&table.value, &col.value),
                 _ => Err(unsup("nested field access")),
             },
             SqlExpr::Nested(inner) => self.expr(inner),
@@ -2153,7 +2163,10 @@ impl Binder<'_> {
                 col: pos as u32,
             },
             ty: col.ty.ty,
-            nullable: sj.kind == JoinKind::Left,
+            // NULL-able on a LEFT miss OR when the static column itself is
+            // declared nullable (TASK-55: NULL values ride as validity+
+            // payload pairs through the probe).
+            nullable: sj.kind == JoinKind::Left || col.ty.nullable,
         }
     }