diff --git a/backlog/tasks/task-43 - Specializer-M-lower-frontend-BTA-produce-consume-lowering-for-the-v0-subset.md b/backlog/tasks/task-43 - Specializer-M-lower-frontend-BTA-produce-consume-lowering-for-the-v0-subset.md index 822baed..040a0d6 100644 --- a/backlog/tasks/task-43 - Specializer-M-lower-frontend-BTA-produce-consume-lowering-for-the-v0-subset.md +++ b/backlog/tasks/task-43 - Specializer-M-lower-frontend-BTA-produce-consume-lowering-for-the-v0-subset.md @@ -3,9 +3,10 @@ id: TASK-43 title: >- Specializer M-lower: frontend + BTA + produce/consume lowering for the v0 subset -status: To Do +status: In Progress assignee: [] created_date: '2026-07-25 02:31' +updated_date: '2026-07-26 05:14' labels: [] milestone: m-7 dependencies: @@ -25,9 +26,38 @@ The load-bearing milestone (design doc §5): sqlparser(DuckDB dialect) frontend ## Acceptance Criteria -- [ ] #1 v0-subset queries prepare end-to-end: sql + static tables -> verified imperative IR running on the interpreter backend -- [ ] #2 All-static subtrees are evaluated at prepare time: a static-tables-only query lowers to a constant emitter with no probe/filter ops in its IR -- [ ] #3 Differential suite vs DuckDB green on hand-written v0 cases; engine-vs-oracle disagreement follows the xfail-strict + ticket protocol -- [ ] #4 Corpus replay reports match / clean-unsupported / FAIL counts; zero FAILs; every unsupported rejection is a clean build-time error naming the construct -- [ ] #5 mise gate-specializer green (corpus replay wired into it) +- [x] #1 v0-subset queries prepare end-to-end: sql + static tables -> verified imperative IR running on the interpreter backend +- [x] #2 All-static subtrees are evaluated at prepare time: a static-tables-only query lowers to a constant emitter with no probe/filter ops in its IR +- [x] #3 Differential suite vs DuckDB green on hand-written v0 cases; engine-vs-oracle disagreement follows the xfail-strict + ticket protocol +- [x] #4 Corpus replay reports match / clean-unsupported / FAIL counts; zero FAILs; every unsupported rejection is a clean build-time error naming the construct +- [x] #5 mise gate-specializer green (corpus replay wired into it) + +## Implementation Plan + + +Stretch plan (~5-6 stretches estimated, recorded 2026-07-26): +1. Frontend spine: sqlparser (DuckDB dialect) -> relational IR (scan/project/filter over __THIS__), reusing shape lessons from src/datafusion; end-to-end sql -> imperative IR -> interpreter for arithmetic projections. +2. 3VL lowering: SQL NULL semantics compiled to flag algebra (comparisons, AND/OR Kleene logic, CASE, IS NULL); type + nullability derivation; the correctness core. +3. BTA + statics: taint __THIS__, equi-joins to static tables lower to probes, scalar folding; Python materialization through the DuckDBInferFn shell (pa.Table -> StaticData). +4. IR builtin extensions (workflow fan-out per op): upper/lower/trim/substr/abs/round/concat + :: casts + COALESCE/NULLIF as lowerings; each new instruction lands across verifier/printer/parser/interp with pin tests. +5.-6. Differential suite vs duckdb-python (pytest backend id "specialized") + corpus replay under the three-outcome contract; grind divergences to zero FAILs; xfail-strict + ticket for genuine oracle disagreements. + + +## Implementation Notes + + +Stretch 3 landed (commit 49362ee on claude/specializer-m-lower, PR #26): INNER/LEFT equi-joins to static tables lower to prologue probes (ordered JoinSpec list, no join tree nodes in v0); valid_hit = map hit AND every nullable key flag; f64 probe keys canonicalize -0.0/NaN at compile AND probe (DuckDB DOUBLE `=` pins: NaN=NaN, 0.0=-0.0); join key promotion both directions (i64 dyn promoted via IntToFloat; f64-vs-i64-col converts the build side at materialization per StaticSpec). fold.rs: conservative const folder, both-sides-const only, never drops a potentially-trapping dynamic operand (`FALSE AND a%0` keeps the rem). DuckDBInferFn replaces the stub: one row table (pydantic, ordered fields), pyarrow statics -> StaticData::Map per StaticSpec (NULL-key rows dropped, NULL value col errors, dup keys rejected at compile), output model synthesized via create_model. Differential pytest vs duckdb-python: 13 cases (joins, promotion, error contracts). Deliberate ceilings: per-block probe re-emission (pure, cached per block); all non-key cols become map values (prune later); key cols referenced outside their ON clause -> clean unsupported; supplied output_model trusted as-is. + +AC #2 (static-only queries fold to a constant emitter) is NOT yet satisfied: needs DuckDB evaluation at prepare time (Python side) and a constant-emitter program shape (emits N fixed rows regardless of input) — parked for the stretch 5-6 grind. Stretch 4 started: builtin catalogue; `::` casts already landed in stretch 2 (CastKind::DoubleColon). Fan-out adaptation: per-op implement-in-worktree would conflict on the same 7 coupled files (ir/mod, verify, print, parse, interp, frontend, fold), so the workflow fans out DuckDB semantics PINNING (8 parallel agents, duckdb-python 1.5.5) and post-integration adversarial verify; implementation is integrated centrally (one integrator, many measurers). + +Stretch 4 landed (commit ac3786a): builtin catalogue implemented strictly from measured pins (8-agent workflow fan-out measured DuckDB 1.5.5; spec at docs/superpowers/specs/2026-07-26-stretch4-builtin-pins.md). New IR insts across all six files + fuzz gen: supper/slower, strim.{both,lead,trail}, ssubstr (virtual-window codepoint arithmetic), iabs/fabs/fround, BinOp::Frem. Frontend catalogue: upper/lower/ltrim/rtrim/abs/round(1-arg)/concat/coalesce/nullif + TRIM/SUBSTRING dedicated AST forms; || is ALWAYS concat (even 1 || 2) with implicit VARCHAR casts; CONCAT skips NULLs (all-NULL -> ''); COALESCE per-row lazy via CASE desugار; NULLIF compares at promoted type, keeps first arg's type. TWO DIVERGENCES IN PREVIOUSLY-LANDED CODE were found by pinning and FIXED in-branch with pin tests (PM: flagging per the disagreement protocol — these were fixes toward the oracle, not semantics patches to pass tests): (1) integer % by zero returns NULL in DuckDB, we trapped -> lowering now CASE-guards the divisor, MIN % -1 still traps like DuckDB; (2) DOUBLE comparisons: DuckDB order is NaN=NaN / NaN above everything / zeros equal, we had IEEE partial -> shared exec::duck_fcmp used by interp AND fold. Known deliberate divergence (strict-xfail + needs a ticket): Rust std lacks Unicode SIMPLE case maps, so upper('ß') gives 'ß' vs DuckDB 'ẞ' and lower('İ') differs; ASCII exact. Deliberate ceilings: round(x, digits) unsupported; decimal literals stay f64 (stretch-1 ceiling). 114 cargo tests + 21 differential pytest + 1 strict xfail; gate green. Adversarial verify fan-out (6 probe agents vs duckdb) running; findings will be triaged fix-vs-xfail before stretch 5 (differential suite backend id + corpus replay in gate). + +Stretch 6 landed (commit 0782bf1): AC #2 satisfied — static-only queries become constant emitters, evaluated once at build time by DuckDB itself (Python boundary; no IR is built, so trivially no probe/filter ops). The fallback fires on Unsupported/Parse prepare errors and self-validates: dynamic queries reference the row table, unknown to DuckDB, so evaluation fails and the original clean error surfaces. Bonus: aggregation/ORDER BY/dialect-beyond-sqlparser all work on the static-only path. Final corpus: 53 match / 625 clean-unsupported / 0 FAIL of 678, wired into the gate via pytest. MILESTONE COMPLETE pending review — hard stop before M-cranelift (TASK-44). Two items for PM: (1) deviation note — AC #3's 'backend id specialized' wording: the DataFusion-oracle differential harness (test_diff_*) can't host the specializer (different oracle, e.g. `/` semantics differ by design), so the specializer has its own duck_check differential suite (32 cases) instead; (2) ticket request per the disagreement protocol — Unicode SIMPLE case mapping (upper('ß')/'İ'/'ᾀ' class): Rust std only exposes full maps; strict-xfail in place; candidate fixes are a small static table of the ~100 divergent codepoints or a unicode-data crate dependency. + + +## Final Summary + + +M-lower delivered on branch claude/specializer-m-lower (PR #26), six stretches: (1) frontend spine sql->IR->interpreter; (2) 3VL flag algebra + CASE/CAST/IS NULL; (3) BTA + statics — equi-joins lower to map probes with canonical f64 key bits, DuckDBInferFn Python boundary, 13-case differential suite; (4) builtin catalogue measured by an 8-agent pin fan-out (upper/lower/trim/substr/abs/round/concat/coalesce/nullif, ||, float %, DuckDB DOUBLE comparison order) — 7 new IR instructions across verify/print/parse/gen/interp; (5) adversarial 6-agent probe fleet (~1,400 probes) found and fixed: NULL-divisor % trap, the trap-under-false-flag payload class bug, Zs trim set, vectorized-path substr semantics (backwards negative lengths, ±2^32 guards), DuckDB float->VARCHAR rendering; plus corpus replay wired into the gate under the three-outcome contract; (6) static-only constant emitters (AC #2). Final: gate green (cargo + 603 pytest, 13 xfail), corpus 53 match / 625 clean-unsupported / 0 FAIL of 678. Known strict-xfail: Unicode simple case maps (ticket requested). DuckDB dual-path substr inconsistency documented; oracle arrow-pushdown artifact harness-fixed. + diff --git a/docs/superpowers/specs/2026-07-26-stretch4-builtin-pins.md b/docs/superpowers/specs/2026-07-26-stretch4-builtin-pins.md new file mode 100644 index 0000000..5fe6a5a --- /dev/null +++ b/docs/superpowers/specs/2026-07-26-stretch4-builtin-pins.md @@ -0,0 +1,146 @@ +# Stretch 4: builtin catalogue — measured DuckDB 1.5.5 pins + implementation spec + +All values below were MEASURED against duckdb-python 1.5.5 (workflow fan-out of 8 +pin agents + local micro-pins, 2026-07-26). DuckDB is the oracle; when in doubt, +re-measure, never assume. + +## Divergences found in already-landed code (fix in this stretch, with pin tests) + +1. **`%` by zero**: `5 % 0` → NULL (NOT an error). `MIN % -1` → traps + ("Out of Range Error: Overflow in division"). Our irem traps on both; keep the + IR inst trapping (MIN % -1 stays a trap ✓) and guard the SQL lowering: + `a % b` → `CASE WHEN b = 0 THEN NULL ELSE a irem b END` (skip the guard when b + is a non-zero literal). Result becomes nullable when guarded. +2. **Float `%`**: currently rejected at bind. DuckDB: `5.0 % 2.0` = 1.0, + `-5.5 % 2.5` = -0.5 (sign of dividend), `5.0 % 0.0` = NaN — exactly Rust's + `%` on f64. Add `BinOp::Frem` (never traps), un-reject at bind. No guard. +3. **F64 comparison order** (interp.rs Cmp F64 + fold.rs cmp mirror are IEEE — + wrong): DuckDB order = IEEE except NaN: `nan = nan` TRUE, `nan > 1` TRUE, + `nan > inf` TRUE, `nan <= nan` TRUE, `nan <> nan` FALSE, `1 <= nan` TRUE; + `-0.0 = 0.0` TRUE, `-0.0 < 0.0` FALSE. Implementation: + both NaN → Equal; one NaN → NaN is Greater; else IEEE partial_cmp. + +## New IR instructions (each lands in ir/mod + verify + print + parse + gen + interp) + +- `supper` / `slower` (Str→Str): DuckDB uses SIMPLE (1:1) case mapping: + `upper('ß')` = 'ẞ' U+1E9E (NOT 'SS'), `lower('İ' U+0130)` = plain 'i' (dot + dropped), emoji pass through. Rust std is FULL mapping → v0: per char, use + to_uppercase()/to_lowercase() iff it yields exactly 1 char, else keep the char + unchanged. KNOWN divergence on ß (we keep 'ß', DuckDB 'ẞ') and İ→lower (we keep + 'İ', DuckDB 'i'): xfail-strict differential case + ticket note. ASCII exact. +- `strim.both|lead|trail s, chars` (Str,Str→Str): removes chars in the SET + `chars` from the side(s). 1-arg SQL trim = chars " " (ONLY space 0x20 — tab/ + newline NOT trimmed). Empty set = no-op. NULL either arg → NULL (flag algebra + at lowering, inst itself total). +- `ssubstr s, start, len` (Str,I64,I64→Str): codepoint-based (NOT grapheme — + slices inside ZWJ emoji). Algorithm (1-based virtual window): + if len < 0 → ""; if start < 0 → start = char_len + start + 1; + window [start, start.saturating_add(len)); intersect with [1, char_len]; + missing SQL len → Lit(i64::MAX) (saturating add makes it "rest of string"). + Pins: substr('hello',0)='hello', (0,3)='he', (-2)='lo', (-6,3)='he', + (-10,8)='hel', (1,0)='', (1,-1)='', (10)=''. +- `iabs` (I64→I64): traps on i64::MIN ("Out of Range"-style). `fabs` (F64→F64): + Rust f64::abs — clears sign bit, abs(-0.0)=+0.0, abs(nan)=nan, abs(-inf)=inf. +- `fround` (F64→F64): Rust f64::round = half AWAY from zero ✓ (2.5→3, -2.5→-3), + nan→nan, inf→inf, 1e300→1e300, -0.4→-0.0 (sign KEPT on double). round(int) is + identity at the FRONTEND (no inst); round(x, digits) → clean unsupported + (scale-then-round algorithm deferred). +- `frem` BinOp (F64,F64→F64): Rust `%`. Never traps. + +## Frontend dispatch (new SKind variants + lowerings) + +- `Expr::Trim` (sqlparser: trim_where BOTH/LEADING/TRAILING, trim_what, + trim_characters) + `ltrim`/`rtrim`/2-arg functions → SKind::Trim{side}. + All forms NULL-propagating on both args. +- `Expr::Substring` (SUBSTR and SUBSTRING both route here) → SKind::Substr. + NULL-propagating on all three args. +- `Expr::Function` name dispatch (case-insensitive): + - upper/lower → SKind::StrCase{upper} — arg must be Str (DuckDB: upper(123) + is a Binder Error, NO numeric coercion). + - abs → SKind::Abs; I64 or F64 only (abs('5')/abs(true) = binder errors). + Result type = arg type. + - round (1-arg) → identity for I64; SKind::Round for F64. 2-arg → unsupported. + - concat(a,...) → NULL-SKIPPING: each nullable arg wraps in + CASE WHEN x IS NULL THEN '' ELSE cast_to_varchar(x) END, chain SKind::Concat. + Result never NULL. concat() zero args → Bind error. Non-string args render + via VARCHAR cast (1→'1', 1.5→'1.5', true→'true'). + - coalesce(a,...) → nested CASE WHEN a IS NOT NULL THEN a ELSE rest END. + Lazy per-row (pinned: untaken erroring arm does NOT fire). Args unify under + existing promotion (I64+F64→F64); else Bind error. coalesce() → parser + error in DuckDB; here Bind error fine. + - nullif(a,b) → CASE WHEN a = b THEN NULL ELSE a END; comparison at PROMOTED + type, result type = FIRST arg's type (never unified!). nullif(nan,nan) → + NULL (needs divergence fix #3). +- `BinaryOperator::StringConcat` (`||`): ALWAYS string concat in DuckDB — even + `1 || 2` = '12', `true || true` = 'truetrue'. NULL-PROPAGATING (unlike + CONCAT). Cast non-Str operands to VARCHAR: bool → 'true'/'false' (measured). +- CONCAT vs ||: CONCAT skips NULLs (all-NULL → ''), || propagates. + +## fold.rs + +New kinds: fold children only (like Case). Update the f64 cmp mirror to the +DuckDB order (divergence #3) — fold and interp MUST stay bit-identical. + +## Out of scope (clean unsupported), deliberate + +round(x, digits); DECIMAL anything (literals stay F64 — known v0 ceiling); +upper/lower non-simple-map codepoints byte-exactness (xfail + ticket). + +## Adversarial-fleet addendum (6 probe agents, ~1,400 probes, 2026-07-26) + +Divergences FOUND and FIXED (each now pinned in Rust + differentially): + +1. **NULL divisor `%`**: the `b = 0` CASE guard alone is NULL for b NULL — + fell through to irem on the garbage zero payload. `b IS NULL OR b = 0` + shields it (TRUE OR NULL = TRUE). +2. **Trap-under-false-flag class bug** (predates stretch 4): computed + garbage payloads are unbounded (`(x + MAX) + MAX` with x NULL overflows + its payload lane). FIX: `FB::masked` forces nullable payloads to the type + default before every trapping instruction (integer arith, iabs, ssubstr + positions, ftoi cast input). +3. **1-arg trim set**: exactly the Unicode Zs space separators (per-codepoint + census) — NBSP/ideographic space etc. trim; tab/newline/ZWSP/BOM do not. +4. **substr**: DuckDB's constant-fold path and vectorized path DISAGREE on + negative starts. We implement the VECTORIZED path (what columns, real + queries, and the mined corpus use): negative start clamps to 1 after + end-resolution (`rs = max(n+start+1, 1)`), start 0 stays virtual, a + NEGATIVE length slices BACKWARDS `[rs+len, rs)`, offsets/lengths outside + ±2^32 trap ("Out of Range"), the 2-arg form never length-traps (why + `len` is `Option`, not a sentinel). Known residual: pure-literal + negative-start substr goes through DuckDB's constant path and can differ. +5. **Float -> VARCHAR**: DuckDB writes an explicit exponent sign with at + least two digits (`1e+300`, `1e-05`) and lowercase `nan` (DuckF64 in + interp.rs). +6. **Oracle artifact, harness-fixed, no engine change**: duckdb-python + pushes constant filters into REGISTERED-ARROW scans with IEEE NaN + semantics, disagreeing with its own native-table order (and violating + 3VL for `x <= NaN`). duck_check now materializes native tables. +7. **Simple-case-map divergence extended**: `upper('ᾀ')` (ypogegrammeni + titlecase U+1F88) joins ß/İ in the strict xfail. + +Also landed with this pass (corpus-driven): implicit numeric->BOOLEAN in +conditional contexts (WHERE/AND/OR/NOT/CASE WHEN; nonzero -> true incl. +NaN, NULL -> NULL); `rowid` and DuckDB lateral aliases reject as clean +unsupported; corpus replay wired into pytest with the three-outcome +contract at 49 match / 629 clean-unsupported / 0 FAIL of 678. + +## Case-mapping divergence: RESOLVED dependency-free (2026-07-26, post-review) + +Decision (AmirHossein): no dependency for the ~100 codepoints; do it measured +and documented. `src/specializer/exec/casemap.rs` (GENERATED by +`scripts/gen_casemap.py`) carries the exception table over Rust's full maps — +139 entries (83 upper, 56 lower) from TWO measured classes: + 1. simple-vs-full mapping divergence (ß/İ/ypogegrammeni block); + 2. Unicode VERSION skew, both directions: codepoints utf8proc maps that + Rust doesn't matter (fallback handles), and Unicode-16 case pairs RUST + maps but duckdb 1.5.5's utf8proc predates (identity there — 56 entries + Python couldn't even see; the generator's phase 2 measures the actual + compiled engine to catch them). +A unicode-data crate would be the wrong spec twice over: some other Unicode +version's tables, and blind to utf8proc's vintage. The full-codepoint census +in tests/test_duckdb_interpreter.py (every scalar value through both engines, +chunked into long strings) is the standing authority — a duckdb bump that +shifts utf8proc fails the census, and the generator is idempotent to rerun. +The former strict xfail is now two passing tests. DataFusion note: per the +same review, DataFusion is the legacy serving line's oracle only — likely +retired later, possibly a supported dialect; the specializer ignores it. diff --git a/scripts/gen_casemap.py b/scripts/gen_casemap.py new file mode 100644 index 0000000..229a5f5 --- /dev/null +++ b/scripts/gen_casemap.py @@ -0,0 +1,207 @@ +"""Generate src/specializer/exec/casemap.rs by measuring DuckDB. + +Why this file exists +-------------------- +DuckDB's upper()/lower() use utf8proc's SIMPLE case mapping: every codepoint +maps to exactly one codepoint (UnicodeData.txt's Simple_*case_Mapping). +Rust std only exposes the FULL mapping (SpecialCasing.txt), where a codepoint +may expand ('ss' -> "SS"). The engine's fallback rule — take Rust's full map +iff it is 1:1, else keep the codepoint — agrees with DuckDB almost everywhere, +and this script measures the exact exceptions and freezes them into a table. + +The exceptions come from two sources, both captured by measurement: + 1. Simple-vs-full divergence: codepoints whose full map is multi-char but + which have a DIFFERENT 1:1 simple map (the sharp s, Turkish dotted I, + the Greek ypogegrammeni block U+1F80..U+1FFC). + 2. Unicode VERSION skew: case pairs added to Unicode after the utf8proc + release inside this duckdb build (e.g. U+019B/U+A7DC from Unicode 16) — + Rust maps them, DuckDB does not. DuckDB is the oracle, so its identity + behavior wins. + +Why no dependency +----------------- +A unicode-data crate would give us *some* Unicode version's simple maps — +which is the wrong spec twice over: it ignores class 2 entirely (the crate's +tables won't match utf8proc's vintage), and it drags a dependency in for +what measurement shows is ~83 codepoints. The measured table is smaller, +exactly right by construction, and verified end-to-end by the full-codepoint +census in tests/test_duckdb_interpreter.py (which will fail loudly if this +table ever drifts from the duckdb the suite runs against). + +Regenerate with `uv run python scripts/gen_casemap.py` after bumping the +duckdb dependency, then run the gate: the census test is the authority. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import duckdb +import pyarrow as pa + +REPO = Path(__file__).resolve().parent.parent +OUT = REPO / "src" / "specializer" / "exec" / "casemap.rs" + + +def rust_rule(c: str, upper: bool) -> str: + """Phase-1 approximation of the engine's dependency-free fallback (full + map iff 1:1, else keep). Python's tables APPROXIMATE Rust's: all three of + Python, Rust, and duckdb's utf8proc can ship different Unicode versions, + so phase 2 measures the actual compiled engine and patches the residue + (e.g. Unicode-16 case pairs Rust maps but Python does not).""" + m = c.upper() if upper else c.lower() + return m if len(m) == 1 else c + + +def engine_residue(chunks: list[str], duck_u: list[str], duck_l: list[str]): + """Phase 2: run the census through the INSTALLED engine and report every + codepoint still diverging from duckdb — Rust-vs-Python table skew the + phase-1 approximation cannot see. Requires the built sql_transform wheel; + skipped (with a warning) when it is not importable.""" + try: + from pydantic import create_model + + from sql_transform._interpreter import DuckDBInferFn + except ImportError as e: # pragma: no cover -- generator convenience + print( + f"WARNING: engine not importable ({e}); phase 2 skipped — " + "rebuild and rerun, the census test is the authority" + ) + return [], [] + model = create_model("Row", s=(str, ...)) + fn = DuckDBInferFn( + "SELECT upper(s) AS u, lower(s) AS l FROM __THIS__", + row_tables={"__THIS__": model}, + static_tables={}, + ) + ru, rl = [], [] + for s, du, dl in zip(chunks, duck_u, duck_l, strict=True): + r = fn.infer({"__THIS__": [model(s=s)]})[0] + ru.extend((ord(c), b) for c, a, b in zip(s, r.u, du, strict=True) if a != b) + rl.extend((ord(c), b) for c, a, b in zip(s, r.l, dl, strict=True) if a != b) + return ru, rl + + +def main() -> int: + con = duckdb.connect() + cps = [cp for cp in range(1, 0x110000) if not (0xD800 <= cp <= 0xDFFF)] + con.register("cps_arrow", pa.table({"cp": cps, "s": [chr(c) for c in cps]})) + con.execute("CREATE TABLE cps AS SELECT * FROM cps_arrow") + rows = con.execute( + "SELECT cp, s, upper(s), lower(s) FROM cps ORDER BY cp" + ).fetchall() + + upper_ex, lower_ex = {}, {} + for cp, s, u, lo_ in rows: + assert len(u) == 1 and len(lo_) == 1, f"non-1:1 duckdb map at U+{cp:04X}" + if rust_rule(s, True) != u: + upper_ex[cp] = u + if rust_rule(s, False) != lo_: + lower_ex[cp] = lo_ + + # Carry forward every codepoint already in casemap.rs, re-measured + # against duckdb. Phase 2 measures the INSTALLED engine, which already + # contains the current table — so skew entries it fixed report no + # residue and would be silently dropped on regeneration without this. + # An entry whose value equals what the fallback gives is harmless. + if OUT.exists(): + import re + + by_cp = {cp: (u, lo_) for cp, _, u, lo_ in rows} + text = OUT.read_text(encoding="utf-8") + entry = r"\('\\u\{([0-9A-F]+)\}', '\\u\{[0-9A-F]+\}'\)" + # Split at the const DEFINITION (the lookup fns mention the names too). + upper_part, _, lower_part = text.partition("const LOWER_EXCEPTIONS") + for part, ex, idx in ((upper_part, upper_ex, 0), (lower_part, lower_ex, 1)): + for m in re.finditer(entry, part): + cp = int(m.group(1), 16) + if cp in by_cp: + ex.setdefault(cp, by_cp[cp][idx]) + + # Phase 2: census the compiled engine in chunks, patch Rust-side skew. + step = 0x8000 + chunks, duck_u, duck_l = [], [], [] + for lo in range(1, 0x110000, step): + s = "".join( + chr(c) + for c in range(lo, min(lo + step, 0x110000)) + if not (0xD800 <= c <= 0xDFFF) + ) + if s: + chunks.append(s) + u, lo_ = con.execute("SELECT upper(?), lower(?)", [s, s]).fetchone() + duck_u.append(u) + duck_l.append(lo_) + ru, rl = engine_residue(chunks, duck_u, duck_l) + n_skew = len([cp for cp, _ in ru if cp not in upper_ex]) + len( + [cp for cp, _ in rl if cp not in lower_ex] + ) + upper_ex.update(ru) + lower_ex.update(rl) + print(f"phase 2: {n_skew} Rust-vs-Python skew entries patched in") + upper_ex = sorted(upper_ex.items()) + lower_ex = sorted(lower_ex.items()) + + def table(name: str, entries: list[tuple[int, str]]) -> str: + body = "\n".join( + f" ('\\u{{{cp:04X}}}', '\\u{{{ord(m):04X}}}'), // {chr(cp)!r} -> {m!r}" + for cp, m in entries + ) + return f"pub(super) const {name}: &[(char, char)] = &[\n{body}\n];\n" + + duck_version = duckdb.__version__ + OUT.write_text( + f"""//! DuckDB's SIMPLE case mapping, as measured — see scripts/gen_casemap.py +//! for the full story (simple-vs-full mapping, Unicode version skew, and why +//! a dependency would be the wrong spec twice over). +//! +//! GENERATED by `uv run python scripts/gen_casemap.py` against duckdb-python +//! {duck_version}; regenerate after a duckdb bump. The full-codepoint census in +//! tests/test_duckdb_interpreter.py fails loudly if this table drifts from +//! the duckdb the suite runs against — the census, not this file, is the +//! authority. +//! +//! Entries are (codepoint, mapped) pairs, sorted by codepoint, covering only +//! the codepoints where the dependency-free fallback (Rust's full map iff it +//! is 1:1, else identity) disagrees with DuckDB. Everything else goes through +//! the fallback in `simple_upper`/`simple_lower`. + +/// Simple uppercase for one codepoint, exactly as DuckDB computes it. +pub fn simple_upper(c: char) -> char {{ + if let Ok(i) = UPPER_EXCEPTIONS.binary_search_by_key(&c, |e| e.0) {{ + return UPPER_EXCEPTIONS[i].1; + }} + let mut it = c.to_uppercase(); + let first = it.next().expect("case iterators are non-empty"); + if it.next().is_none() {{ + first + }} else {{ + c + }} +}} + +/// Simple lowercase for one codepoint, exactly as DuckDB computes it. +pub fn simple_lower(c: char) -> char {{ + if let Ok(i) = LOWER_EXCEPTIONS.binary_search_by_key(&c, |e| e.0) {{ + return LOWER_EXCEPTIONS[i].1; + }} + let mut it = c.to_lowercase(); + let first = it.next().expect("case iterators are non-empty"); + if it.next().is_none() {{ + first + }} else {{ + c + }} +}} + +{table("UPPER_EXCEPTIONS", upper_ex)} +{table("LOWER_EXCEPTIONS", lower_ex)}""", + encoding="utf-8", + ) + print(f"wrote {OUT}: {len(upper_ex)} upper + {len(lower_ex)} lower exceptions") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/datafusion/expr.rs b/src/datafusion/expr.rs index b5efef6..00aa62e 100644 --- a/src/datafusion/expr.rs +++ b/src/datafusion/expr.rs @@ -9,7 +9,6 @@ use crate::datafusion::types::FieldType; // keep resolving now that the value representation is shared. pub use crate::value::{type_name, Value}; - /// String form used by CONCAT and CAST(.. AS VARCHAR). pub fn display_value(v: &Value) -> String { match v { @@ -50,7 +49,6 @@ pub fn display_value(v: &Value) -> String { } } - #[derive(Clone)] pub enum Expr { Column { @@ -124,7 +122,10 @@ pub enum BinOp { Concat, } -pub fn eval(expr: &Expr, row: &crate::datafusion::plan::Row) -> Result { +pub fn eval( + expr: &Expr, + row: &crate::datafusion::plan::Row, +) -> Result { match expr { Expr::Column { table, name } => resolve_column(row, table.as_deref(), name), Expr::Literal(v) => Ok(v.clone()), @@ -174,7 +175,9 @@ pub fn eval(expr: &Expr, row: &crate::datafusion::plan::Row) -> Result Err(crate::datafusion::plan::InterpError::Eval(format!( "Cannot access field '{field}' on a {} value", @@ -200,75 +203,79 @@ pub fn eval(expr: &Expr, row: &crate::datafusion::plan::Row) -> Result Result { - // Reorder the struct's fields to feature_names_in_ order, then - // build a 1-row Python list-of-lists. sklearn's check_array - // coerces it -- no numpy on the Rust side. - let mut ordered: Vec> = Vec::with_capacity(input_features.len()); - for feat in input_features { - let value = fields - .iter() - .find(|(n, _)| n == feat) - .map(|(_, v)| v) - .ok_or_else(|| { + Python::attach( + |py| -> Result { + // Reorder the struct's fields to feature_names_in_ order, then + // build a 1-row Python list-of-lists. sklearn's check_array + // coerces it -- no numpy on the Rust side. + let mut ordered: Vec> = Vec::with_capacity(input_features.len()); + for feat in input_features { + let value = fields + .iter() + .find(|(n, _)| n == feat) + .map(|(_, v)| v) + .ok_or_else(|| { + crate::datafusion::plan::InterpError::Eval(format!( + "transformer input struct is missing field '{feat}'" + )) + })?; + let py_val = value.to_pyobject(py).map_err(|e| { crate::datafusion::plan::InterpError::Eval(format!( - "transformer input struct is missing field '{feat}'" + "marshalling transformer input failed: {e}" )) })?; - let py_val = value.to_pyobject(py).map_err(|e| { + ordered.push(py_val); + } + let row_list = PyList::new(py, ordered).map_err(|e| { crate::datafusion::plan::InterpError::Eval(format!( - "marshalling transformer input failed: {e}" + "building transformer input row failed: {e}" )) })?; - ordered.push(py_val); - } - let row_list = PyList::new(py, ordered).map_err(|e| { - crate::datafusion::plan::InterpError::Eval(format!( - "building transformer input row failed: {e}" - )) - })?; - let x = PyList::new(py, [row_list]).map_err(|e| { - crate::datafusion::plan::InterpError::Eval(format!( - "building transformer input matrix failed: {e}" - )) - })?; - let y = obj.bind(py).call_method1("transform", (x,)).map_err(|e| { - crate::datafusion::plan::InterpError::Eval(format!("transformer.transform failed: {e}")) - })?; - // .tolist() turns numpy scalars into Python builtins so - // from_pyobject sees float/int/str, not opaque numpy objects. - let y_list = y.call_method0("tolist").map_err(|e| { - crate::datafusion::plan::InterpError::Eval(format!( - "transformer output .tolist() failed: {e}" - )) - })?; - let y0 = y_list.get_item(0).map_err(|e| { - crate::datafusion::plan::InterpError::Eval(format!( - "transformer produced no output row: {e}" - )) - })?; - // Marshal each output position through the declared field - // type. Parity invariant: the declared output dtype must equal - // the transform's natural dtype -- here it drives pydantic - // coercion at model-validate time, and the DataFusion oracle - // reaches the same type via a pyarrow cast; the two agree only - // when no real coercion is needed (see _transformer_udf.py). - let mut out = Vec::with_capacity(output_fields.len()); - for (i, (fname, ft)) in output_fields.iter().enumerate() { - let elem = y0.get_item(i).map_err(|e| { + let x = PyList::new(py, [row_list]).map_err(|e| { crate::datafusion::plan::InterpError::Eval(format!( - "transformer output missing position {i} for field '{fname}': {e}" + "building transformer input matrix failed: {e}" )) })?; - let val = Value::from_pyobject_typed(&elem, &ft.base).map_err(|e| { + let y = obj.bind(py).call_method1("transform", (x,)).map_err(|e| { crate::datafusion::plan::InterpError::Eval(format!( - "marshalling transformer output field '{fname}' failed: {e}" + "transformer.transform failed: {e}" )) })?; - out.push((fname.clone(), val)); - } - Ok(Value::Struct(out)) - }) + // .tolist() turns numpy scalars into Python builtins so + // from_pyobject sees float/int/str, not opaque numpy objects. + let y_list = y.call_method0("tolist").map_err(|e| { + crate::datafusion::plan::InterpError::Eval(format!( + "transformer output .tolist() failed: {e}" + )) + })?; + let y0 = y_list.get_item(0).map_err(|e| { + crate::datafusion::plan::InterpError::Eval(format!( + "transformer produced no output row: {e}" + )) + })?; + // Marshal each output position through the declared field + // type. Parity invariant: the declared output dtype must equal + // the transform's natural dtype -- here it drives pydantic + // coercion at model-validate time, and the DataFusion oracle + // reaches the same type via a pyarrow cast; the two agree only + // when no real coercion is needed (see _transformer_udf.py). + let mut out = Vec::with_capacity(output_fields.len()); + for (i, (fname, ft)) in output_fields.iter().enumerate() { + let elem = y0.get_item(i).map_err(|e| { + crate::datafusion::plan::InterpError::Eval(format!( + "transformer output missing position {i} for field '{fname}': {e}" + )) + })?; + let val = Value::from_pyobject_typed(&elem, &ft.base).map_err(|e| { + crate::datafusion::plan::InterpError::Eval(format!( + "marshalling transformer output field '{fname}' failed: {e}" + )) + })?; + out.push((fname.clone(), val)); + } + Ok(Value::Struct(out)) + }, + ) } Expr::Case { arms, default } => { // Short-circuit: evaluate conditions left to right, stop at the first @@ -318,7 +325,11 @@ fn resolve_column( .ok_or_else(|| InterpError::Build(format!("Unknown column: {name}"))) } -fn eval_binary_op(op: BinOp, l: Value, r: Value) -> Result { +fn eval_binary_op( + op: BinOp, + l: Value, + r: Value, +) -> Result { match op { BinOp::Add | BinOp::Sub | BinOp::Mul | BinOp::Div | BinOp::Mod => arithmetic(op, l, r), BinOp::Eq | BinOp::NotEq | BinOp::Lt | BinOp::Gt | BinOp::LtEq | BinOp::GtEq => { @@ -342,7 +353,11 @@ fn concat_op(l: Value, r: Value) -> Result Result { +fn arithmetic( + op: BinOp, + l: Value, + r: Value, +) -> Result { if matches!(l, Value::Null) || matches!(r, Value::Null) { return Ok(Value::Null); } @@ -388,7 +403,11 @@ fn as_f64(v: &Value) -> Result { } } -fn comparison(op: BinOp, l: Value, r: Value) -> Result { +fn comparison( + op: BinOp, + l: Value, + r: Value, +) -> Result { if matches!(l, Value::Null) || matches!(r, Value::Null) { return Ok(Value::Null); } @@ -420,7 +439,10 @@ fn comparison(op: BinOp, l: Value, r: Value) -> Result Result { +fn compare_values( + l: &Value, + r: &Value, +) -> Result { match (l, r) { (Value::Int(a), Value::Int(b)) => Ok(a.cmp(b)), (Value::Str(a), Value::Str(b)) => Ok(a.cmp(b)), @@ -472,7 +494,10 @@ fn as_tribool(v: &Value) -> Result, crate::datafusion::plan::Interp } } -fn eval_builtin(name: &str, args: Vec) -> Result { +fn eval_builtin( + name: &str, + args: Vec, +) -> Result { use crate::datafusion::plan::InterpError; if matches!(name, "upper" | "lower" | "trim" | "substr" | "substring") diff --git a/src/datafusion/mod.rs b/src/datafusion/mod.rs index e753a1a..8f1e497 100644 --- a/src/datafusion/mod.rs +++ b/src/datafusion/mod.rs @@ -105,7 +105,10 @@ struct ResolvedTransformer { /// Reads `obj.feature_names_in_.tolist()`. Absence is a clear build error: /// the object was fit on bare arrays, so we cannot align inputs by name. -fn read_feature_names_in(py: Python<'_>, obj: &Py) -> Result, plan::InterpError> { +fn read_feature_names_in( + py: Python<'_>, + obj: &Py, +) -> Result, plan::InterpError> { let bound = obj.bind(py); let attr = bound.getattr("feature_names_in_").map_err(|_| { plan::InterpError::Build( @@ -148,7 +151,10 @@ fn resolve_transformers( arg: Box::new(arg), }); } - Ok(Expr::Function { name, args: new_args }) + Ok(Expr::Function { + name, + args: new_args, + }) } Expr::BinaryOp { op, left, right } => Ok(Expr::BinaryOp { op, @@ -190,7 +196,10 @@ fn resolve_transformers( Some(d) => Some(Box::new(resolve_transformers(*d, resolved)?)), None => None, }; - Ok(Expr::Case { arms: new_arms, default }) + Ok(Expr::Case { + arms: new_arms, + default, + }) } // Column, Literal, and an already-built Transform pass through. other => Ok(other), diff --git a/src/datafusion/plan.rs b/src/datafusion/plan.rs index b22b9c6..78c27db 100644 --- a/src/datafusion/plan.rs +++ b/src/datafusion/plan.rs @@ -565,8 +565,10 @@ fn execute_rel( row.insert(table.clone(), null_row); } None => { - let key_repr: Vec = - key.iter().map(crate::datafusion::expr::display_value).collect(); + let key_repr: Vec = key + .iter() + .map(crate::datafusion::expr::display_value) + .collect(); return Err(InterpError::MissingKey(format!( "No row in static table '{table}' matches key ({})", key_repr.join(", ") @@ -661,10 +663,19 @@ fn resolve_tables( left, right, outer, .. } => { resolve_tables(left, row_table_names, nullable, out, nullable_out); - resolve_tables(right, row_table_names, nullable || *outer, out, nullable_out); + resolve_tables( + right, + row_table_names, + nullable || *outer, + out, + nullable_out, + ); } RelNode::LookupJoin { - input, table, outer, .. + input, + table, + outer, + .. } => { resolve_tables(input, row_table_names, nullable, out, nullable_out); out.insert(table.clone(), (table.clone(), false)); @@ -745,8 +756,12 @@ pub fn validate_columns( )); } unnest_seen = true; - let old_input = - std::mem::replace(&mut plan.input, RelNode::TableScan { table: String::new() }); + let old_input = std::mem::replace( + &mut plan.input, + RelNode::TableScan { + table: String::new(), + }, + ); plan.input = RelNode::Unnest { input: Box::new(old_input), list_expr, @@ -1198,17 +1213,29 @@ fn validate_expr( Expr::Case { arms, default } => { for (cond, result) in arms { validate_expr( - cond, resolved, row_schemas, static_schemas, effective_schemas, + cond, + resolved, + row_schemas, + static_schemas, + effective_schemas, used_columns, )?; validate_expr( - result, resolved, row_schemas, static_schemas, effective_schemas, + result, + resolved, + row_schemas, + static_schemas, + effective_schemas, used_columns, )?; } if let Some(d) = default { validate_expr( - d, resolved, row_schemas, static_schemas, effective_schemas, + d, + resolved, + row_schemas, + static_schemas, + effective_schemas, used_columns, )?; } diff --git a/src/datafusion/types.rs b/src/datafusion/types.rs index 9bbf4af..c9593d4 100644 --- a/src/datafusion/types.rs +++ b/src/datafusion/types.rs @@ -271,7 +271,12 @@ fn common_base(args: &[FieldType]) -> Base { match args.first() { None => Base::Other, Some(first) if args.iter().all(|a| a.base == first.base) => first.base.clone(), - _ if args.iter().all(|a| matches!(a.base, Base::Int | Base::Float)) => Base::Float, + _ if args + .iter() + .all(|a| matches!(a.base, Base::Int | Base::Float)) => + { + Base::Float + } _ => Base::Other, } } diff --git a/src/duckdb/mod.rs b/src/duckdb/mod.rs index bcdf907..7e8ca14 100644 --- a/src/duckdb/mod.rs +++ b/src/duckdb/mod.rs @@ -1,29 +1,191 @@ -//! The DuckDB-semantics interpreter: `DuckDBInferFn`. +//! The DuckDB-semantics engine: `DuckDBInferFn`. //! -//! Same Python API as `InferFn` minus the transformer callout. Stub: the SQL -//! is parsed in the DuckDB dialect (so the wiring is real) and nothing else -//! happens — `infer()` raises rather than returning results that would -//! silently carry DataFusion semantics. -//! -//! Plan building, expression evaluation and type inference get their own -//! modules here as they land; none of them exist yet. +//! Same Python API as `InferFn` minus the transformer callout: SQL + +//! pydantic row model + pyarrow static tables in the constructor, then +//! `infer()` maps row objects through the specializer's compiled program. +//! `prepare()` does the SQL work (frontend -> lower -> verify); this module +//! is only the Python boundary — schema extraction on the way in, map +//! materialization for the join probes, output-model rows on the way out. use std::collections::HashMap; -use pyo3::exceptions::PyNotImplementedError; use pyo3::prelude::*; use pyo3::types::PyDict; -use sqlparser::dialect::DuckDbDialect; -use sqlparser::parser::Parser; use crate::error::InterpError; +use crate::schema; +use crate::specializer::exec::interp::{compile, InterpFn}; +use crate::specializer::exec::{Batch, ColData, KeyBits, OutCol, ScalarVal, StaticData}; +use crate::specializer::ir::{Col, ColTy, StaticTy, Ty}; +use crate::specializer::plan::StaticTable; +use crate::specializer::{prepare, StaticSpec}; +use crate::types::{Base, FieldType}; + +/// The scalar slice of the type lattice the specializer handles. `None` +/// means the column can't cross this boundary (struct/list/Other). +fn base_to_ty(b: &Base) -> Option { + match b { + Base::Bool => Some(Ty::I1), + Base::Int => Some(Ty::I64), + Base::Float => Some(Ty::F64), + Base::Str => Some(Ty::Str), + _ => None, + } +} + +fn ty_to_base(t: Ty) -> Base { + match t { + Ty::I1 => Base::Bool, + Ty::I64 => Base::Int, + Ty::F64 => Base::Float, + Ty::Str => Base::Str, + } +} + +fn build_err(msg: impl Into) -> PyErr { + InterpError::Build(msg.into()).into() +} + +/// One map static from a pyarrow Table, per its `StaticSpec` recipe: rows +/// with a NULL key are dropped (a NULL never equi-matches), a NULL in a +/// value column is an error, and an int column joined against a float +/// expression converts here (the declared key type is the expression's). +fn materialize_map( + py: Python<'_>, + table: &Py, + spec: &StaticSpec, + key_tys: &[Ty], + val_tys: &[Ty], +) -> PyResult { + let rows = table.bind(py).call_method0("to_pylist")?; + let mut entries = Vec::new(); + 'row: for item in rows.try_iter()? { + let row = item?; + let row = row.cast::().map_err(|_| { + build_err(format!( + "static table '{}': to_pylist row is not a dict", + spec.table + )) + })?; + let get = |name: &str| { + row.get_item(name)?.ok_or_else(|| { + build_err(format!( + "static table '{}' row is missing column '{name}'", + spec.table + )) + }) + }; + let mut keys = Vec::with_capacity(key_tys.len()); + for (name, ty) in spec.key_cols.iter().zip(key_tys) { + let v = get(name)?; + if v.is_none() { + continue 'row; + } + keys.push(match ty { + Ty::I1 => KeyBits::I1(v.extract()?), + Ty::I64 => KeyBits::I64(v.extract()?), + Ty::F64 => KeyBits::F64(v.extract::()?.to_bits()), + Ty::Str => KeyBits::Str(v.extract()?), + }); + } + let mut vals = Vec::with_capacity(val_tys.len()); + for (name, ty) in spec.val_cols.iter().zip(val_tys) { + let v = get(name)?; + if v.is_none() { + return Err(build_err(format!( + "static table '{}' has a NULL in value column '{name}' — joins to NULL \ + values are not supported", + spec.table + ))); + } + vals.push(match ty { + Ty::I1 => ScalarVal::I1(v.extract()?), + Ty::I64 => ScalarVal::I64(v.extract()?), + Ty::F64 => ScalarVal::F64(v.extract()?), + Ty::Str => ScalarVal::Str(v.extract()?), + }); + } + entries.push((keys, vals)); + } + Ok(StaticData::Map(entries)) +} -#[pyclass] +fn model_from_fields(py: Python<'_>, fields: Vec<(String, FieldType)>) -> PyResult> { + let create_model = PyModule::import(py, "pydantic")?.getattr("create_model")?; + let ellipsis = PyModule::import(py, "builtins")?.getattr("Ellipsis")?; + let kwargs = PyDict::new(py); + for (name, ft) in fields { + kwargs.set_item(name, (schema::field_type_to_python(py, ft)?, &ellipsis))?; + } + Ok(create_model.call(("OutputRow",), Some(&kwargs))?.unbind()) +} + +fn synthesize_output_model(py: Python<'_>, out_cols: &[Col]) -> PyResult> { + let fields = out_cols + .iter() + .map(|c| { + ( + c.name.clone(), + FieldType { + base: ty_to_base(c.ty.ty), + nullable: c.ty.nullable, + }, + ) + }) + .collect(); + model_from_fields(py, fields) +} + +/// AC #2's constant emitter: a static-tables-only query is evaluated ONCE, +/// here at build time, by DuckDB itself — nothing dynamic remains and no IR +/// is built at all. Statics materialize as native tables (duckdb's +/// registered-arrow scan path has divergent filter semantics — see the +/// builtin-pins spec). Returns the fixed row dicts plus the result schema. +fn eval_static_only( + py: Python<'_>, + sql: &str, + static_tables: &HashMap>, +) -> PyResult<(Vec>, Vec<(String, FieldType)>)> { + let duckdb = PyModule::import(py, "duckdb")?; + let con = duckdb.call_method0("connect")?; + for (name, table) in static_tables { + con.call_method1("register", (format!("__arrow_{name}"), table))?; + con.call_method1( + "execute", + (format!( + "CREATE TABLE \"{name}\" AS SELECT * FROM \"__arrow_{name}\"" + ),), + )?; + } + let arrow = con + .call_method1("execute", (sql,))? + .call_method0("to_arrow_table")?; + let schema_obj = arrow.getattr("schema")?.unbind(); + let fields = schema::arrow_schema_to_ordered_fields(py, &schema_obj)?; + let mut rows = Vec::new(); + for r in arrow.call_method0("to_pylist")?.try_iter()? { + rows.push(r?.unbind()); + } + Ok((rows, fields)) +} + +enum Engine { + Compiled { + fun: InterpFn, + in_cols: Vec, + out_cols: Vec, + }, + /// Fixed row dicts from a static-only query, re-validated through the + /// output model on every `infer` call. + Constant { rows: Vec> }, +} + +#[pyclass(unsendable)] pub struct DuckDBInferFn { - /// No output model is derived yet, so this is whatever the caller declared - /// (or `None`) rather than a synthesized one. + engine: Engine, + row_table: String, #[pyo3(get)] - output_model: Option>, + output_model: Py, } #[pymethods] @@ -31,26 +193,250 @@ impl DuckDBInferFn { #[new] #[pyo3(signature = (sql, row_tables, static_tables, output_model=None))] fn new( + py: Python<'_>, sql: String, row_tables: HashMap>, static_tables: HashMap>, output_model: Option>, ) -> PyResult { - let _ = (row_tables, static_tables); - Parser::parse_sql(&DuckDbDialect {}, &sql) - .map_err(|e| InterpError::Build(format!("SQL parse error: {e}")))?; - Ok(DuckDBInferFn { output_model }) + let (row_table, model) = match row_tables.len() { + 1 => row_tables.into_iter().next().unwrap(), + n => { + return Err(build_err(format!( + "unsupported: the specializer takes exactly one row table, got {n}" + ))) + } + }; + let mut in_cols = Vec::new(); + for (name, ft) in schema::pydantic_model_fields_ordered(py, &model)? { + let ty = base_to_ty(&ft.base).ok_or_else(|| { + build_err(format!( + "unsupported: row column '{name}' has a non-scalar type" + )) + })?; + in_cols.push(Col { + name, + ty: ColTy { + ty, + nullable: ft.nullable, + }, + }); + } + + // Non-scalar static columns are omitted from the catalog rather than + // rejected: unreferenced ones cost nothing, referenced ones fail the + // bind by name. + let mut catalog = Vec::new(); + for (name, table) in &static_tables { + let schema_obj = table + .bind(py) + .getattr("schema") + .map_err(|e| { + build_err(format!("static table '{name}' is not a pyarrow.Table: {e}")) + })? + .unbind(); + let mut cols = Vec::new(); + for (cname, ft) in schema::arrow_schema_to_ordered_fields(py, &schema_obj)? { + if let Some(ty) = base_to_ty(&ft.base) { + cols.push(Col { + name: cname, + ty: ColTy { + ty, + nullable: ft.nullable, + }, + }); + } + } + catalog.push(StaticTable { + name: name.clone(), + cols, + }); + } + + use super::specializer::PrepareError; + let prepared = match prepare(&sql, &row_table, &in_cols, &catalog) { + Ok(p) => p, + // Unsupported/unparseable SQL might still be a static-tables-only + // query (static driving table, aggregation, ORDER BY, DuckDB + // dialect beyond sqlparser): try the constant-emitter path. It + // self-validates — a dynamic query references the row table, + // which DuckDB does not know, so evaluation fails and the + // original clean error surfaces unchanged. Bind errors stay hard. + Err(e @ (PrepareError::Unsupported(_) | PrepareError::Parse(_))) => { + match eval_static_only(py, &sql, &static_tables) { + Ok((rows, fields)) => { + let output_model = match output_model { + Some(m) => m, + None => model_from_fields(py, fields)?, + }; + return Ok(DuckDBInferFn { + engine: Engine::Constant { rows }, + row_table, + output_model, + }); + } + Err(_) => return Err(build_err(e.to_string())), + } + } + Err(e) => return Err(build_err(e.to_string())), + }; + + // Program statics and StaticSpecs are both indexed by join id. + let mut data = Vec::with_capacity(prepared.statics.len()); + for (spec, sty) in prepared.statics.iter().zip(&prepared.program.statics) { + let StaticTy::Map { keys, values } = sty else { + return Err(build_err("internal: v0 lowering emits only map statics")); + }; + let table = static_tables + .get(&spec.table) + .expect("spec names come from the catalog"); + data.push(materialize_map(py, table, spec, keys, values)?); + } + + let fun = compile(&prepared.program, data).map_err(|e| build_err(e.to_string()))?; + let output_model = match output_model { + // Supplied models are trusted as-is in v0 (no shape validation). + Some(m) => m, + None => synthesize_output_model(py, &prepared.program.out_cols)?, + }; + Ok(DuckDBInferFn { + engine: Engine::Compiled { + fun, + in_cols, + out_cols: prepared.program.out_cols.clone(), + }, + row_table, + output_model, + }) } #[pyo3(signature = (tables=None, **kwargs))] fn infer( &self, + py: Python<'_>, tables: Option>>>, kwargs: Option>, ) -> PyResult>> { - let _ = (tables, kwargs); - Err(PyNotImplementedError::new_err( - "the DuckDB interpreter is a stub; use InferFn", - )) + let mut merged: HashMap>> = tables.unwrap_or_default(); + if let Some(kwargs) = kwargs { + for (k, v) in kwargs.iter() { + merged.insert(k.extract()?, v.extract()?); + } + } + if let Some(bad) = merged.keys().find(|k| **k != self.row_table) { + return Err(pyo3::exceptions::PyValueError::new_err(format!( + "unknown table '{bad}' (the row table is '{}')", + self.row_table + ))); + } + let rows = merged.remove(&self.row_table).unwrap_or_default(); + + let (fun, in_cols, out_cols) = match &self.engine { + Engine::Compiled { + fun, + in_cols, + out_cols, + } => (fun, in_cols, out_cols), + Engine::Constant { rows: fixed } => { + let model = self.output_model.bind(py); + let mut out = Vec::with_capacity(fixed.len()); + for r in fixed { + out.push(model.call_method1("model_validate", (r,))?.unbind()); + } + return Ok(out); + } + }; + + let n = rows.len(); + let mut cols: Vec = in_cols + .iter() + .map(|c| match c.ty.ty { + Ty::I1 => ColData::I1 { + valid: Vec::with_capacity(n), + data: Vec::with_capacity(n), + }, + Ty::I64 => ColData::I64 { + valid: Vec::with_capacity(n), + data: Vec::with_capacity(n), + }, + Ty::F64 => ColData::F64 { + valid: Vec::with_capacity(n), + data: Vec::with_capacity(n), + }, + Ty::Str => ColData::Str { + valid: Vec::with_capacity(n), + data: Vec::with_capacity(n), + }, + }) + .collect(); + for row_obj in &rows { + let bound = row_obj.bind(py); + for (c, col) in in_cols.iter().zip(&mut cols) { + let attr = bound.getattr(c.name.as_str()).map_err(|e| { + pyo3::exceptions::PyValueError::new_err(format!( + "Row for table '{}' is missing attribute '{}': {e}", + self.row_table, c.name + )) + })?; + let null = attr.is_none(); + if null && !c.ty.nullable { + return Err(pyo3::exceptions::PyValueError::new_err(format!( + "column '{}' is not nullable but a row has None", + c.name + ))); + } + match col { + ColData::I1 { valid, data } => { + valid.push(!null); + data.push(if null { false } else { attr.extract()? }); + } + ColData::I64 { valid, data } => { + valid.push(!null); + data.push(if null { 0 } else { attr.extract()? }); + } + ColData::F64 { valid, data } => { + valid.push(!null); + data.push(if null { 0.0 } else { attr.extract()? }); + } + ColData::Str { valid, data } => { + valid.push(!null); + data.push(if null { String::new() } else { attr.extract()? }); + } + } + } + } + + let batch = Batch { rows: n, cols }; + let mut st = fun.new_state(); + fun.run(&batch, &mut st) + .map_err(|t| PyErr::from(InterpError::Eval(t.0)))?; + + let model = self.output_model.bind(py); + let mut out = Vec::with_capacity(st.emitted); + for r in 0..st.emitted { + let dict = PyDict::new(py); + for (c, oc) in out_cols.iter().zip(&st.out) { + match oc { + OutCol::I1(v) => { + let (ok, x) = v[r]; + dict.set_item(&c.name, ok.then_some(x))?; + } + OutCol::I64(v) => { + let (ok, x) = v[r]; + dict.set_item(&c.name, ok.then_some(x))?; + } + OutCol::F64(v) => { + let (ok, x) = v[r]; + dict.set_item(&c.name, ok.then_some(x))?; + } + OutCol::Str(v) => { + let (ok, s) = v[r]; + dict.set_item(&c.name, ok.then(|| st.arena.get(s)))?; + } + } + } + out.push(model.call_method1("model_validate", (dict,))?.unbind()); + } + Ok(out) } } diff --git a/src/lookup.rs b/src/lookup.rs index 9d6f76b..d39d0c8 100644 --- a/src/lookup.rs +++ b/src/lookup.rs @@ -3,8 +3,8 @@ use std::collections::HashMap; use pyo3::prelude::*; use pyo3::types::PyDict; -use crate::value::Value; use crate::error::InterpError; +use crate::value::Value; pub struct LookupIndex { pub index: HashMap, HashMap>, diff --git a/src/schema.rs b/src/schema.rs index f2a2464..3002f9e 100644 --- a/src/schema.rs +++ b/src/schema.rs @@ -21,7 +21,7 @@ pub fn from_pydantic_model(py: Python<'_>, model_class: &Py) -> Result, model_class: &Py, ) -> Result, InterpError> { @@ -92,17 +92,20 @@ pub fn arrow_schema_to_ordered_fields( .getattr("names") .and_then(|n| n.extract()) .map_err(|e| { - InterpError::Build(format!("transformer output schema is not a pyarrow.Schema: {e}")) + InterpError::Build(format!( + "transformer output schema is not a pyarrow.Schema: {e}" + )) })?; let pa_types = PyModule::import(py, "pyarrow.types") .map_err(|e| InterpError::Build(format!("Failed to import pyarrow.types: {e}")))?; let mut out = Vec::with_capacity(names.len()); for name in names { - let field = bound - .call_method1("field", (name.as_str(),)) - .map_err(|e| InterpError::Build(format!("Failed to read output field '{name}': {e}")))?; - let ft = arrow_field_to_field_type(&pa_types, &field) - .map_err(|e| InterpError::Build(format!("Failed to read type of output field '{name}': {e}")))?; + let field = bound.call_method1("field", (name.as_str(),)).map_err(|e| { + InterpError::Build(format!("Failed to read output field '{name}': {e}")) + })?; + let ft = arrow_field_to_field_type(&pa_types, &field).map_err(|e| { + InterpError::Build(format!("Failed to read type of output field '{name}': {e}")) + })?; out.push((name, ft)); } Ok(out) @@ -123,7 +126,10 @@ fn arrow_field_to_field_type( } fn arrow_pytype_to_base(pa_types: &Bound<'_, PyModule>, ty: &Bound<'_, PyAny>) -> PyResult { - if pa_types.call_method1("is_struct", (ty,))?.extract::()? { + if pa_types + .call_method1("is_struct", (ty,))? + .extract::()? + { let num_fields: usize = ty.getattr("num_fields")?.extract()?; let mut fields = Vec::with_capacity(num_fields); for i in 0..num_fields { @@ -134,7 +140,9 @@ fn arrow_pytype_to_base(pa_types: &Bound<'_, PyModule>, ty: &Bound<'_, PyAny>) - return Ok(Base::Struct(fields)); } let is_list = pa_types.call_method1("is_list", (ty,))?.extract::()? - || pa_types.call_method1("is_large_list", (ty,))?.extract::()?; + || pa_types + .call_method1("is_large_list", (ty,))? + .extract::()?; if is_list { let value_field = ty.getattr("value_field")?; let inner = arrow_field_to_field_type(pa_types, &value_field)?; @@ -218,8 +226,7 @@ fn annotation_to_field_type( } let base = if non_none.len() == 1 { - let inner = - annotation_to_field_type(py, typing, types_module, non_none[0].bind(py))?; + let inner = annotation_to_field_type(py, typing, types_module, non_none[0].bind(py))?; nullable = nullable || inner.nullable; inner.base } else { @@ -256,7 +263,10 @@ fn annotation_to_field_type( /// Is `annotation` a `pydantic.BaseModel` subclass (a nested struct field)? /// `issubclass()` raises `TypeError` for a non-class annotation (e.g. /// `typing.Any`, `list[int]`) — treated as "not a model", not an error. -fn is_pydantic_model_class(py: Python<'_>, annotation: &Bound<'_, PyAny>) -> Result { +fn is_pydantic_model_class( + py: Python<'_>, + annotation: &Bound<'_, PyAny>, +) -> Result { let pydantic = PyModule::import(py, "pydantic") .map_err(|e| InterpError::Build(format!("Failed to import pydantic: {e}")))?; let base_model = pydantic diff --git a/src/specializer/exec/casemap.rs b/src/specializer/exec/casemap.rs new file mode 100644 index 0000000..2e36398 --- /dev/null +++ b/src/specializer/exec/casemap.rs @@ -0,0 +1,187 @@ +//! DuckDB's SIMPLE case mapping, as measured — see scripts/gen_casemap.py +//! for the full story (simple-vs-full mapping, Unicode version skew, and why +//! a dependency would be the wrong spec twice over). +//! +//! GENERATED by `uv run python scripts/gen_casemap.py` against duckdb-python +//! 1.5.5; regenerate after a duckdb bump. The full-codepoint census in +//! tests/test_duckdb_interpreter.py fails loudly if this table drifts from +//! the duckdb the suite runs against — the census, not this file, is the +//! authority. +//! +//! Entries are (codepoint, mapped) pairs, sorted by codepoint, covering only +//! the codepoints where the dependency-free fallback (Rust's full map iff it +//! is 1:1, else identity) disagrees with DuckDB. Everything else goes through +//! the fallback in `simple_upper`/`simple_lower`. + +/// Simple uppercase for one codepoint, exactly as DuckDB computes it. +pub fn simple_upper(c: char) -> char { + if let Ok(i) = UPPER_EXCEPTIONS.binary_search_by_key(&c, |e| e.0) { + return UPPER_EXCEPTIONS[i].1; + } + let mut it = c.to_uppercase(); + let first = it.next().expect("case iterators are non-empty"); + if it.next().is_none() { + first + } else { + c + } +} + +/// Simple lowercase for one codepoint, exactly as DuckDB computes it. +pub fn simple_lower(c: char) -> char { + if let Ok(i) = LOWER_EXCEPTIONS.binary_search_by_key(&c, |e| e.0) { + return LOWER_EXCEPTIONS[i].1; + } + let mut it = c.to_lowercase(); + let first = it.next().expect("case iterators are non-empty"); + if it.next().is_none() { + first + } else { + c + } +} + +pub(super) const UPPER_EXCEPTIONS: &[(char, char)] = &[ + ('\u{00DF}', '\u{1E9E}'), // 'ß' -> 'ẞ' + ('\u{019B}', '\u{019B}'), // 'ƛ' -> 'ƛ' + ('\u{0264}', '\u{0264}'), // 'ɤ' -> 'ɤ' + ('\u{1C8A}', '\u{1C8A}'), // 'ᲊ' -> 'ᲊ' + ('\u{1F80}', '\u{1F88}'), // 'ᾀ' -> 'ᾈ' + ('\u{1F81}', '\u{1F89}'), // 'ᾁ' -> 'ᾉ' + ('\u{1F82}', '\u{1F8A}'), // 'ᾂ' -> 'ᾊ' + ('\u{1F83}', '\u{1F8B}'), // 'ᾃ' -> 'ᾋ' + ('\u{1F84}', '\u{1F8C}'), // 'ᾄ' -> 'ᾌ' + ('\u{1F85}', '\u{1F8D}'), // 'ᾅ' -> 'ᾍ' + ('\u{1F86}', '\u{1F8E}'), // 'ᾆ' -> 'ᾎ' + ('\u{1F87}', '\u{1F8F}'), // 'ᾇ' -> 'ᾏ' + ('\u{1F90}', '\u{1F98}'), // 'ᾐ' -> 'ᾘ' + ('\u{1F91}', '\u{1F99}'), // 'ᾑ' -> 'ᾙ' + ('\u{1F92}', '\u{1F9A}'), // 'ᾒ' -> 'ᾚ' + ('\u{1F93}', '\u{1F9B}'), // 'ᾓ' -> 'ᾛ' + ('\u{1F94}', '\u{1F9C}'), // 'ᾔ' -> 'ᾜ' + ('\u{1F95}', '\u{1F9D}'), // 'ᾕ' -> 'ᾝ' + ('\u{1F96}', '\u{1F9E}'), // 'ᾖ' -> 'ᾞ' + ('\u{1F97}', '\u{1F9F}'), // 'ᾗ' -> 'ᾟ' + ('\u{1FA0}', '\u{1FA8}'), // 'ᾠ' -> 'ᾨ' + ('\u{1FA1}', '\u{1FA9}'), // 'ᾡ' -> 'ᾩ' + ('\u{1FA2}', '\u{1FAA}'), // 'ᾢ' -> 'ᾪ' + ('\u{1FA3}', '\u{1FAB}'), // 'ᾣ' -> 'ᾫ' + ('\u{1FA4}', '\u{1FAC}'), // 'ᾤ' -> 'ᾬ' + ('\u{1FA5}', '\u{1FAD}'), // 'ᾥ' -> 'ᾭ' + ('\u{1FA6}', '\u{1FAE}'), // 'ᾦ' -> 'ᾮ' + ('\u{1FA7}', '\u{1FAF}'), // 'ᾧ' -> 'ᾯ' + ('\u{1FB3}', '\u{1FBC}'), // 'ᾳ' -> 'ᾼ' + ('\u{1FC3}', '\u{1FCC}'), // 'ῃ' -> 'ῌ' + ('\u{1FF3}', '\u{1FFC}'), // 'ῳ' -> 'ῼ' + ('\u{A7CD}', '\u{A7CD}'), // 'ꟍ' -> 'ꟍ' + ('\u{A7CF}', '\u{A7CF}'), // '\ua7cf' -> '\ua7cf' + ('\u{A7D3}', '\u{A7D3}'), // 'ꟓ' -> 'ꟓ' + ('\u{A7D5}', '\u{A7D5}'), // 'ꟕ' -> 'ꟕ' + ('\u{A7DB}', '\u{A7DB}'), // 'ꟛ' -> 'ꟛ' + ('\u{10D70}', '\u{10D70}'), // '𐵰' -> '𐵰' + ('\u{10D71}', '\u{10D71}'), // '𐵱' -> '𐵱' + ('\u{10D72}', '\u{10D72}'), // '𐵲' -> '𐵲' + ('\u{10D73}', '\u{10D73}'), // '𐵳' -> '𐵳' + ('\u{10D74}', '\u{10D74}'), // '𐵴' -> '𐵴' + ('\u{10D75}', '\u{10D75}'), // '𐵵' -> '𐵵' + ('\u{10D76}', '\u{10D76}'), // '𐵶' -> '𐵶' + ('\u{10D77}', '\u{10D77}'), // '𐵷' -> '𐵷' + ('\u{10D78}', '\u{10D78}'), // '𐵸' -> '𐵸' + ('\u{10D79}', '\u{10D79}'), // '𐵹' -> '𐵹' + ('\u{10D7A}', '\u{10D7A}'), // '𐵺' -> '𐵺' + ('\u{10D7B}', '\u{10D7B}'), // '𐵻' -> '𐵻' + ('\u{10D7C}', '\u{10D7C}'), // '𐵼' -> '𐵼' + ('\u{10D7D}', '\u{10D7D}'), // '𐵽' -> '𐵽' + ('\u{10D7E}', '\u{10D7E}'), // '𐵾' -> '𐵾' + ('\u{10D7F}', '\u{10D7F}'), // '𐵿' -> '𐵿' + ('\u{10D80}', '\u{10D80}'), // '𐶀' -> '𐶀' + ('\u{10D81}', '\u{10D81}'), // '𐶁' -> '𐶁' + ('\u{10D82}', '\u{10D82}'), // '𐶂' -> '𐶂' + ('\u{10D83}', '\u{10D83}'), // '𐶃' -> '𐶃' + ('\u{10D84}', '\u{10D84}'), // '𐶄' -> '𐶄' + ('\u{10D85}', '\u{10D85}'), // '𐶅' -> '𐶅' + ('\u{16EBB}', '\u{16EBB}'), // '\U00016ebb' -> '\U00016ebb' + ('\u{16EBC}', '\u{16EBC}'), // '\U00016ebc' -> '\U00016ebc' + ('\u{16EBD}', '\u{16EBD}'), // '\U00016ebd' -> '\U00016ebd' + ('\u{16EBE}', '\u{16EBE}'), // '\U00016ebe' -> '\U00016ebe' + ('\u{16EBF}', '\u{16EBF}'), // '\U00016ebf' -> '\U00016ebf' + ('\u{16EC0}', '\u{16EC0}'), // '\U00016ec0' -> '\U00016ec0' + ('\u{16EC1}', '\u{16EC1}'), // '\U00016ec1' -> '\U00016ec1' + ('\u{16EC2}', '\u{16EC2}'), // '\U00016ec2' -> '\U00016ec2' + ('\u{16EC3}', '\u{16EC3}'), // '\U00016ec3' -> '\U00016ec3' + ('\u{16EC4}', '\u{16EC4}'), // '\U00016ec4' -> '\U00016ec4' + ('\u{16EC5}', '\u{16EC5}'), // '\U00016ec5' -> '\U00016ec5' + ('\u{16EC6}', '\u{16EC6}'), // '\U00016ec6' -> '\U00016ec6' + ('\u{16EC7}', '\u{16EC7}'), // '\U00016ec7' -> '\U00016ec7' + ('\u{16EC8}', '\u{16EC8}'), // '\U00016ec8' -> '\U00016ec8' + ('\u{16EC9}', '\u{16EC9}'), // '\U00016ec9' -> '\U00016ec9' + ('\u{16ECA}', '\u{16ECA}'), // '\U00016eca' -> '\U00016eca' + ('\u{16ECB}', '\u{16ECB}'), // '\U00016ecb' -> '\U00016ecb' + ('\u{16ECC}', '\u{16ECC}'), // '\U00016ecc' -> '\U00016ecc' + ('\u{16ECD}', '\u{16ECD}'), // '\U00016ecd' -> '\U00016ecd' + ('\u{16ECE}', '\u{16ECE}'), // '\U00016ece' -> '\U00016ece' + ('\u{16ECF}', '\u{16ECF}'), // '\U00016ecf' -> '\U00016ecf' + ('\u{16ED0}', '\u{16ED0}'), // '\U00016ed0' -> '\U00016ed0' + ('\u{16ED1}', '\u{16ED1}'), // '\U00016ed1' -> '\U00016ed1' + ('\u{16ED2}', '\u{16ED2}'), // '\U00016ed2' -> '\U00016ed2' + ('\u{16ED3}', '\u{16ED3}'), // '\U00016ed3' -> '\U00016ed3' +]; + +pub(super) const LOWER_EXCEPTIONS: &[(char, char)] = &[ + ('\u{0130}', '\u{0069}'), // 'İ' -> 'i' + ('\u{1C89}', '\u{1C89}'), // 'Ᲊ' -> 'Ᲊ' + ('\u{A7CB}', '\u{A7CB}'), // 'Ɤ' -> 'Ɤ' + ('\u{A7CC}', '\u{A7CC}'), // 'Ꟍ' -> 'Ꟍ' + ('\u{A7CE}', '\u{A7CE}'), // '\ua7ce' -> '\ua7ce' + ('\u{A7D2}', '\u{A7D2}'), // '\ua7d2' -> '\ua7d2' + ('\u{A7D4}', '\u{A7D4}'), // '\ua7d4' -> '\ua7d4' + ('\u{A7DA}', '\u{A7DA}'), // 'Ꟛ' -> 'Ꟛ' + ('\u{A7DC}', '\u{A7DC}'), // 'Ƛ' -> 'Ƛ' + ('\u{10D50}', '\u{10D50}'), // '𐵐' -> '𐵐' + ('\u{10D51}', '\u{10D51}'), // '𐵑' -> '𐵑' + ('\u{10D52}', '\u{10D52}'), // '𐵒' -> '𐵒' + ('\u{10D53}', '\u{10D53}'), // '𐵓' -> '𐵓' + ('\u{10D54}', '\u{10D54}'), // '𐵔' -> '𐵔' + ('\u{10D55}', '\u{10D55}'), // '𐵕' -> '𐵕' + ('\u{10D56}', '\u{10D56}'), // '𐵖' -> '𐵖' + ('\u{10D57}', '\u{10D57}'), // '𐵗' -> '𐵗' + ('\u{10D58}', '\u{10D58}'), // '𐵘' -> '𐵘' + ('\u{10D59}', '\u{10D59}'), // '𐵙' -> '𐵙' + ('\u{10D5A}', '\u{10D5A}'), // '𐵚' -> '𐵚' + ('\u{10D5B}', '\u{10D5B}'), // '𐵛' -> '𐵛' + ('\u{10D5C}', '\u{10D5C}'), // '𐵜' -> '𐵜' + ('\u{10D5D}', '\u{10D5D}'), // '𐵝' -> '𐵝' + ('\u{10D5E}', '\u{10D5E}'), // '𐵞' -> '𐵞' + ('\u{10D5F}', '\u{10D5F}'), // '𐵟' -> '𐵟' + ('\u{10D60}', '\u{10D60}'), // '𐵠' -> '𐵠' + ('\u{10D61}', '\u{10D61}'), // '𐵡' -> '𐵡' + ('\u{10D62}', '\u{10D62}'), // '𐵢' -> '𐵢' + ('\u{10D63}', '\u{10D63}'), // '𐵣' -> '𐵣' + ('\u{10D64}', '\u{10D64}'), // '𐵤' -> '𐵤' + ('\u{10D65}', '\u{10D65}'), // '𐵥' -> '𐵥' + ('\u{16EA0}', '\u{16EA0}'), // '\U00016ea0' -> '\U00016ea0' + ('\u{16EA1}', '\u{16EA1}'), // '\U00016ea1' -> '\U00016ea1' + ('\u{16EA2}', '\u{16EA2}'), // '\U00016ea2' -> '\U00016ea2' + ('\u{16EA3}', '\u{16EA3}'), // '\U00016ea3' -> '\U00016ea3' + ('\u{16EA4}', '\u{16EA4}'), // '\U00016ea4' -> '\U00016ea4' + ('\u{16EA5}', '\u{16EA5}'), // '\U00016ea5' -> '\U00016ea5' + ('\u{16EA6}', '\u{16EA6}'), // '\U00016ea6' -> '\U00016ea6' + ('\u{16EA7}', '\u{16EA7}'), // '\U00016ea7' -> '\U00016ea7' + ('\u{16EA8}', '\u{16EA8}'), // '\U00016ea8' -> '\U00016ea8' + ('\u{16EA9}', '\u{16EA9}'), // '\U00016ea9' -> '\U00016ea9' + ('\u{16EAA}', '\u{16EAA}'), // '\U00016eaa' -> '\U00016eaa' + ('\u{16EAB}', '\u{16EAB}'), // '\U00016eab' -> '\U00016eab' + ('\u{16EAC}', '\u{16EAC}'), // '\U00016eac' -> '\U00016eac' + ('\u{16EAD}', '\u{16EAD}'), // '\U00016ead' -> '\U00016ead' + ('\u{16EAE}', '\u{16EAE}'), // '\U00016eae' -> '\U00016eae' + ('\u{16EAF}', '\u{16EAF}'), // '\U00016eaf' -> '\U00016eaf' + ('\u{16EB0}', '\u{16EB0}'), // '\U00016eb0' -> '\U00016eb0' + ('\u{16EB1}', '\u{16EB1}'), // '\U00016eb1' -> '\U00016eb1' + ('\u{16EB2}', '\u{16EB2}'), // '\U00016eb2' -> '\U00016eb2' + ('\u{16EB3}', '\u{16EB3}'), // '\U00016eb3' -> '\U00016eb3' + ('\u{16EB4}', '\u{16EB4}'), // '\U00016eb4' -> '\U00016eb4' + ('\u{16EB5}', '\u{16EB5}'), // '\U00016eb5' -> '\U00016eb5' + ('\u{16EB6}', '\u{16EB6}'), // '\U00016eb6' -> '\U00016eb6' + ('\u{16EB7}', '\u{16EB7}'), // '\U00016eb7' -> '\U00016eb7' + ('\u{16EB8}', '\u{16EB8}'), // '\U00016eb8' -> '\U00016eb8' +]; diff --git a/src/specializer/exec/interp.rs b/src/specializer/exec/interp.rs index aabae64..16e09b2 100644 --- a/src/specializer/exec/interp.rs +++ b/src/specializer/exec/interp.rs @@ -15,8 +15,9 @@ //! * `itof` is `as f64` (may round — same as DuckDB's BIGINT->DOUBLE). //! * `ftoi.trunc` rounds toward zero; `ftoi.round` half-away-from-zero //! (matches DuckDB CAST); both trap on non-finite or out-of-i64-range. -//! * `stoi.opt`/`stof.opt` are exact `str::parse` — no whitespace trimming; -//! whether SQL CAST trims is a lowering decision to pin at M-lower. +//! * `stoi.opt`/`stof.opt` trim ASCII whitespace then `str::parse` — pinned +//! at M-lower against DuckDB CAST (measured: `' 5'::BIGINT = 5`); the +//! empty/whitespace-only string fails the parse. //! * `itos`/`ftos` format into the arena; `ftos` uses Rust's shortest //! round-trip form (provisional; oracle-pinned at M-lower). //! * On a false validity flag the payload is the type default; `load.opt` @@ -25,10 +26,11 @@ use std::collections::HashMap; use std::fmt::Write as _; +use super::super::ir::verify::{verify, VerifyError}; use super::super::ir::{ - self, BinOp, CmpPred, Inst, Program, RoundMode, StaticTy, Term, Ty, Value, + self, BinOp, CmpPred, Inst, NumOp1, Program, RoundMode, StaticTy, StrOp1, Term, TrimSide, Ty, + Value, }; -use super::super::ir::verify::{verify, VerifyError}; use super::{ Arena, Batch, ColData, KeyBits, OutCol, RegVal, RunState, ScalarVal, StaticData, StrRef, Trap, }; @@ -71,7 +73,10 @@ struct Ctx<'a> { } enum PreparedStatic { - Scalar { valid: bool, val: ScalarVal }, + Scalar { + valid: bool, + val: ScalarVal, + }, /// Sorted by key; probed by allocation-free binary search. Map { entries: Vec<(Vec, Vec)>, @@ -88,8 +93,17 @@ struct CBlock { /// sources and destinations are disjoint by SSA single-definition, so /// sequential copies are safe. enum CTerm { - Jump { to: usize, moves: Vec<(u32, u32)> }, - Brif { cond: u32, then_to: usize, then_moves: Vec<(u32, u32)>, else_to: usize, else_moves: Vec<(u32, u32)> }, + Jump { + to: usize, + moves: Vec<(u32, u32)>, + }, + Brif { + cond: u32, + then_to: usize, + then_moves: Vec<(u32, u32)>, + else_to: usize, + else_moves: Vec<(u32, u32)>, + }, Emit, Skip, Trap(String), @@ -131,7 +145,10 @@ pub fn compile(p: &Program, statics: Vec) -> Result { + CTerm::Brif { + cond, + then_to, + then_moves, + else_to, + else_moves, + } => { if as_i1(ctx.regs[*cond as usize]) { do_moves(ctx.regs, then_moves); bi = *then_to; @@ -349,6 +372,15 @@ fn prepare_statics( ))); } } + // Canonicalize f64 key bits BEFORE sorting, so the stored + // order agrees with the canonical bits cmp_key searches by. + for (k, _) in entries.iter_mut() { + for kb in k.iter_mut() { + if let KeyBits::F64(bits) = kb { + *bits = super::canon_f64_bits(f64::from_bits(*bits)); + } + } + } entries.sort_by(|a, b| a.0.cmp(&b.0)); if entries.windows(2).any(|w| w[0].0 == w[1].0) { return Err(CompileError::Static(format!("@{i}: duplicate map key"))); @@ -437,10 +469,22 @@ fn compile_term(p: &Program, t: &Term, slots: &HashMap) -> CTerm { let (to, moves) = mk_moves(*to, args); CTerm::Jump { to, moves } } - Term::Brif { cond, then_to, then_args, else_to, else_args } => { + Term::Brif { + cond, + then_to, + then_args, + else_to, + else_args, + } => { let (then_to, then_moves) = mk_moves(*then_to, then_args); let (else_to, else_moves) = mk_moves(*else_to, else_args); - CTerm::Brif { cond: sl(slots, *cond) as u32, then_to, then_moves, else_to, else_moves } + CTerm::Brif { + cond: sl(slots, *cond) as u32, + then_to, + then_moves, + else_to, + else_moves, + } } Term::Emit => CTerm::Emit, Term::Skip => CTerm::Skip, @@ -459,10 +503,83 @@ impl std::fmt::Write for ArenaWriter<'_> { } } +/// DuckDB's substr window arithmetic (measured 1.5.5), on codepoints — NOT +/// grapheme clusters (substr slices inside ZWJ emoji). 1-based virtual +/// positions: negative start counts from the end (`start = n + start + 1`), +/// start <= 0 consumes length before character 1, negative len is "". A +/// missing SQL length arrives as i64::MAX; the saturating add makes that +/// "rest of the string". +/// DuckDB's substr window arithmetic — the VECTORIZED path, which columns +/// (and therefore every real query and the mined corpus) take; DuckDB's own +/// constant-fold path disagrees with it on negative starts (measured +/// 2026-07-26, see the builtin-pins spec). Codepoints, NOT grapheme +/// clusters. 1-based positions: a negative start counts from the end and +/// clamps to 1 (`rs = max(n + start + 1, 1)`) while start 0 stays virtual; +/// a non-negative length runs forward `[rs, rs+len)`, a NEGATIVE length +/// slices BACKWARDS `[rs+len, rs)`; `len: None` is the 2-arg rest-of-string +/// form. +fn substr_window(s: &str, start: i64, len: Option) -> String { + let n = s.chars().count() as i64; + let rs = if start < 0 { + (n + start + 1).max(1) + } else { + start + }; + let (lo, hi) = match len { + Some(l) if l >= 0 => (rs, rs.saturating_add(l)), + Some(l) => (rs.saturating_add(l), rs), + None => (rs, n + 1), + }; + let (lo, hi) = (lo.max(1), hi.min(n + 1)); + if hi <= lo { + return String::new(); + } + s.chars() + .skip((lo - 1) as usize) + .take((hi - lo) as usize) + .collect() +} + +/// The offset/length guard DuckDB applies before the window: values outside +/// [-2^32, 2^32-1] raise an Out of Range error (measured boundary-exactly). +fn substr_range_ok(v: i64) -> bool { + (-(1i64 << 32)..(1i64 << 32)).contains(&v) +} + +/// DuckDB's DOUBLE -> VARCHAR text (measured 1.5.5): Rust's shortest +/// round-trip form, except the exponent carries an explicit sign and at +/// least two digits (`1e+300`, `1e-05`) and NaN is lowercase `nan`. +struct DuckF64(f64); + +impl std::fmt::Display for DuckF64 { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + if self.0.is_nan() { + return f.write_str("nan"); + } + let s = format!("{:?}", self.0); + match s.find('e') { + None => f.write_str(&s), + Some(pos) => { + let exp: i64 = s[pos + 1..].parse().expect("float exponent"); + write!( + f, + "{}e{}{:02}", + &s[..pos], + if exp < 0 { '-' } else { '+' }, + exp.abs() + ) + } + } + } +} + fn fmt_into_arena(arena: &mut Arena, args: std::fmt::Arguments<'_>) -> StrRef { let off = arena.0.len(); let _ = ArenaWriter(&mut arena.0).write_fmt(args); - StrRef { off, len: arena.0.len() - off } + StrRef { + off, + len: arena.0.len() - off, + } } /// Value id -> dense register slot. @@ -528,17 +645,20 @@ fn compile_inst(p: &Program, inst: &Inst, slots: &HashMap) -> InstFn { None => Err(Trap(format!("i64 overflow in {}", op.name()))), } }), - BinOp::Fadd | BinOp::Fsub | BinOp::Fmul | BinOp::Fdiv => Box::new(move |ctx| { - let (x, y) = (as_f64(ctx.regs[a]), as_f64(ctx.regs[b])); - let v = match op { - BinOp::Fadd => x + y, - BinOp::Fsub => x - y, - BinOp::Fmul => x * y, - _ => x / y, - }; - ctx.regs[dst] = RegVal::F64(v); - Ok(()) - }), + BinOp::Fadd | BinOp::Fsub | BinOp::Fmul | BinOp::Fdiv | BinOp::Frem => { + Box::new(move |ctx| { + let (x, y) = (as_f64(ctx.regs[a]), as_f64(ctx.regs[b])); + let v = match op { + BinOp::Fadd => x + y, + BinOp::Fsub => x - y, + BinOp::Fmul => x * y, + BinOp::Fdiv => x / y, + _ => x % y, + }; + ctx.regs[dst] = RegVal::F64(v); + Ok(()) + }) + } BinOp::And | BinOp::Or | BinOp::Xor => Box::new(move |ctx| { let (x, y) = (as_i1(ctx.regs[a]), as_i1(ctx.regs[b])); let v = match op { @@ -551,23 +671,22 @@ fn compile_inst(p: &Program, inst: &Inst, slots: &HashMap) -> InstFn { }), } } - Inst::Cmp { pred, ty, dst, a, b } => { + Inst::Cmp { + pred, + ty, + dst, + a, + b, + } => { let (dst, a, b) = (sl(slots, dst), sl(slots, a), sl(slots, b)); Box::new(move |ctx| { let v = match ty { Ty::I64 => apply_ord(pred, as_i64(ctx.regs[a]).cmp(&as_i64(ctx.regs[b]))), Ty::F64 => { - // IEEE partial order: NaN makes eq/lt/le/gt/ge false - // and ne true. + // DuckDB DOUBLE order, not IEEE: NaN = NaN, NaN above + // everything, zeros equal (see exec::duck_fcmp). let (x, y) = (as_f64(ctx.regs[a]), as_f64(ctx.regs[b])); - match pred { - CmpPred::Eq => x == y, - CmpPred::Ne => x != y, - CmpPred::Lt => x < y, - CmpPred::Le => x <= y, - CmpPred::Gt => x > y, - CmpPred::Ge => x >= y, - } + apply_ord(pred, super::duck_fcmp(x, y)) } Ty::Str => { let (x, y) = (as_str(ctx.regs[a]), as_str(ctx.regs[b])); @@ -589,7 +708,11 @@ fn compile_inst(p: &Program, inst: &Inst, slots: &HashMap) -> InstFn { Inst::Select { dst, cond, a, b } => { let (dst, cond, a, b) = (sl(slots, dst), sl(slots, cond), sl(slots, a), sl(slots, b)); Box::new(move |ctx| { - ctx.regs[dst] = if as_i1(ctx.regs[cond]) { ctx.regs[a] } else { ctx.regs[b] }; + ctx.regs[dst] = if as_i1(ctx.regs[cond]) { + ctx.regs[a] + } else { + ctx.regs[b] + }; Ok(()) }) } @@ -630,7 +753,8 @@ fn compile_inst(p: &Program, inst: &Inst, slots: &HashMap) -> InstFn { let (dst, a) = (sl(slots, dst), sl(slots, a)); Box::new(move |ctx| { let v = as_f64(ctx.regs[a]); - ctx.regs[dst] = RegVal::Str(fmt_into_arena(ctx.arena, format_args!("{v:?}"))); + ctx.regs[dst] = + RegVal::Str(fmt_into_arena(ctx.arena, format_args!("{}", DuckF64(v)))); Ok(()) }) } @@ -638,7 +762,7 @@ fn compile_inst(p: &Program, inst: &Inst, slots: &HashMap) -> InstFn { let (flag, dst, a) = (sl(slots, flag), sl(slots, dst), sl(slots, a)); Box::new(move |ctx| { let s = ctx.arena.get(as_str(ctx.regs[a])); - match s.parse::() { + match s.trim_ascii().parse::() { Ok(v) => { ctx.regs[flag] = RegVal::I1(true); ctx.regs[dst] = RegVal::I64(v); @@ -655,7 +779,7 @@ fn compile_inst(p: &Program, inst: &Inst, slots: &HashMap) -> InstFn { let (flag, dst, a) = (sl(slots, flag), sl(slots, dst), sl(slots, a)); Box::new(move |ctx| { let s = ctx.arena.get(as_str(ctx.regs[a])); - match s.parse::() { + match s.trim_ascii().parse::() { Ok(v) => { ctx.regs[flag] = RegVal::I1(true); ctx.regs[dst] = RegVal::F64(v); @@ -676,6 +800,91 @@ fn compile_inst(p: &Program, inst: &Inst, slots: &HashMap) -> InstFn { Ok(()) }) } + Inst::Str1 { op, dst, a } => { + let (dst, a) = (sl(slots, dst), sl(slots, a)); + // DuckDB uses SIMPLE (1:1) case maps; casemap.rs carries the + // measured exception table over Rust's full maps. + let map: fn(char) -> char = match op { + StrOp1::Upper => super::casemap::simple_upper, + StrOp1::Lower => super::casemap::simple_lower, + }; + Box::new(move |ctx| { + let s = ctx.arena.get(as_str(ctx.regs[a])); + let out: String = s.chars().map(map).collect(); + ctx.regs[dst] = RegVal::Str(ctx.arena.push_str(&out)); + Ok(()) + }) + } + Inst::Strim { + side, + dst, + a, + chars, + } => { + let (dst, a, chars) = (sl(slots, dst), sl(slots, a), sl(slots, chars)); + Box::new(move |ctx| { + let set: Vec = ctx.arena.get(as_str(ctx.regs[chars])).chars().collect(); + let s = ctx.arena.get(as_str(ctx.regs[a])); + let hit = |c: char| set.contains(&c); + let t = match side { + TrimSide::Both => s.trim_matches(hit), + TrimSide::Lead => s.trim_start_matches(hit), + TrimSide::Trail => s.trim_end_matches(hit), + } + .to_owned(); + ctx.regs[dst] = RegVal::Str(ctx.arena.push_str(&t)); + Ok(()) + }) + } + Inst::Ssubstr { dst, a, start, len } => { + let (dst, a) = (sl(slots, dst), sl(slots, a)); + let start = sl(slots, start); + let len = len.map(|l| sl(slots, l)); + Box::new(move |ctx| { + let st = as_i64(ctx.regs[start]); + if !substr_range_ok(st) { + return Err(Trap( + "substring offset outside of supported range".to_string(), + )); + } + let ln = match len { + Some(l) => { + let v = as_i64(ctx.regs[l]); + if !substr_range_ok(v) { + return Err(Trap( + "substring length outside of supported range".to_string(), + )); + } + Some(v) + } + None => None, + }; + let s = ctx.arena.get(as_str(ctx.regs[a])); + let out = substr_window(s, st, ln); + ctx.regs[dst] = RegVal::Str(ctx.arena.push_str(&out)); + Ok(()) + }) + } + Inst::Num1 { op, dst, a } => { + let (dst, a) = (sl(slots, dst), sl(slots, a)); + match op { + NumOp1::Iabs => Box::new(move |ctx| match as_i64(ctx.regs[a]).checked_abs() { + Some(v) => { + ctx.regs[dst] = RegVal::I64(v); + Ok(()) + } + None => Err(Trap("i64 overflow in iabs".to_string())), + }), + NumOp1::Fabs => Box::new(move |ctx| { + ctx.regs[dst] = RegVal::F64(as_f64(ctx.regs[a]).abs()); + Ok(()) + }), + NumOp1::Fround => Box::new(move |ctx| { + ctx.regs[dst] = RegVal::F64(as_f64(ctx.regs[a]).round()); + Ok(()) + }), + } + } Inst::Load { dst, col } => { let (dst, col) = (sl(slots, dst), col as usize); Box::new(move |ctx| { @@ -717,7 +926,12 @@ fn compile_inst(p: &Program, inst: &Inst, slots: &HashMap) -> InstFn { Ok(()) }) } - Inst::Probe { static_id, hit, dsts, keys } => { + Inst::Probe { + static_id, + hit, + dsts, + keys, + } => { let static_id = static_id as usize; let hit = sl(slots, hit); let dsts: Vec = dsts.iter().map(|d| sl(slots, *d)).collect(); @@ -762,7 +976,11 @@ fn compile_inst(p: &Program, inst: &Inst, slots: &HashMap) -> InstFn { Ok(()) }) } - Inst::SloadOpt { static_id, flag, dst } => { + Inst::SloadOpt { + static_id, + flag, + dst, + } => { let (static_id, flag, dst) = (static_id as usize, sl(slots, flag), sl(slots, dst)); let ty = match &p.statics[static_id] { StaticTy::Scalar(ct) => ct.ty, @@ -793,7 +1011,7 @@ fn cmp_key(stored: &[KeyBits], key_regs: &[usize], ctx: &Ctx<'_>) -> std::cmp::O let ord = match (kb, ctx.regs[*reg]) { (KeyBits::I1(s), RegVal::I1(v)) => s.cmp(&v), (KeyBits::I64(s), RegVal::I64(v)) => s.cmp(&v), - (KeyBits::F64(s), RegVal::F64(v)) => s.cmp(&v.to_bits()), + (KeyBits::F64(s), RegVal::F64(v)) => s.cmp(&super::canon_f64_bits(v)), (KeyBits::Str(s), RegVal::Str(v)) => s.as_str().cmp(ctx.arena.get(v)), _ => unreachable!("probe key types checked at compile"), }; diff --git a/src/specializer/exec/mod.rs b/src/specializer/exec/mod.rs index 0c4bb66..822f438 100644 --- a/src/specializer/exec/mod.rs +++ b/src/specializer/exec/mod.rs @@ -10,11 +10,15 @@ //! backend actually shares it — one home per concept until two consumers //! exist. +pub mod casemap; pub mod interp; #[cfg(test)] mod tests; +#[cfg(test)] +pub mod testutil; + use super::ir::Ty; /// A runtime error that aborts the whole call (division by zero, integer @@ -102,7 +106,10 @@ impl Arena { let off = self.0.len(); self.0.extend_from_within(a.off..a.off + a.len); self.0.extend_from_within(b.off..b.off + b.len); - StrRef { off, len: a.len + b.len } + StrRef { + off, + len: a.len + b.len, + } } pub fn get(&self, r: StrRef) -> &str { @@ -133,9 +140,11 @@ impl ScalarVal { } } -/// One component of a map-static key. F64 keys compare and match by bit -/// pattern (total, deterministic; NaN == NaN by bits) — an internal -/// convention of the prepared structure, not a SQL semantics statement. +/// One component of a map-static key. F64 keys compare and match by +/// *canonical* bit pattern: compile rewrites build-side keys and the probe +/// canonicalizes the searched value with [`canon_f64_bits`], so `-0.0` +/// matches `0.0` and every NaN is one key class — which is what DuckDB's +/// `=` does for doubles (NaN equals itself there). #[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)] pub enum KeyBits { I1(bool), @@ -155,10 +164,38 @@ impl KeyBits { } } +/// DuckDB's DOUBLE comparison order (measured 1.5.5): IEEE except that NaN +/// equals NaN and sorts above everything (`nan > inf` is TRUE); zeros are +/// equal (`-0.0 = 0.0`). NOT Rust `total_cmp` (which orders -0.0 < 0.0). +pub fn duck_fcmp(x: f64, y: f64) -> std::cmp::Ordering { + match (x.is_nan(), y.is_nan()) { + (true, true) => std::cmp::Ordering::Equal, + (true, false) => std::cmp::Ordering::Greater, + (false, true) => std::cmp::Ordering::Less, + (false, false) => x.partial_cmp(&y).expect("no NaN on either side"), + } +} + +/// The canonical key bits of an f64: all NaNs collapse to the one Rust +/// `f64::NAN` payload, `-0.0` collapses to `+0.0`. Everything else is +/// already unique per bit pattern. +pub fn canon_f64_bits(x: f64) -> u64 { + if x.is_nan() { + f64::NAN.to_bits() + } else if x == 0.0 { + 0f64.to_bits() + } else { + x.to_bits() + } +} + /// Runtime payload for a static structure, supplied to `compile` alongside /// the program and type-checked against its `StaticTy` declaration. pub enum StaticData { - Scalar { valid: bool, val: ScalarVal }, + Scalar { + valid: bool, + val: ScalarVal, + }, /// Entries are sorted + deduped at compile into a probe table; the /// binary-search probe is allocation-free (design doc: how a map is /// materialized is a backend decision — the oracle picks the simplest diff --git a/src/specializer/exec/tests.rs b/src/specializer/exec/tests.rs index 378f3fc..93d0478 100644 --- a/src/specializer/exec/tests.rs +++ b/src/specializer/exec/tests.rs @@ -7,8 +7,9 @@ use std::alloc::{GlobalAlloc, Layout, System}; use std::cell::Cell; use super::super::ir::{fixtures, gen, parse::parse, verify::verify, Program, StaticTy, Ty}; -use super::interp::{compile, CompileError, InterpFn}; -use super::{Batch, ColData, KeyBits, OutCol, RunState, ScalarVal, StaticData, Trap}; +use super::interp::{compile, CompileError}; +use super::testutil::{batch, built, c_f64, c_i1, c_i64, c_str, rows, run_snapshot}; +use super::{Batch, ColData, KeyBits, OutCol, ScalarVal, StaticData, Trap}; // ------------------------------------------------- counting allocator -- // Thread-local counter so parallel tests don't disturb each other; const- @@ -52,83 +53,6 @@ static GA: CountingAlloc = CountingAlloc; // ------------------------------------------------------------- helpers -- -fn c_i1(vals: &[Option]) -> ColData { - ColData::I1 { - valid: vals.iter().map(|v| v.is_some()).collect(), - data: vals.iter().map(|v| v.unwrap_or(false)).collect(), - } -} - -fn c_i64(vals: &[Option]) -> ColData { - ColData::I64 { - valid: vals.iter().map(|v| v.is_some()).collect(), - data: vals.iter().map(|v| v.unwrap_or(0)).collect(), - } -} - -fn c_f64(vals: &[Option]) -> ColData { - ColData::F64 { - valid: vals.iter().map(|v| v.is_some()).collect(), - data: vals.iter().map(|v| v.unwrap_or(0.0)).collect(), - } -} - -fn c_str(vals: &[Option<&str>]) -> ColData { - ColData::Str { - valid: vals.iter().map(|v| v.is_some()).collect(), - data: vals.iter().map(|v| v.unwrap_or("").to_string()).collect(), - } -} - -fn batch(rows: usize, cols: Vec) -> Batch { - Batch { rows, cols } -} - -fn built(text: &str) -> Program { - let p = parse(text).expect("fixture parses"); - verify(&p).expect("fixture verifies"); - p -} - -/// Snapshot the output as strings, masking NULL payloads (a NULL's payload -/// is meaningless downstream by contract). Allocates — test side only. -fn snapshot(st: &RunState) -> Vec> { - let ncols = st.out.len(); - let nrows = st.out.first().map(|c| c.len()).unwrap_or(0); - (0..nrows) - .map(|r| { - (0..ncols) - .map(|c| match &st.out[c] { - OutCol::I1(v) => render(v[r].0, format!("{}", v[r].1)), - OutCol::I64(v) => render(v[r].0, format!("{}", v[r].1)), - OutCol::F64(v) => render(v[r].0, format!("{:?}", v[r].1)), - OutCol::Str(v) => render(v[r].0, st.arena.get(v[r].1).to_string()), - }) - .collect() - }) - .collect() -} - -fn render(valid: bool, s: String) -> String { - if valid { - s - } else { - "NULL".to_string() - } -} - -fn run_snapshot(f: &InterpFn, input: &Batch) -> Result>, Trap> { - let mut st = f.new_state(); - f.run(input, &mut st)?; - Ok(snapshot(&st)) -} - -fn rows(v: &[&[&str]]) -> Vec> { - v.iter() - .map(|r| r.iter().map(|s| s.to_string()).collect()) - .collect() -} - // ------------------------------------------------------------ fixtures -- #[test] @@ -168,11 +92,7 @@ fn case_diamond_fixture_executes() { let input = batch(3, vec![c_i64(&[Some(31), Some(30), Some(5)])]); assert_eq!( run_snapshot(&f, &input).unwrap(), - rows(&[ - &["old", "false"], - &["old", "false"], - &["young", "false"], - ]) + rows(&[&["old", "false"], &["old", "false"], &["young", "false"],]) ); } @@ -181,7 +101,10 @@ fn casts_fixture_executes_and_traps() { let p = built(fixtures::CASTS); let statics = |snd: StaticData| { vec![ - StaticData::Scalar { valid: true, val: ScalarVal::F64(2.5) }, + StaticData::Scalar { + valid: true, + val: ScalarVal::F64(2.5), + }, snd, ] }; @@ -189,7 +112,10 @@ fn casts_fixture_executes_and_traps() { // scmp.eq("2.5:", ":") = false -> ":". let f = compile( &p, - statics(StaticData::Scalar { valid: false, val: ScalarVal::I64(0) }), + statics(StaticData::Scalar { + valid: false, + val: ScalarVal::I64(0), + }), ) .unwrap(); let input = batch(1, vec![c_str(&[Some("12")])]); @@ -198,7 +124,10 @@ fn casts_fixture_executes_and_traps() { // @1 = 7: the select picks the static instead. let f7 = compile( &p, - statics(StaticData::Scalar { valid: true, val: ScalarVal::I64(7) }), + statics(StaticData::Scalar { + valid: true, + val: ScalarVal::I64(7), + }), ) .unwrap(); assert_eq!(run_snapshot(&f7, &input).unwrap(), rows(&[&["7", ":"]])); @@ -253,7 +182,10 @@ entry: }"#, ) .unwrap(); - assert!(verify(&p).is_err(), "precondition: program must be unverifiable"); + assert!( + verify(&p).is_err(), + "precondition: program must be unverifiable" + ); match compile(&p, vec![]) { Err(CompileError::Verify(errs)) => assert!(!errs.is_empty()), Err(CompileError::Static(m)) => panic!("wrong error kind: {m}"), @@ -267,7 +199,10 @@ fn rejects_mismatched_statics() { for (data, needle) in [ (vec![], "declares 1 static(s), 0 provided"), ( - vec![StaticData::Scalar { valid: true, val: ScalarVal::F64(1.0) }], + vec![StaticData::Scalar { + valid: true, + val: ScalarVal::F64(1.0), + }], "declared map, got scalar", ), ( @@ -303,9 +238,20 @@ fn rejects_input_shape_mismatch() { let wrong_ty = batch(1, vec![c_i64(&[Some(1)])]); assert!(f.run(&wrong_ty, &mut st).unwrap_err().0.contains("is i64")); let wrong_count = batch(1, vec![]); - assert!(f.run(&wrong_count, &mut st).unwrap_err().0.contains("0 column(s)")); - let wrong_len = Batch { rows: 2, cols: vec![c_f64(&[Some(1.0)])] }; - assert!(f.run(&wrong_len, &mut st).unwrap_err().0.contains("1 row(s)")); + assert!(f + .run(&wrong_count, &mut st) + .unwrap_err() + .0 + .contains("0 column(s)")); + let wrong_len = Batch { + rows: 2, + cols: vec![c_f64(&[Some(1.0)])], + }; + assert!(f + .run(&wrong_len, &mut st) + .unwrap_err() + .0 + .contains("1 row(s)")); } // ---------------------------------------------------------- allocation -- @@ -370,7 +316,11 @@ entry: match &st.out[1] { OutCol::Str(v) => { assert!(!v[0].0); - assert_eq!(st.arena.get(v[0].1), "", "payload must be the empty default"); + assert_eq!( + st.arena.get(v[0].1), + "", + "payload must be the empty default" + ); } _ => unreachable!(), } @@ -392,7 +342,10 @@ entry: let mut st = f.new_state(); let bad = Batch { rows: 1, - cols: vec![ColData::I64 { valid: vec![], data: vec![7] }], + cols: vec![ColData::I64 { + valid: vec![], + data: vec![7], + }], }; let err = f.run(&bad, &mut st).unwrap_err(); assert!(err.0.contains("validity vector"), "wrong trap: {}", err.0); @@ -406,19 +359,35 @@ fn sparse_value_ids_use_dense_register_slots() { statics: vec![], name: "sparse".into(), in_cols: vec![], - out_cols: vec![Col { name: "o".into(), ty: ColTy { ty: Ty::I64, nullable: false } }], + out_cols: vec![Col { + name: "o".into(), + ty: ColTy { + ty: Ty::I64, + nullable: false, + }, + }], blocks: vec![Block { params: vec![], insts: vec![ - Inst::Const { dst: Value(9_999_999), lit: Lit::I64(5) }, - Inst::Store { col: 0, val: Value(9_999_999) }, + Inst::Const { + dst: Value(9_999_999), + lit: Lit::I64(5), + }, + Inst::Store { + col: 0, + val: Value(9_999_999), + }, ], term: Term::Emit, }], }; let f = compile(&p, vec![]).unwrap(); let st = f.new_state(); - assert!(st.regs.len() <= 4, "frame sized by ids, not defs: {}", st.regs.len()); + assert!( + st.regs.len() <= 4, + "frame sized by ids, not defs: {}", + st.regs.len() + ); } /// The emitted counter reports the row count even without reading columns. @@ -427,7 +396,11 @@ fn emitted_counts_rows() { let p = built(fixtures::FILTER); let f = compile(&p, vec![]).unwrap(); let mut st = f.new_state(); - f.run(&batch(3, vec![c_f64(&[Some(1.5), Some(-2.0), Some(0.0)])]), &mut st).unwrap(); + f.run( + &batch(3, vec![c_f64(&[Some(1.5), Some(-2.0), Some(0.0)])]), + &mut st, + ) + .unwrap(); assert_eq!(st.emitted, 1); } @@ -448,23 +421,49 @@ fn eval1(body: &str, out_col: &str) -> Result>, Trap> { #[test] fn pin_integer_overflow_and_division_traps() { for (expr, needle) in [ - (" %a = const.i64 9223372036854775807\n %b = const.i64 1\n %r = iadd %a, %b", "overflow in iadd"), - (" %a = const.i64 -9223372036854775808\n %b = const.i64 1\n %r = isub %a, %b", "overflow in isub"), - (" %a = const.i64 4611686018427387904\n %b = const.i64 4\n %r = imul %a, %b", "overflow in imul"), - (" %a = const.i64 1\n %b = const.i64 0\n %r = idiv %a, %b", "division by zero in idiv"), - (" %a = const.i64 1\n %b = const.i64 0\n %r = irem %a, %b", "division by zero in irem"), - (" %a = const.i64 -9223372036854775808\n %b = const.i64 -1\n %r = idiv %a, %b", "overflow in idiv"), - (" %a = const.i64 -9223372036854775808\n %b = const.i64 -1\n %r = irem %a, %b", "overflow in irem"), + ( + " %a = const.i64 9223372036854775807\n %b = const.i64 1\n %r = iadd %a, %b", + "overflow in iadd", + ), + ( + " %a = const.i64 -9223372036854775808\n %b = const.i64 1\n %r = isub %a, %b", + "overflow in isub", + ), + ( + " %a = const.i64 4611686018427387904\n %b = const.i64 4\n %r = imul %a, %b", + "overflow in imul", + ), + ( + " %a = const.i64 1\n %b = const.i64 0\n %r = idiv %a, %b", + "division by zero in idiv", + ), + ( + " %a = const.i64 1\n %b = const.i64 0\n %r = irem %a, %b", + "division by zero in irem", + ), + ( + " %a = const.i64 -9223372036854775808\n %b = const.i64 -1\n %r = idiv %a, %b", + "overflow in idiv", + ), + ( + " %a = const.i64 -9223372036854775808\n %b = const.i64 -1\n %r = irem %a, %b", + "overflow in irem", + ), ] { let body = format!("{expr}\n store out.o, %r"); let err = eval1(&body, "o: i64").unwrap_err(); - assert!(err.0.contains(needle), "expected '{needle}', got '{}'", err.0); + assert!( + err.0.contains(needle), + "expected '{needle}', got '{}'", + err.0 + ); } } #[test] fn pin_fcmp_nan_ordering() { - // Every predicate involving NaN is false except ne. + // DuckDB DOUBLE order (measured 1.5.5, stretch-4 pins): NaN sorts ABOVE + // everything, so NaN vs 1.0 is gt/ge/ne — not the IEEE all-false. let body = " %n = const.f64 nan\n %x = const.f64 1.0\n\ \x20 %eq = fcmp.eq %n, %x\n %ne = fcmp.ne %n, %x\n\ \x20 %lt = fcmp.lt %n, %x\n %le = fcmp.le %n, %x\n\ @@ -472,7 +471,122 @@ fn pin_fcmp_nan_ordering() { \x20 store out.eq, %eq\n store out.ne, %ne\n store out.lt, %lt\n\ \x20 store out.le, %le\n store out.gt, %gt\n store out.ge, %ge"; let got = eval1(body, "eq: i1, ne: i1, lt: i1, le: i1, gt: i1, ge: i1").unwrap(); - assert_eq!(got, rows(&[&["false", "true", "false", "false", "false", "false"]])); + assert_eq!( + got, + rows(&[&["false", "true", "false", "false", "true", "true"]]) + ); +} + +#[test] +fn pin_fcmp_nan_eq_nan_and_zero_order() { + // DuckDB DOUBLE order: nan = nan TRUE, nan > inf TRUE, -0.0 = 0.0 TRUE, + // -0.0 < 0.0 FALSE (measured 1.5.5). + let body = " %n = const.f64 nan\n %i = const.f64 inf\n\ + \x20 %nz = const.f64 -0.0\n %pz = const.f64 0.0\n\ + \x20 %a = fcmp.eq %n, %n\n %b = fcmp.gt %n, %i\n\ + \x20 %c = fcmp.eq %nz, %pz\n %d = fcmp.lt %nz, %pz\n\ + \x20 store out.a, %a\n store out.b, %b\n\ + \x20 store out.c, %c\n store out.d, %d"; + let got = eval1(body, "a: i1, b: i1, c: i1, d: i1").unwrap(); + assert_eq!(got, rows(&[&["true", "true", "true", "false"]])); +} + +#[test] +fn pin_frem_is_ieee_and_new_unaries() { + // frem: sign of the dividend, x % 0.0 = NaN, never traps. iabs traps on + // MIN (covered below); fabs clears the sign bit; fround is half away + // from zero (all measured DuckDB 1.5.5). + let body = " %a = const.f64 -5.5\n %b = const.f64 2.5\n %z = const.f64 0.0\n\ + \x20 %r = frem %a, %b\n %rz = frem %a, %z\n\ + \x20 %nz = const.f64 -0.0\n %ab = fabs %nz\n\ + \x20 %h = const.f64 -2.5\n %ro = fround %h\n\ + \x20 %i = const.i64 -5\n %ia = iabs %i\n\ + \x20 %s = const.str \" hi \"\n %sp = const.str \" \"\n\ + \x20 %t = strim.both %s, %sp\n %u = supper %t\n\ + \x20 %one = const.i64 1\n %sub = ssubstr %u, %one, %one\n\ + \x20 store out.r, %r\n store out.rz, %rz\n store out.ab, %ab\n\ + \x20 store out.ro, %ro\n store out.ia, %ia\n store out.sub, %sub"; + let got = eval1(body, "r: f64, rz: f64, ab: f64, ro: f64, ia: i64, sub: str").unwrap(); + assert_eq!(got, rows(&[&["-0.5", "NaN", "0.0", "-3.0", "5", "H"]])); +} + +#[test] +fn pin_iabs_min_traps() { + let body = " %m = const.i64 -9223372036854775808\n %a = iabs %m\n store out.o, %a"; + let err = eval1(body, "o: i64").unwrap_err(); + assert!(err.0.contains("overflow"), "got '{}'", err.0); +} + +#[test] +fn pin_ssubstr_window_arithmetic() { + // DuckDB virtual-window semantics (measured 1.5.5 + adversarial census): + // start 0 and negative starts map through n+start+1; a non-negative len + // runs forward, a negative len slices BACKWARDS from the resolved start; + // ssubstr.rest is the 2-arg "rest of the string" form. + for (start, len, expect) in [ + (2i64, Some(3i64), "ell"), + (0, Some(3), "he"), + (-2, None, "lo"), + (-6, Some(3), "hel"), + (-10, Some(8), "hello"), + (1, Some(0), ""), + (1, Some(-1), ""), + (3, Some(-2), "he"), + (2, Some(-1), "h"), + (6, Some(-5), "hello"), + (-2, Some(-3), "hel"), + (10, None, ""), + (0, None, "hello"), + (-4294967296, None, "hello"), + ] { + let op = match len { + Some(l) => format!(" %ln = const.i64 {l}\n %r = ssubstr %s, %st, %ln\n"), + None => " %r = ssubstr.rest %s, %st\n".to_string(), + }; + let body = + format!(" %s = const.str \"hello\"\n %st = const.i64 {start}\n{op} store out.o, %r"); + assert_eq!( + eval1(&body, "o: str").unwrap(), + rows(&[&[expect]]), + "substr('hello', {start}, {len:?})" + ); + } +} + +#[test] +fn pin_ssubstr_range_guards_trap() { + // DuckDB errors for offsets/lengths outside [-2^32, 2^32-1]; the 2-arg + // form has no length to guard. + for (start, len, needle) in [ + (4294967296i64, Some(2i64), "offset outside"), + (-4294967297, Some(2), "offset outside"), + (1, Some(4294967296), "length outside"), + (1, Some(-4294967297), "length outside"), + ] { + let op = match len { + Some(l) => format!(" %ln = const.i64 {l}\n %r = ssubstr %s, %st, %ln\n"), + None => " %r = ssubstr.rest %s, %st\n".to_string(), + }; + let body = + format!(" %s = const.str \"hello\"\n %st = const.i64 {start}\n{op} store out.o, %r"); + let err = eval1(&body, "o: str").unwrap_err(); + assert!( + err.0.contains(needle), + "({start}, {len:?}): got '{}'", + err.0 + ); + } + // Boundary values inside the guard execute. + for start in [4294967295i64, -4294967296] { + let body = format!( + " %s = const.str \"hello\"\n %st = const.i64 {start}\n\ + \x20 %r = ssubstr.rest %s, %st\n store out.o, %r" + ); + assert!( + eval1(&body, "o: str").is_ok(), + "start {start} should not trap" + ); + } } #[test] @@ -512,8 +626,15 @@ fn pin_ieee_flow_and_scmp_and_concat() { } #[test] -fn pin_stoi_exact_parse_no_trim() { - for (s, ok) in [(" 5", false), ("5 ", false), ("+5", true), ("0x10", false), ("", false)] { +fn pin_stoi_trims_whitespace_like_duckdb_cast() { + for (s, ok) in [ + (" 5", true), + ("5 ", true), + ("+5", true), + ("0x10", false), + ("", false), + (" ", false), + ] { let body = format!( " %s = const.str \"{s}\"\n %f, %v = stoi.opt %s\n store out.f, %f\n store out.v, %v" ); @@ -537,7 +658,10 @@ entry: let f = compile(&p, vec![]).unwrap(); let input = Batch { rows: 1, - cols: vec![ColData::I64 { valid: vec![false], data: vec![999] }], + cols: vec![ColData::I64 { + valid: vec![false], + data: vec![999], + }], }; assert_eq!(run_snapshot(&f, &input).unwrap(), rows(&[&["0"]])); } @@ -564,7 +688,11 @@ fn gen_statics(rng: &mut gen::Rng, p: &Program) -> Vec { val: gen_scalar(rng, ct.ty), }, StaticTy::Map { keys, values } => { - let n = if keys[0] == Ty::I1 { 2 } else { 1 + rng.below(3) as usize }; + let n = if keys[0] == Ty::I1 { + 2 + } else { + 1 + rng.below(3) as usize + }; let entries = (0..n) .map(|j| { let key: Vec = keys @@ -667,3 +795,26 @@ fn fuzz_generated_programs_execute_deterministically() { } } } + +#[test] +fn casemap_tables_sorted_and_marquee_pins() { + use super::casemap::{simple_lower, simple_upper, LOWER_EXCEPTIONS, UPPER_EXCEPTIONS}; + // binary_search precondition: strictly sorted by codepoint. + for table in [UPPER_EXCEPTIONS, LOWER_EXCEPTIONS] { + assert!( + table.windows(2).all(|w| w[0].0 < w[1].0), + "table not sorted" + ); + } + // The two exception classes (see scripts/gen_casemap.py): simple-vs-full + // divergence, and Unicode version skew where duckdb's utf8proc predates + // the case pair (identity there). + assert_eq!(simple_upper('ß'), 'ẞ'); + assert_eq!(simple_lower('İ'), 'i'); + assert_eq!(simple_upper('ᾀ'), 'ᾈ'); + assert_eq!(simple_upper('ƛ'), 'ƛ'); + // The fallback path stays exact where full mapping is 1:1. + assert_eq!(simple_upper('a'), 'A'); + assert_eq!(simple_lower('É'), 'é'); + assert_eq!(simple_upper('fi'), 'fi'); +} diff --git a/src/specializer/exec/testutil.rs b/src/specializer/exec/testutil.rs new file mode 100644 index 0000000..09f26d5 --- /dev/null +++ b/src/specializer/exec/testutil.rs @@ -0,0 +1,98 @@ +//! Shared test helpers: batch builders and output snapshots. cfg(test)-only. + +use super::super::ir::Program; +use super::interp::InterpFn; +use super::{Batch, ColData, OutCol, RunState, Trap}; + +pub fn c_i1(vals: &[Option]) -> ColData { + ColData::I1 { + valid: vals.iter().map(|v| v.is_some()).collect(), + data: vals.iter().map(|v| v.unwrap_or(false)).collect(), + } +} + +pub fn c_i64(vals: &[Option]) -> ColData { + ColData::I64 { + valid: vals.iter().map(|v| v.is_some()).collect(), + data: vals.iter().map(|v| v.unwrap_or(0)).collect(), + } +} + +pub fn c_f64(vals: &[Option]) -> ColData { + ColData::F64 { + valid: vals.iter().map(|v| v.is_some()).collect(), + data: vals.iter().map(|v| v.unwrap_or(0.0)).collect(), + } +} + +pub fn c_str(vals: &[Option<&str>]) -> ColData { + ColData::Str { + valid: vals.iter().map(|v| v.is_some()).collect(), + data: vals.iter().map(|v| v.unwrap_or("").to_string()).collect(), + } +} + +pub fn batch(rows: usize, cols: Vec) -> Batch { + Batch { rows, cols } +} + +/// Parse + verify a fixture text into a Program, panicking with context. +pub fn built(text: &str) -> Program { + let p = match super::super::ir::parse::parse(text) { + Ok(p) => p, + Err(e) => panic!( + "parse failed: {e} +--- +{text}" + ), + }; + if let Err(errs) = super::super::ir::verify::verify(&p) { + let msgs: Vec = errs.iter().map(|e| e.to_string()).collect(); + panic!( + "verify failed: {} +--- +{text}", + msgs.join("; ") + ); + } + p +} + +/// Snapshot the output as strings, masking NULL payloads (a NULL's payload +/// is meaningless downstream by contract). Allocates — test side only. +pub fn snapshot(st: &RunState) -> Vec> { + let ncols = st.out.len(); + let nrows = st.out.first().map(|c| c.len()).unwrap_or(0); + (0..nrows) + .map(|r| { + (0..ncols) + .map(|c| match &st.out[c] { + OutCol::I1(v) => render(v[r].0, format!("{}", v[r].1)), + OutCol::I64(v) => render(v[r].0, format!("{}", v[r].1)), + OutCol::F64(v) => render(v[r].0, format!("{:?}", v[r].1)), + OutCol::Str(v) => render(v[r].0, st.arena.get(v[r].1).to_string()), + }) + .collect() + }) + .collect() +} + +fn render(valid: bool, s: String) -> String { + if valid { + s + } else { + "NULL".to_string() + } +} + +pub fn run_snapshot(f: &InterpFn, input: &Batch) -> Result>, Trap> { + let mut st = f.new_state(); + f.run(input, &mut st)?; + Ok(snapshot(&st)) +} + +pub fn rows(v: &[&[&str]]) -> Vec> { + v.iter() + .map(|r| r.iter().map(|s| s.to_string()).collect()) + .collect() +} diff --git a/src/specializer/fold.rs b/src/specializer/fold.rs new file mode 100644 index 0000000..5400fbb --- /dev/null +++ b/src/specializer/fold.rs @@ -0,0 +1,226 @@ +//! Prepare-time scalar folding: the expression half of binding-time +//! analysis. Every all-constant subtree collapses to `Lit`/`NullOf` before +//! lowering, so a static computation never reaches the per-row IR. +//! +//! The folder is deliberately conservative — it folds ONLY when the result +//! provably equals what the interpreter would compute at run time: +//! * every operand must itself be a constant (`Lit`/`NullOf`); a constant +//! that merely dominates (`FALSE AND dynamic`) is never folded, because +//! dropping the dynamic side could drop its trap; +//! * an operation that would trap at run time (integer overflow, `% 0`) is +//! left unfolded — the trap stays a run-time trap, same timing as before; +//! * f64 arithmetic and comparisons mirror exec/interp.rs exactly (IEEE: +//! `x/0 = inf`, NaN compares false except `!=`); +//! * CASE and CAST nodes fold their children but never themselves — CAST +//! can trap and CASE guards branch traps, both wrong to evaluate eagerly. + +use super::ir::{CmpPred, Lit, Ty}; +use super::plan::{ArithOp, SExpr, SKind}; + +/// A constant operand: a payload or a typed NULL. +enum K { + Val(Lit), + Null, +} + +fn as_const(e: &SExpr) -> Option { + match &e.kind { + SKind::Lit(l) => Some(K::Val(l.clone())), + SKind::NullOf => Some(K::Null), + _ => None, + } +} + +fn lit(l: Lit, ty: Ty) -> SExpr { + SExpr { + kind: SKind::Lit(l), + ty, + nullable: false, + } +} + +fn null(ty: Ty) -> SExpr { + SExpr { + kind: SKind::NullOf, + ty, + nullable: true, + } +} + +/// Fold every all-constant subtree of `e`, bottom-up. +pub fn fold(e: SExpr) -> SExpr { + let SExpr { kind, ty, nullable } = e; + let e = |kind| SExpr { kind, ty, nullable }; + match kind { + SKind::Col(_) | SKind::StaticCol { .. } | SKind::Lit(_) | SKind::NullOf => e(kind), + SKind::IntToFloat(inner) => { + let inner = fold(*inner); + match as_const(&inner) { + Some(K::Val(Lit::I64(i))) => lit(Lit::F64(i as f64), ty), + Some(K::Null) => null(ty), + _ => e(SKind::IntToFloat(Box::new(inner))), + } + } + SKind::Not(inner) => { + let inner = fold(*inner); + match as_const(&inner) { + Some(K::Val(Lit::I1(b))) => lit(Lit::I1(!b), ty), + Some(K::Null) => null(ty), + _ => e(SKind::Not(Box::new(inner))), + } + } + SKind::IsNull { negated, inner } => { + let inner = fold(*inner); + match as_const(&inner) { + Some(K::Val(_)) => lit(Lit::I1(negated), ty), + Some(K::Null) => lit(Lit::I1(!negated), ty), + None => e(SKind::IsNull { + negated, + inner: Box::new(inner), + }), + } + } + SKind::Arith { op, a, b } => { + let (a, b) = (fold(*a), fold(*b)); + match (as_const(&a), as_const(&b)) { + (Some(K::Null), Some(_)) | (Some(_), Some(K::Null)) => null(ty), + (Some(K::Val(x)), Some(K::Val(y))) => match arith(op, &x, &y) { + Some(l) => lit(l, ty), + // Would trap at run time — keep the node, keep the trap. + None => e(SKind::Arith { + op, + a: Box::new(a), + b: Box::new(b), + }), + }, + _ => e(SKind::Arith { + op, + a: Box::new(a), + b: Box::new(b), + }), + } + } + SKind::Cmp { pred, a, b } => { + let (a, b) = (fold(*a), fold(*b)); + match (as_const(&a), as_const(&b)) { + (Some(K::Null), Some(_)) | (Some(_), Some(K::Null)) => null(ty), + (Some(K::Val(x)), Some(K::Val(y))) => lit(Lit::I1(cmp(pred, &x, &y)), ty), + _ => e(SKind::Cmp { + pred, + a: Box::new(a), + b: Box::new(b), + }), + } + } + SKind::And { a, b } => kleene(true, *a, *b, ty, nullable), + SKind::Or { a, b } => kleene(false, *a, *b, ty, nullable), + SKind::Case { arms, default } => e(SKind::Case { + arms: arms.into_iter().map(|(c, r)| (fold(c), fold(r))).collect(), + default: default.map(|d| Box::new(fold(*d))), + }), + SKind::Cast { inner, trying } => e(SKind::Cast { + inner: Box::new(fold(*inner)), + trying, + }), + // Builtin nodes fold children only (ponytail: constant upper('a') + // etc. can fold later if a corpus query ever cares). + SKind::StrCase { upper, a } => e(SKind::StrCase { + upper, + a: Box::new(fold(*a)), + }), + SKind::Trim { side, a, chars } => e(SKind::Trim { + side, + a: Box::new(fold(*a)), + chars: Box::new(fold(*chars)), + }), + SKind::Substr { a, start, len } => e(SKind::Substr { + a: Box::new(fold(*a)), + start: Box::new(fold(*start)), + len: len.map(|l| Box::new(fold(*l))), + }), + SKind::Abs(a) => e(SKind::Abs(Box::new(fold(*a)))), + SKind::Round(a) => e(SKind::Round(Box::new(fold(*a)))), + SKind::Concat { a, b } => e(SKind::Concat { + a: Box::new(fold(*a)), + b: Box::new(fold(*b)), + }), + } +} + +/// Kleene AND/OR, folded only when BOTH sides are constants — the full +/// three-valued table, including `FALSE AND NULL = FALSE`. +fn kleene(is_and: bool, a: SExpr, b: SExpr, ty: Ty, nullable: bool) -> SExpr { + let (a, b) = (fold(a), fold(b)); + let tri = |e: &SExpr| match as_const(e) { + Some(K::Val(Lit::I1(v))) => Some(Some(v)), + Some(K::Null) => Some(None), + _ => None, + }; + if let (Some(x), Some(y)) = (tri(&a), tri(&b)) { + let r = if is_and { + match (x, y) { + (Some(false), _) | (_, Some(false)) => Some(false), + (Some(true), Some(true)) => Some(true), + _ => None, + } + } else { + match (x, y) { + (Some(true), _) | (_, Some(true)) => Some(true), + (Some(false), Some(false)) => Some(false), + _ => None, + } + }; + return match r { + Some(v) => lit(Lit::I1(v), ty), + None => null(ty), + }; + } + let (a, b) = (Box::new(a), Box::new(b)); + let kind = if is_and { + SKind::And { a, b } + } else { + SKind::Or { a, b } + }; + SExpr { kind, ty, nullable } +} + +/// `None` = the interpreter would trap on this — do not fold. +fn arith(op: ArithOp, a: &Lit, b: &Lit) -> Option { + match (a, b) { + (Lit::I64(x), Lit::I64(y)) => match op { + ArithOp::Add => x.checked_add(*y).map(Lit::I64), + ArithOp::Sub => x.checked_sub(*y).map(Lit::I64), + ArithOp::Mul => x.checked_mul(*y).map(Lit::I64), + ArithOp::Rem => x.checked_rem(*y).map(Lit::I64), + ArithOp::Div => unreachable!("/ is promoted to f64 by the frontend"), + }, + (Lit::F64(x), Lit::F64(y)) => Some(Lit::F64(match op { + ArithOp::Add => x + y, + ArithOp::Sub => x - y, + ArithOp::Mul => x * y, + ArithOp::Div => x / y, + // IEEE, exactly as exec/interp.rs: x % 0.0 is NaN, never traps. + ArithOp::Rem => x % y, + })), + _ => unreachable!("operands are promoted to a common type at bind"), + } +} + +fn cmp(pred: CmpPred, a: &Lit, b: &Lit) -> bool { + use std::cmp::Ordering; + let ord = |o: Ordering| match pred { + CmpPred::Eq => o == Ordering::Equal, + CmpPred::Ne => o != Ordering::Equal, + CmpPred::Lt => o == Ordering::Less, + CmpPred::Le => o != Ordering::Greater, + CmpPred::Gt => o == Ordering::Greater, + CmpPred::Ge => o != Ordering::Less, + }; + match (a, b) { + (Lit::I64(x), Lit::I64(y)) => ord(x.cmp(y)), + (Lit::Str(x), Lit::Str(y)) => ord(x.cmp(y)), + // DuckDB DOUBLE order, exactly as exec/interp.rs computes it. + (Lit::F64(x), Lit::F64(y)) => ord(super::exec::duck_fcmp(*x, *y)), + _ => unreachable!("cmp operands share a type; i1 cmp rejected at bind"), + } +} diff --git a/src/specializer/frontend.rs b/src/specializer/frontend.rs new file mode 100644 index 0000000..32f728b --- /dev/null +++ b/src/specializer/frontend.rs @@ -0,0 +1,1531 @@ +//! Frontend: SQL text -> bound, typed relational IR. Parsing is sqlparser's +//! DuckDB dialect; binding and type derivation follow DuckDB semantics as +//! measured (see plan.rs notes and the pins in exec/interp.rs). +//! +//! Error discipline (the corpus three-outcome contract depends on it): +//! * [`PrepareError::Unsupported`] — the construct is real SQL we don't do +//! YET; the message names it. Corpus replay counts these as clean. +//! * [`PrepareError::Bind`] — the query is wrong against this schema +//! (unknown column, type mismatch). Never used for missing features. +//! +//! Identifier semantics: DuckDB matches case-insensitively and preserves +//! spelling — `SELECT AGE` binds a column named `age` and the output column +//! is spelled `AGE`. +//! +//! NULL literal: typed by context (the other operand, the CASE unification, +//! the CAST target). A bare `SELECT NULL` has no context and stays +//! unsupported. +//! +//! Known v0 divergences, deliberate: DuckDB types `1.5` as DECIMAL(2,1); we +//! map decimal literals to f64. Integer-ish CAST targets (including HUGEINT) +//! all collapse to i64. + +use sqlparser::ast::{ + BinaryOperator, CastKind, Expr as SqlExpr, JoinConstraint, JoinOperator, SelectItem, SetExpr, + Statement, TableFactor, UnaryOperator, Value as SqlValue, +}; +use sqlparser::dialect::DuckDbDialect; +use sqlparser::parser::Parser; + +use super::fold::fold; +use super::ir::{CmpPred, Col, Lit, TrimSide, Ty}; +use super::plan::{ArithOp, JoinKind, JoinSpec, Rel, SExpr, SKind, StaticTable}; + +#[derive(Debug, PartialEq, Eq)] +pub enum PrepareError { + Parse(String), + /// Real SQL, not lowered yet — names the construct (clean-unsupported). + Unsupported(String), + /// Wrong against this schema/type system. + Bind(String), + /// Lowering produced unverifiable IR — always a bug in the specializer. + Internal(String), +} + +impl std::fmt::Display for PrepareError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + PrepareError::Parse(m) => write!(f, "parse error: {m}"), + PrepareError::Unsupported(m) => write!(f, "unsupported: {m}"), + PrepareError::Bind(m) => write!(f, "bind error: {m}"), + PrepareError::Internal(m) => write!(f, "internal specializer bug: {m}"), + } + } +} + +fn unsup(what: impl Into) -> PrepareError { + PrepareError::Unsupported(what.into()) +} + +/// SQL text + the dynamic table's name/schema + the static-table catalog -> +/// bound relational tree, the equi-joins in FROM order, and the derived +/// output schema. +pub fn frontend( + sql: &str, + this_name: &str, + in_cols: &[Col], + statics: &[StaticTable], +) -> Result<(Rel, Vec, Vec), PrepareError> { + let statements = Parser::parse_sql(&DuckDbDialect {}, sql) + .map_err(|e| PrepareError::Parse(e.to_string()))?; + let [statement] = statements.as_slice() else { + return Err(unsup("multiple SQL statements")); + }; + let query = match statement { + Statement::Query(q) => q, + other => return Err(unsup(format!("statement kind: {other}"))), + }; + if query.with.is_some() { + return Err(unsup("WITH / common table expressions")); + } + if query.order_by.is_some() { + return Err(unsup("ORDER BY")); + } + if query.limit_clause.is_some() { + return Err(unsup("LIMIT/OFFSET")); + } + let select = match query.body.as_ref() { + SetExpr::Select(s) => s.as_ref(), + SetExpr::SetOperation { .. } => return Err(unsup("UNION/INTERSECT/EXCEPT")), + other => return Err(unsup(format!("query body: {other}"))), + }; + if select.distinct.is_some() { + return Err(unsup("DISTINCT")); + } + let grouped = match &select.group_by { + sqlparser::ast::GroupByExpr::Expressions(exprs, modifiers) => { + !exprs.is_empty() || !modifiers.is_empty() + } + sqlparser::ast::GroupByExpr::All(_) => true, + }; + if grouped || select.having.is_some() { + return Err(unsup("GROUP BY / HAVING / aggregation")); + } + + let (binder, joins) = bind_from(select, this_name, in_cols, statics)?; + + let mut rel = Rel::Scan; + if let Some(pred) = &select.selection { + let pred = fold(bool_context(binder.expr(pred)?, "WHERE predicate")?); + rel = Rel::Filter { + input: Box::new(rel), + pred, + }; + } + + let mut out_cols = Vec::new(); + let mut exprs = Vec::new(); + for item in &select.projection { + let (name, e) = match item { + SelectItem::UnnamedExpr(e) => (default_name(e), fold(binder.expr(e)?)), + SelectItem::ExprWithAlias { expr, alias } => { + (alias.value.clone(), fold(binder.expr(expr)?)) + } + SelectItem::Wildcard(_) | SelectItem::QualifiedWildcard(..) => { + return Err(unsup("SELECT * (star expansion)")) + } + SelectItem::ExprWithAliases { .. } => return Err(unsup("multi-alias SELECT item")), + }; + if out_cols.iter().any(|c: &Col| c.name == name) { + // DuckDB allows duplicate output names; our IR requires unique + // columns. Rare in real queries — punt cleanly for now. + return Err(unsup(format!("duplicate output column name '{name}'"))); + } + out_cols.push(Col { + name, + ty: super::ir::ColTy { + ty: e.ty, + nullable: e.nullable, + }, + }); + exprs.push(e); + } + if exprs.is_empty() { + return Err(PrepareError::Bind("SELECT list is empty".to_string())); + } + + let named = out_cols + .iter() + .map(|c| c.name.clone()) + .zip(exprs) + .collect::>(); + Ok(( + Rel::Project { + input: Box::new(rel), + exprs: named, + }, + joins, + out_cols, + )) +} + +/// Parse and bind the FROM clause: the dynamic table, then zero or more +/// equi-joins to static tables. Returns the fully-scoped binder (every join +/// visible) and the join specs in FROM order. +fn bind_from<'a>( + select: &sqlparser::ast::Select, + this_name: &str, + in_cols: &'a [Col], + statics: &'a [StaticTable], +) -> Result<(Binder<'a>, Vec), PrepareError> { + let [table] = select.from.as_slice() else { + return Err(match select.from.len() { + 0 => unsup("FROM-less SELECT"), + _ => unsup("multiple FROM relations (comma join)"), + }); + }; + let dyn_name = match &table.relation { + TableFactor::Table { name, alias, .. } => { + if alias.is_some() { + return Err(unsup("alias on the dynamic table")); + } + let n = name.to_string(); + if !n.eq_ignore_ascii_case(this_name) { + return Err(unsup(format!( + "table '{n}' as the driving relation (must be the dynamic table '{this_name}')" + ))); + } + n + } + other => return Err(unsup(format!("FROM {other}"))), + }; + + let mut binder = Binder { + this_name: dyn_name, + in_cols, + joins: Vec::new(), + select_aliases: select + .projection + .iter() + .filter_map(|item| match item { + SelectItem::ExprWithAlias { alias, .. } => Some(alias.value.clone()), + _ => None, + }) + .collect(), + }; + let mut specs: Vec = Vec::new(); + + for join in &table.joins { + let (kind, constraint) = match &join.join_operator { + JoinOperator::Join(c) | JoinOperator::Inner(c) => (JoinKind::Inner, c), + JoinOperator::Left(c) | JoinOperator::LeftOuter(c) => (JoinKind::Left, c), + other => return Err(unsup(format!("join type {other:?}"))), + }; + let on = match constraint { + JoinConstraint::On(e) => e, + JoinConstraint::Using(_) => return Err(unsup("JOIN USING")), + JoinConstraint::Natural => return Err(unsup("NATURAL JOIN")), + JoinConstraint::None => return Err(unsup("JOIN without ON (cross join)")), + }; + let (raw_name, scope_name) = match &join.relation { + TableFactor::Table { name, alias, .. } => { + let n = name.to_string(); + let s = alias + .as_ref() + .map(|a| a.name.value.clone()) + .unwrap_or_else(|| n.clone()); + (n, s) + } + other => return Err(unsup(format!("JOIN {other}"))), + }; + if raw_name.eq_ignore_ascii_case(this_name) { + return Err(unsup("joining the dynamic table to itself")); + } + let mut table_idx = None; + for (i, st) in statics.iter().enumerate() { + if st.name.eq_ignore_ascii_case(&raw_name) { + if table_idx.is_some() { + return Err(PrepareError::Bind(format!( + "ambiguous static table '{raw_name}'" + ))); + } + table_idx = Some(i); + } + } + let Some(table_idx) = table_idx else { + return Err(PrepareError::Bind(format!( + "table '{raw_name}' was not provided as a static table" + ))); + }; + if binder.this_name.eq_ignore_ascii_case(&scope_name) + || binder + .joins + .iter() + .any(|j| j.name.eq_ignore_ascii_case(&scope_name)) + { + return Err(PrepareError::Bind(format!( + "duplicate table name '{scope_name}' in FROM" + ))); + } + + let st = &statics[table_idx]; + let (keys, key_cols) = bind_on(&binder, st, &scope_name, on)?; + let val_cols: Vec = (0..st.cols.len() as u32) + .filter(|c| !key_cols.contains(c)) + .collect(); + if val_cols.is_empty() { + return Err(unsup(format!( + "join to '{raw_name}' where every column is a key (no value columns)" + ))); + } + + binder.joins.push(ScopeJoin { + name: scope_name, + table: st, + kind, + key_cols: key_cols.clone(), + val_cols: val_cols.clone(), + }); + specs.push(JoinSpec { + table: table_idx, + kind, + keys, + key_cols, + val_cols, + }); + } + Ok((binder, specs)) +} + +/// Bind a JOIN ... ON condition: a conjunction of ` = ` +/// equalities. Returns the dynamic-side key expressions (promoted to the +/// map's key types) and the static key columns they match, aligned. +fn bind_on( + binder: &Binder<'_>, + st: &StaticTable, + scope_name: &str, + on: &SqlExpr, +) -> Result<(Vec, Vec), PrepareError> { + let mut conjuncts = Vec::new(); + collect_conjuncts(on, &mut conjuncts); + let mut keys = Vec::new(); + let mut key_cols = Vec::new(); + for c in conjuncts { + let SqlExpr::BinaryOp { + left, + op: BinaryOperator::Eq, + right, + } = c + else { + return Err(unsup(format!( + "JOIN ON condition '{c}' (only AND-ed equalities are supported)" + ))); + }; + let l = static_col_of(left, st, scope_name)?; + let r = static_col_of(right, st, scope_name)?; + let (col, dyn_side, static_side) = match (l, r) { + (Some(c), None) => (c, right.as_ref(), left.as_ref()), + (None, Some(c)) => (c, left.as_ref(), right.as_ref()), + (Some(_), Some(_)) => { + return Err(unsup(format!( + "JOIN ON '{c}': both sides are columns of '{scope_name}'" + ))) + } + (None, None) => { + return Err(unsup(format!( + "JOIN ON '{c}': neither side is a column of '{scope_name}'" + ))) + } + }; + // A bare identifier on the static side that also binds in the outer + // scope is ambiguous (DuckDB rejects it too). + if let SqlExpr::Identifier(id) = static_side { + if binder.column(&id.value).is_ok() { + return Err(PrepareError::Bind(format!( + "ambiguous column '{}' in JOIN ON (qualify it)", + id.value + ))); + } + } + let key = fold(binder.expr(dyn_side)?); + let col_ty = st.cols[col as usize].ty.ty; + let key = match (key.ty, col_ty) { + (a, b) if a == b => key, + (Ty::I64, Ty::F64) => promote_f64(key), + // Static-side ints promote at materialization: the map key type + // (the key expression's type) becomes F64 and the build side is + // converted while the probe table is built. + (Ty::F64, Ty::I64) => key, + (a, b) => { + return Err(PrepareError::Bind(format!( + "cannot join {} with {} (ON '{}')", + a.name(), + b.name(), + st.cols[col as usize].name + ))) + } + }; + keys.push(key); + key_cols.push(col); + } + Ok((keys, key_cols)) +} + +fn collect_conjuncts<'e>(e: &'e SqlExpr, out: &mut Vec<&'e SqlExpr>) { + match e { + SqlExpr::BinaryOp { + left, + op: BinaryOperator::And, + right, + } => { + collect_conjuncts(left, out); + collect_conjuncts(right, out); + } + SqlExpr::Nested(inner) => collect_conjuncts(inner, out), + other => out.push(other), + } +} + +/// Does `e` name a column of the static table being joined? Qualified form +/// matches on the join's scope name; a bare identifier matches if the table +/// has that column. +fn static_col_of( + e: &SqlExpr, + st: &StaticTable, + scope_name: &str, +) -> Result, PrepareError> { + let name = match e { + SqlExpr::Identifier(id) => &id.value, + SqlExpr::CompoundIdentifier(parts) => match parts.as_slice() { + [t, c] if t.value.eq_ignore_ascii_case(scope_name) => &c.value, + _ => return Ok(None), + }, + SqlExpr::Nested(inner) => return static_col_of(inner, st, scope_name), + _ => return Ok(None), + }; + let mut hit = None; + for (i, c) in st.cols.iter().enumerate() { + if c.name.eq_ignore_ascii_case(name) { + if hit.is_some() { + return Err(PrepareError::Bind(format!( + "ambiguous column '{name}' in static table '{}'", + st.name + ))); + } + hit = Some(i as u32); + } + } + // Qualified misses are errors; bare misses just mean "not the static + // side" — the caller will try binding it dynamically. + if hit.is_none() { + if let SqlExpr::CompoundIdentifier(_) = e { + return Err(PrepareError::Bind(format!( + "column '{name}' does not exist in '{scope_name}'" + ))); + } + } + Ok(hit) +} + +/// DuckDB names an unaliased projection after the identifier it selects +/// (spelling preserved), else after the expression's text. +fn default_name(e: &SqlExpr) -> String { + match e { + SqlExpr::Identifier(ident) => ident.value.clone(), + SqlExpr::CompoundIdentifier(parts) if !parts.is_empty() => { + parts.last().unwrap().value.clone() + } + other => other.to_string(), + } +} + +/// One joined static table in scope: how it is named, which of its columns +/// are probe values (bindable) vs keys (ON-clause only). +struct ScopeJoin<'a> { + name: String, + table: &'a StaticTable, + kind: JoinKind, + key_cols: Vec, + val_cols: Vec, +} + +struct Binder<'a> { + /// The dynamic table's name as spelled in FROM. + this_name: String, + in_cols: &'a [Col], + joins: Vec>, + /// SELECT-list aliases, for cleanly rejecting DuckDB's lateral-alias + /// extension (an alias referenced inside WHERE) as unsupported. + select_aliases: Vec, +} + +impl Binder<'_> { + /// Bind an expression that must have a definite type on its own — a bare + /// NULL literal here is unsupported (no context to type it). + fn expr(&self, e: &SqlExpr) -> Result { + self.expr_or_null(e)? + .ok_or_else(|| unsup("bare NULL literal without a typing context")) + } + + /// Like `expr`, but a bare NULL literal comes back as `None` for the + /// caller to type from context. + fn expr_or_null(&self, e: &SqlExpr) -> Result, PrepareError> { + match e { + SqlExpr::Value(v) if matches!(v.value, SqlValue::Null) => Ok(None), + SqlExpr::Nested(inner) => self.expr_or_null(inner), + other => self.bind(other).map(Some), + } + } + + fn bind(&self, e: &SqlExpr) -> Result { + match e { + SqlExpr::Identifier(ident) => self.column(&ident.value), + SqlExpr::CompoundIdentifier(parts) => match parts.as_slice() { + [table, col] => self.qualified(&table.value, &col.value), + _ => Err(unsup("nested field access")), + }, + SqlExpr::Nested(inner) => self.expr(inner), + SqlExpr::Value(v) => literal(&v.value), + SqlExpr::BinaryOp { left, op, right } => self.binary(op, left, right), + SqlExpr::UnaryOp { + op: UnaryOperator::Minus, + expr, + } => { + // DuckDB `-x`: lower as 0 - x, reusing Sub's promotion. + let zero = SExpr { + kind: SKind::Lit(Lit::I64(0)), + ty: Ty::I64, + nullable: false, + }; + self.arith(ArithOp::Sub, zero, self.expr(expr)?) + } + SqlExpr::UnaryOp { + op: UnaryOperator::Plus, + expr, + } => self.expr(expr), + SqlExpr::UnaryOp { + op: UnaryOperator::Not, + expr, + } => { + let inner = bool_context(self.expr(expr)?, "NOT operand")?; + if inner.ty != Ty::I1 { + return Err(PrepareError::Bind(format!( + "NOT requires BOOLEAN, got {}", + inner.ty.name() + ))); + } + let nullable = inner.nullable; + Ok(SExpr { + kind: SKind::Not(Box::new(inner)), + ty: Ty::I1, + nullable, + }) + } + SqlExpr::UnaryOp { op, .. } => Err(unsup(format!("unary operator {op:?}"))), + SqlExpr::IsNull(inner) => self.is_null(inner, false), + SqlExpr::IsNotNull(inner) => self.is_null(inner, true), + SqlExpr::Case { + operand, + conditions, + else_result, + .. + } => self.case(operand.as_deref(), conditions, else_result.as_deref()), + SqlExpr::Cast { + kind, + expr, + data_type, + .. + } => { + let trying = match kind { + CastKind::Cast | CastKind::DoubleColon => false, + CastKind::TryCast | CastKind::SafeCast => true, + }; + self.cast(expr, data_type, trying) + } + SqlExpr::Function(f) => self.function(f), + SqlExpr::Trim { + expr, + trim_where, + trim_what, + trim_characters, + } => { + let side = match trim_where { + None | Some(sqlparser::ast::TrimWhereField::Both) => TrimSide::Both, + Some(sqlparser::ast::TrimWhereField::Leading) => TrimSide::Lead, + Some(sqlparser::ast::TrimWhereField::Trailing) => TrimSide::Trail, + }; + let chars: Option<&SqlExpr> = match (trim_what, trim_characters) { + (Some(w), _) => Some(w), + (None, Some(cs)) if cs.len() == 1 => Some(&cs[0]), + (None, Some(cs)) if cs.is_empty() => None, + (None, Some(_)) => return Err(unsup("TRIM with multiple character args")), + (None, None) => None, + }; + self.trim_node(side, expr, chars) + } + SqlExpr::Substring { + expr, + substring_from, + substring_for, + .. + } => self.substr_node(expr, substring_from.as_deref(), substring_for.as_deref()), + SqlExpr::Between { .. } => Err(unsup("BETWEEN")), + SqlExpr::InList { .. } => Err(unsup("IN (...)")), + SqlExpr::Like { .. } => Err(unsup("LIKE")), + other => Err(unsup(format!("expression: {other}"))), + } + } + + /// The lane of value column `pos` of join `j`: NULL-able exactly when + /// the join is LEFT (a miss makes it NULL); INNER misses never reach an + /// expression (the row was skipped). + fn static_lane(&self, j: usize, pos: usize) -> SExpr { + let sj = &self.joins[j]; + let col = &sj.table.cols[sj.val_cols[pos] as usize]; + SExpr { + kind: SKind::StaticCol { + join: j as u32, + col: pos as u32, + }, + ty: col.ty.ty, + nullable: sj.kind == JoinKind::Left, + } + } + + /// Case-insensitive, spelling-preserving bare-column bind over the whole + /// scope: the dynamic table plus every joined static table's value + /// columns (DuckDB semantics; ambiguity is an error). + fn column(&self, name: &str) -> Result { + let mut hits: Vec = Vec::new(); + for (i, c) in self.in_cols.iter().enumerate() { + if c.name.eq_ignore_ascii_case(name) { + hits.push(SExpr { + kind: SKind::Col(i as u32), + ty: c.ty.ty, + nullable: c.ty.nullable, + }); + } + } + let mut key_only = false; + for (j, sj) in self.joins.iter().enumerate() { + for pos in 0..sj.val_cols.len() { + if sj.table.cols[sj.val_cols[pos] as usize] + .name + .eq_ignore_ascii_case(name) + { + hits.push(self.static_lane(j, pos)); + } + } + key_only |= sj + .key_cols + .iter() + .any(|&ci| sj.table.cols[ci as usize].name.eq_ignore_ascii_case(name)); + } + match hits.len() { + 1 => Ok(hits.pop().expect("len checked")), + 0 if key_only => Err(unsup(format!( + "referencing join key column '{name}' outside its ON clause" + ))), + // Real DuckDB features we don't model reject cleanly, not as + // bind errors: the rowid pseudo-column, and DuckDB's lateral + // alias extension (a SELECT alias visible inside WHERE). + 0 if name.eq_ignore_ascii_case("rowid") => Err(unsup("rowid pseudo-column")), + 0 if self + .select_aliases + .iter() + .any(|a| a.eq_ignore_ascii_case(name)) => + { + Err(unsup(format!( + "SELECT alias '{name}' referenced outside the SELECT list (lateral alias)" + ))) + } + 0 => Err(PrepareError::Bind(format!( + "column '{name}' does not exist in scope" + ))), + _ => Err(PrepareError::Bind(format!("ambiguous column '{name}'"))), + } + } + + /// `table.col` bind: the dynamic table by its FROM spelling, a joined + /// static table by its alias (or name). + fn qualified(&self, table: &str, name: &str) -> Result { + if table.eq_ignore_ascii_case(&self.this_name) { + let mut hit = None; + for (i, c) in self.in_cols.iter().enumerate() { + if c.name.eq_ignore_ascii_case(name) { + if hit.is_some() { + return Err(PrepareError::Bind(format!("ambiguous column '{name}'"))); + } + hit = Some((i, c)); + } + } + let (i, c) = hit.ok_or_else(|| { + PrepareError::Bind(format!("column '{name}' does not exist in '{table}'")) + })?; + return Ok(SExpr { + kind: SKind::Col(i as u32), + ty: c.ty.ty, + nullable: c.ty.nullable, + }); + } + for (j, sj) in self.joins.iter().enumerate() { + if !sj.name.eq_ignore_ascii_case(table) { + continue; + } + let mut hit = None; + for pos in 0..sj.val_cols.len() { + if sj.table.cols[sj.val_cols[pos] as usize] + .name + .eq_ignore_ascii_case(name) + { + if hit.is_some() { + return Err(PrepareError::Bind(format!("ambiguous column '{name}'"))); + } + hit = Some(pos); + } + } + if let Some(pos) = hit { + return Ok(self.static_lane(j, pos)); + } + if sj + .key_cols + .iter() + .any(|&ci| sj.table.cols[ci as usize].name.eq_ignore_ascii_case(name)) + { + return Err(unsup(format!( + "referencing join key column '{table}.{name}' outside its ON clause" + ))); + } + return Err(PrepareError::Bind(format!( + "column '{name}' does not exist in '{table}'" + ))); + } + Err(PrepareError::Bind(format!("unknown table '{table}'"))) + } + + fn binary( + &self, + op: &BinaryOperator, + left: &SqlExpr, + right: &SqlExpr, + ) -> Result { + let a = self.expr_or_null(left)?; + let b = self.expr_or_null(right)?; + // A NULL literal adopts the other side's type; the op itself is not + // folded (NULL AND FALSE is FALSE, so folding would be wrong). + let (a, b) = match (a, b) { + (Some(a), Some(b)) => (a, b), + (Some(a), None) => { + let n = null_of(null_context_ty(op, a.ty)); + (a, n) + } + (None, Some(b)) => { + let n = null_of(null_context_ty(op, b.ty)); + (n, b) + } + (None, None) => return Err(unsup("NULL NULL without a typing context")), + }; + match op { + BinaryOperator::Plus => self.arith(ArithOp::Add, a, b), + BinaryOperator::Minus => self.arith(ArithOp::Sub, a, b), + BinaryOperator::Multiply => self.arith(ArithOp::Mul, a, b), + BinaryOperator::Divide => self.arith(ArithOp::Div, a, b), + BinaryOperator::Modulo => self.arith(ArithOp::Rem, a, b), + BinaryOperator::Eq => self.cmp(CmpPred::Eq, a, b), + BinaryOperator::NotEq => self.cmp(CmpPred::Ne, a, b), + BinaryOperator::Lt => self.cmp(CmpPred::Lt, a, b), + BinaryOperator::LtEq => self.cmp(CmpPred::Le, a, b), + BinaryOperator::Gt => self.cmp(CmpPred::Gt, a, b), + BinaryOperator::GtEq => self.cmp(CmpPred::Ge, a, b), + BinaryOperator::And | BinaryOperator::Or => { + let a = bool_context(a, "AND/OR operand")?; + let b = bool_context(b, "AND/OR operand")?; + let nullable = a.nullable || b.nullable; + let (a, b) = (Box::new(a), Box::new(b)); + let kind = if matches!(op, BinaryOperator::And) { + SKind::And { a, b } + } else { + SKind::Or { a, b } + }; + Ok(SExpr { + kind, + ty: Ty::I1, + nullable, + }) + } + BinaryOperator::StringConcat => { + // DuckDB: || is ALWAYS string concat (1 || 2 = '12', + // true || true = 'truetrue'), NULL-propagating; operands + // implicitly cast to VARCHAR. + let (a, b) = (to_varchar(a), to_varchar(b)); + let nullable = a.nullable || b.nullable; + Ok(SExpr { + kind: SKind::Concat { + a: Box::new(a), + b: Box::new(b), + }, + ty: Ty::Str, + nullable, + }) + } + other => Err(unsup(format!("operator {other}"))), + } + } + + fn is_null(&self, inner: &SqlExpr, negated: bool) -> Result { + // NULL IS NULL is legal and constant; type the literal as i64 + // arbitrarily (only its flag matters). + let inner = match self.expr_or_null(inner)? { + Some(e) => e, + None => null_of(Ty::I64), + }; + Ok(SExpr { + kind: SKind::IsNull { + negated, + inner: Box::new(inner), + }, + ty: Ty::I1, + nullable: false, + }) + } + + fn case( + &self, + operand: Option<&SqlExpr>, + conditions: &[sqlparser::ast::CaseWhen], + else_result: Option<&SqlExpr>, + ) -> Result { + if conditions.is_empty() { + return Err(PrepareError::Bind("CASE with no WHEN arms".to_string())); + } + // Bind conditions: searched form directly; simple form desugars to + // `operand = value` per arm (operand re-bound per arm via clone — + // pure re-evaluation, same result). + let bound_operand = operand.map(|op| self.expr(op)).transpose()?; + let mut conds = Vec::with_capacity(conditions.len()); + for when in conditions { + let c = match &bound_operand { + Some(op) => { + let v = match self.expr_or_null(&when.condition)? { + Some(v) => v, + None => null_of(op.ty), + }; + self.cmp(CmpPred::Eq, op.clone(), v)? + } + None => match self.expr_or_null(&when.condition)? { + Some(c) => bool_context(c, "CASE WHEN condition")?, + None => null_of(Ty::I1), + }, + }; + conds.push(c); + } + + // Bind results (NULL allowed), then unify their types. + let mut results: Vec> = Vec::with_capacity(conditions.len()); + for when in conditions { + results.push(self.expr_or_null(&when.result)?); + } + let else_bound: Option> = + else_result.map(|e| self.expr_or_null(e)).transpose()?; + + let mut unified: Option = None; + for r in results.iter().chain(else_bound.iter()).flatten() { + unified = Some(match unified { + None => r.ty, + Some(u) if u == r.ty => u, + Some(Ty::I64) if r.ty == Ty::F64 => Ty::F64, + Some(Ty::F64) if r.ty == Ty::I64 => Ty::F64, + Some(u) => { + return Err(PrepareError::Bind(format!( + "CASE branches disagree: {} vs {}", + u.name(), + r.ty.name() + ))) + } + }); + } + let Some(unified) = unified else { + return Err(unsup("CASE where every branch is NULL")); + }; + + let coerce = |r: Option| -> SExpr { + match r { + None => null_of(unified), + Some(e) if e.ty == Ty::I64 && unified == Ty::F64 => promote_f64(e), + Some(e) => e, + } + }; + let results: Vec = results.into_iter().map(coerce).collect(); + let default = else_bound.map(coerce); + + let nullable = default.is_none() + || results.iter().any(|r| r.nullable) + || default.as_ref().is_some_and(|d| d.nullable); + let arms = conds.into_iter().zip(results).collect(); + Ok(SExpr { + kind: SKind::Case { + arms, + default: default.map(Box::new), + }, + ty: unified, + nullable, + }) + } + + fn cast( + &self, + expr: &SqlExpr, + data_type: &sqlparser::ast::DataType, + trying: bool, + ) -> Result { + let to = cast_target(data_type)?; + let inner = match self.expr_or_null(expr)? { + Some(e) => e, + // CAST(NULL AS T) is just a typed NULL, both forms. + None => return Ok(null_of(to)), + }; + if inner.ty == to && !trying { + return Ok(inner); + } + if inner.ty == Ty::Str && to == Ty::I1 { + return Err(unsup("CAST VARCHAR -> BOOLEAN")); + } + let nullable = trying || inner.nullable; + Ok(SExpr { + kind: SKind::Cast { + inner: Box::new(inner), + trying, + }, + ty: to, + nullable, + }) + } + + fn arith(&self, op: ArithOp, a: SExpr, b: SExpr) -> Result { + let (a, b, ty) = numeric_promote(op, a, b)?; + let nullable = a.nullable || b.nullable; + // DuckDB pin (2026-07-26): integer % by zero is NULL, not an error — + // guard with a CASE unless the divisor is a provably non-zero + // literal. The irem trap stays reachable only for MIN % -1, where + // DuckDB traps too. Float % is IEEE (x % 0.0 = NaN), no guard. + if op == ArithOp::Rem + && ty == Ty::I64 + && !matches!(b.kind, SKind::Lit(Lit::I64(n)) if n != 0) + { + let zero = SExpr { + kind: SKind::Lit(Lit::I64(0)), + ty: Ty::I64, + nullable: false, + }; + // The guard must fire for a NULL divisor too: `b = 0` alone is + // NULL there (arm not taken) and the irem would run on the + // garbage payload. TRUE OR NULL = TRUE makes IS NULL the shield. + let is_zero = self.cmp(CmpPred::Eq, b.clone(), zero)?; + let cond = if b.nullable { + let is_null = SExpr { + kind: SKind::IsNull { + negated: false, + inner: Box::new(b.clone()), + }, + ty: Ty::I1, + nullable: false, + }; + SExpr { + kind: SKind::Or { + a: Box::new(is_null), + b: Box::new(is_zero), + }, + ty: Ty::I1, + nullable: true, + } + } else { + is_zero + }; + let rem = SExpr { + kind: SKind::Arith { + op, + a: Box::new(a), + b: Box::new(b), + }, + ty, + nullable, + }; + return Ok(SExpr { + kind: SKind::Case { + arms: vec![(cond, null_of(Ty::I64))], + default: Some(Box::new(rem)), + }, + ty: Ty::I64, + nullable: true, + }); + } + Ok(SExpr { + kind: SKind::Arith { + op, + a: Box::new(a), + b: Box::new(b), + }, + ty, + nullable, + }) + } + + fn cmp(&self, pred: CmpPred, a: SExpr, b: SExpr) -> Result { + let (a, b) = match (a.ty, b.ty) { + (x, y) if x == y => (a, b), + (Ty::I64, Ty::F64) => (promote_f64(a), b), + (Ty::F64, Ty::I64) => (a, promote_f64(b)), + (x, y) => { + return Err(PrepareError::Bind(format!( + "cannot compare {} with {}", + x.name(), + y.name() + ))) + } + }; + if a.ty == Ty::I1 { + return Err(unsup("comparison on BOOLEAN")); + } + let nullable = a.nullable || b.nullable; + Ok(SExpr { + kind: SKind::Cmp { + pred, + a: Box::new(a), + b: Box::new(b), + }, + ty: Ty::I1, + nullable, + }) + } + + /// The v0 builtin catalogue. Everything here follows the measured pins + /// in docs/superpowers/specs/2026-07-26-stretch4-builtin-pins.md; names + /// not listed reject as clean unsupported. + fn function(&self, f: &sqlparser::ast::Function) -> Result { + use sqlparser::ast::{FunctionArg, FunctionArgExpr, FunctionArguments}; + let name = f.name.to_string().to_lowercase(); + let FunctionArguments::List(list) = &f.args else { + return Err(unsup(format!( + "function {} without an argument list", + f.name + ))); + }; + if !list.clauses.is_empty() || list.duplicate_treatment.is_some() { + return Err(unsup(format!("function {} argument clauses", f.name))); + } + let mut args: Vec<&SqlExpr> = Vec::with_capacity(list.args.len()); + for a in &list.args { + match a { + FunctionArg::Unnamed(FunctionArgExpr::Expr(e)) => args.push(e), + _ => return Err(unsup(format!("function {} argument form", f.name))), + } + } + match name.as_str() { + "upper" | "lower" => { + let [arg] = args[..] else { + return Err(PrepareError::Bind(format!( + "{name} takes exactly 1 argument" + ))); + }; + let Some(inner) = self.expr_or_null(arg)? else { + return Ok(null_of(Ty::Str)); + }; + if inner.ty != Ty::Str { + // DuckDB has no implicit numeric->VARCHAR coercion here. + return Err(PrepareError::Bind(format!( + "no function matches {name}({})", + inner.ty.name() + ))); + } + let nullable = inner.nullable; + Ok(SExpr { + kind: SKind::StrCase { + upper: name == "upper", + a: Box::new(inner), + }, + ty: Ty::Str, + nullable, + }) + } + "ltrim" | "rtrim" => { + let side = if name == "ltrim" { + TrimSide::Lead + } else { + TrimSide::Trail + }; + match args[..] { + [s] => self.trim_node(side, s, None), + [s, c] => self.trim_node(side, s, Some(c)), + _ => Err(PrepareError::Bind(format!("{name} takes 1 or 2 arguments"))), + } + } + "abs" => { + let [arg] = args[..] else { + return Err(PrepareError::Bind( + "abs takes exactly 1 argument".to_string(), + )); + }; + // abs(NULL) binds to abs(BIGINT) in DuckDB. + let Some(inner) = self.expr_or_null(arg)? else { + return Ok(null_of(Ty::I64)); + }; + if !matches!(inner.ty, Ty::I64 | Ty::F64) { + return Err(PrepareError::Bind(format!( + "no function matches abs({})", + inner.ty.name() + ))); + } + let (ty, nullable) = (inner.ty, inner.nullable); + Ok(SExpr { + kind: SKind::Abs(Box::new(inner)), + ty, + nullable, + }) + } + "round" => match args[..] { + [arg] => { + let Some(inner) = self.expr_or_null(arg)? else { + return Ok(null_of(Ty::I64)); + }; + match inner.ty { + // Measured: integer round is identity, type preserved. + Ty::I64 => Ok(inner), + Ty::F64 => { + let nullable = inner.nullable; + Ok(SExpr { + kind: SKind::Round(Box::new(inner)), + ty: Ty::F64, + nullable, + }) + } + other => Err(PrepareError::Bind(format!( + "no function matches round({})", + other.name() + ))), + } + } + [_, _] => Err(unsup("round with digits (scale-then-round algorithm)")), + _ => Err(PrepareError::Bind( + "round takes 1 or 2 arguments".to_string(), + )), + }, + "concat" => { + if args.is_empty() { + return Err(PrepareError::Bind( + "concat needs at least 1 argument".to_string(), + )); + } + // CONCAT skips NULLs (measured): a literal NULL contributes + // nothing, a nullable arg becomes CASE WHEN x IS NULL THEN '' + // ELSE x END, and the all-NULL call is ''. + let mut acc: Option = None; + for arg in &args { + let Some(e) = self.expr_or_null(arg)? else { + continue; + }; + let e = to_varchar(e); + let piece = if e.nullable { + let cond = SExpr { + kind: SKind::IsNull { + negated: false, + inner: Box::new(e.clone()), + }, + ty: Ty::I1, + nullable: false, + }; + SExpr { + kind: SKind::Case { + arms: vec![(cond, lit_str(""))], + default: Some(Box::new(e)), + }, + ty: Ty::Str, + // Never NULL: either arm produces a value. The + // default's flag is provably true on its path. + nullable: false, + } + } else { + e + }; + acc = Some(match acc { + None => piece, + Some(p) => SExpr { + kind: SKind::Concat { + a: Box::new(p), + b: Box::new(piece), + }, + ty: Ty::Str, + nullable: false, + }, + }); + } + Ok(acc.unwrap_or_else(|| lit_str(""))) + } + "coalesce" => { + // Lazy per-row (measured: untaken erroring arms don't fire) — + // guaranteed here because CASE branches run only when taken. + let mut bound = Vec::with_capacity(args.len()); + for arg in &args { + if let Some(e) = self.expr_or_null(arg)? { + bound.push(e); + } // literal NULL args never produce a value: drop them + } + if bound.is_empty() { + return Err(unsup("COALESCE of only NULL literals")); + } + let mut unified = bound[0].ty; + for e in &bound[1..] { + unified = match (unified, e.ty) { + (u, t) if u == t => u, + (Ty::I64, Ty::F64) | (Ty::F64, Ty::I64) => Ty::F64, + (u, t) => { + return Err(PrepareError::Bind(format!( + "COALESCE arguments disagree: {} vs {}", + u.name(), + t.name() + ))) + } + }; + } + let mut bound: Vec = bound + .into_iter() + .map(|e| { + if e.ty == Ty::I64 && unified == Ty::F64 { + promote_f64(e) + } else { + e + } + }) + .collect(); + // Args after the first non-nullable one are unreachable. + if let Some(stop) = bound.iter().position(|e| !e.nullable) { + bound.truncate(stop + 1); + } + let mut it = bound.into_iter().rev(); + let mut acc = it.next().expect("non-empty"); + for a in it { + let nullable = a.nullable && acc.nullable; + let cond = SExpr { + kind: SKind::IsNull { + negated: true, + inner: Box::new(a.clone()), + }, + ty: Ty::I1, + nullable: false, + }; + acc = SExpr { + kind: SKind::Case { + arms: vec![(cond, a)], + default: Some(Box::new(acc)), + }, + ty: unified, + nullable, + }; + } + Ok(acc) + } + "nullif" => { + let [a, b] = args[..] else { + return Err(PrepareError::Bind( + "nullif takes exactly 2 arguments".to_string(), + )); + }; + match (self.expr_or_null(a)?, self.expr_or_null(b)?) { + (None, Some(b)) => Ok(null_of(b.ty)), + // a = NULL is never TRUE, so nullif(a, NULL) is a. + (Some(a), None) => Ok(a), + (None, None) => Err(unsup("NULLIF(NULL, NULL)")), + (Some(a), Some(b)) => { + // Comparison at the promoted type; result keeps a's + // ORIGINAL type (measured: nullif(1, 1.0) -> INTEGER). + let cond = self.cmp(CmpPred::Eq, a.clone(), b)?; + let ty = a.ty; + Ok(SExpr { + kind: SKind::Case { + arms: vec![(cond, null_of(ty))], + default: Some(Box::new(a)), + }, + ty, + nullable: true, + }) + } + } + } + _ => Err(unsup(format!( + "function {} (not in the v0 catalogue)", + f.name + ))), + } + } + + /// All TRIM forms plus ltrim/rtrim. `chars` is the optional trim-set + /// expression; absent means DuckDB's default — the single space (only + /// 0x20 is trimmed, never tabs/newlines). + fn trim_node( + &self, + side: TrimSide, + s: &SqlExpr, + chars: Option<&SqlExpr>, + ) -> Result { + let Some(s) = self.expr_or_null(s)? else { + return Ok(null_of(Ty::Str)); + }; + if s.ty != Ty::Str { + return Err(PrepareError::Bind(format!( + "trim needs VARCHAR, got {}", + s.ty.name() + ))); + } + let chars = match chars { + Some(c) => match self.expr_or_null(c)? { + // A NULL trim-set propagates NULL (measured). + None => return Ok(null_of(Ty::Str)), + Some(c) if c.ty == Ty::Str => c, + Some(c) => { + return Err(PrepareError::Bind(format!( + "trim characters must be VARCHAR, got {}", + c.ty.name() + ))) + } + }, + None => lit_str(ZS_SPACES), + }; + let nullable = s.nullable || chars.nullable; + Ok(SExpr { + kind: SKind::Trim { + side, + a: Box::new(s), + chars: Box::new(chars), + }, + ty: Ty::Str, + nullable, + }) + } + + /// SUBSTR / SUBSTRING (both syntaxes). Missing start means 1; missing + /// length means i64::MAX ("rest of the string" under the saturating + /// window arithmetic in the interpreter). + fn substr_node( + &self, + s: &SqlExpr, + from: Option<&SqlExpr>, + for_: Option<&SqlExpr>, + ) -> Result { + let Some(s) = self.expr_or_null(s)? else { + return Ok(null_of(Ty::Str)); + }; + if s.ty != Ty::Str { + return Err(PrepareError::Bind(format!( + "substr needs VARCHAR, got {}", + s.ty.name() + ))); + } + // Ok(None) = a literal NULL argument: the whole call is NULL. + let num = |e: &SqlExpr| -> Result, PrepareError> { + match self.expr_or_null(e)? { + None => Ok(None), + Some(x) if x.ty == Ty::I64 => Ok(Some(x)), + Some(x) => Err(PrepareError::Bind(format!( + "substr position/length must be INTEGER, got {}", + x.ty.name() + ))), + } + }; + let start = match from { + Some(e) => match num(e)? { + Some(x) => x, + None => return Ok(null_of(Ty::Str)), + }, + None => lit_i64(1), + }; + let len = match for_ { + Some(e) => match num(e)? { + Some(x) => Some(Box::new(x)), + None => return Ok(null_of(Ty::Str)), + }, + None => None, + }; + let nullable = s.nullable || start.nullable || len.as_ref().is_some_and(|l| l.nullable); + Ok(SExpr { + kind: SKind::Substr { + a: Box::new(s), + start: Box::new(start), + len, + }, + ty: Ty::Str, + nullable, + }) + } +} + +/// DuckDB's default trim set (adversarial census, 1.5.5): exactly the +/// Unicode Zs space separators — NOT tab/newline/ZWSP/BOM/LS/PS/NEL. +const ZS_SPACES: &str = "\u{20}\u{A0}\u{1680}\u{2000}\u{2001}\u{2002}\u{2003}\u{2004}\u{2005}\ + \u{2006}\u{2007}\u{2008}\u{2009}\u{200A}\u{202F}\u{205F}\u{3000}"; + +fn lit_str(s: &str) -> SExpr { + SExpr { + kind: SKind::Lit(Lit::Str(s.to_string())), + ty: Ty::Str, + nullable: false, + } +} + +fn lit_i64(n: i64) -> SExpr { + SExpr { + kind: SKind::Lit(Lit::I64(n)), + ty: Ty::I64, + nullable: false, + } +} + +/// DuckDB coerces numeric values to BOOLEAN in conditional contexts — +/// WHERE, AND/OR/NOT operands, CASE WHEN conditions (measured 1.5.5: +/// nonzero -> true including NaN, 0 and -0.0 -> false, NULL -> NULL). +/// Strings stay a bind error (DuckDB errors at runtime; such queries never +/// mine into the corpus). +fn bool_context(e: SExpr, what: &str) -> Result { + match e.ty { + Ty::I1 => Ok(e), + Ty::I64 | Ty::F64 => { + if matches!(e.kind, SKind::NullOf) { + return Ok(null_of(Ty::I1)); + } + let nullable = e.nullable; + Ok(SExpr { + kind: SKind::Cast { + inner: Box::new(e), + trying: false, + }, + ty: Ty::I1, + nullable, + }) + } + other => Err(PrepareError::Bind(format!( + "{what} must be BOOLEAN, got {}", + other.name() + ))), + } +} + +/// DuckDB's implicit VARCHAR coercion for concatenation: ints, floats and +/// bools all render through the same conversion CAST uses. +fn to_varchar(e: SExpr) -> SExpr { + if e.ty == Ty::Str { + return e; + } + if matches!(e.kind, SKind::NullOf) { + return null_of(Ty::Str); + } + let nullable = e.nullable; + SExpr { + kind: SKind::Cast { + inner: Box::new(e), + trying: false, + }, + ty: Ty::Str, + nullable, + } +} + +fn null_of(ty: Ty) -> SExpr { + SExpr { + kind: SKind::NullOf, + ty, + nullable: true, + } +} + +/// The type a bare NULL adopts next to a typed operand. +fn null_context_ty(op: &BinaryOperator, other: Ty) -> Ty { + match op { + BinaryOperator::And | BinaryOperator::Or => Ty::I1, + _ => other, + } +} + +fn cast_target(dt: &sqlparser::ast::DataType) -> Result { + let name = dt.to_string().to_uppercase(); + if name.contains("INT") { + // BIGINT/INTEGER/SMALLINT/TINYINT/HUGEINT/U* all collapse to i64 + // (range divergence for HUGEINT noted in the module docs). + Ok(Ty::I64) + } else if name.starts_with("DOUBLE") + || name.starts_with("FLOAT") + || name.starts_with("REAL") + || name.starts_with("DECIMAL") + || name.starts_with("NUMERIC") + { + Ok(Ty::F64) + } else if name.starts_with("VARCHAR") + || name.starts_with("TEXT") + || name.starts_with("STRING") + || name.starts_with("CHAR") + { + Ok(Ty::Str) + } else if name.starts_with("BOOL") { + Ok(Ty::I1) + } else { + Err(unsup(format!("CAST target type {name}"))) + } +} + +fn literal(v: &SqlValue) -> Result { + let (lit, ty) = match v { + SqlValue::Number(text, _) => { + if text.contains('.') || text.to_ascii_lowercase().contains('e') { + // DuckDB would type this DECIMAL; v0 collapses to f64. + let f = text + .parse::() + .map_err(|_| PrepareError::Bind(format!("bad numeric literal '{text}'")))?; + (Lit::F64(f), Ty::F64) + } else { + let i = text + .parse::() + .map_err(|_| PrepareError::Bind(format!("bad integer literal '{text}'")))?; + (Lit::I64(i), Ty::I64) + } + } + SqlValue::SingleQuotedString(s) => (Lit::Str(s.clone()), Ty::Str), + SqlValue::Boolean(b) => (Lit::I1(*b), Ty::I1), + SqlValue::Null => unreachable!("NULL handled by expr_or_null"), + other => return Err(unsup(format!("literal {other}"))), + }; + Ok(SExpr { + kind: SKind::Lit(lit), + ty, + nullable: false, + }) +} + +/// DuckDB numeric promotion for the v0 type lattice: `/` is always float +/// division; otherwise int op int stays int, anything touching f64 is f64. +fn numeric_promote(op: ArithOp, a: SExpr, b: SExpr) -> Result<(SExpr, SExpr, Ty), PrepareError> { + let numeric = |e: &SExpr| matches!(e.ty, Ty::I64 | Ty::F64); + if !numeric(&a) || !numeric(&b) { + return Err(PrepareError::Bind(format!( + "arithmetic needs numeric operands, got {} and {}", + a.ty.name(), + b.ty.name() + ))); + } + if op == ArithOp::Div { + return Ok((promote_f64(a), promote_f64(b), Ty::F64)); + } + match (a.ty, b.ty) { + (Ty::I64, Ty::I64) => Ok((a, b, Ty::I64)), + (Ty::F64, Ty::F64) => Ok((a, b, Ty::F64)), + (Ty::I64, Ty::F64) => Ok((promote_f64(a), b, Ty::F64)), + (Ty::F64, Ty::I64) => Ok((a, promote_f64(b), Ty::F64)), + _ => unreachable!("guarded numeric above"), + } +} + +fn promote_f64(e: SExpr) -> SExpr { + if e.ty == Ty::F64 { + return e; + } + // A typed NULL promotes by retyping — no conversion node needed. + if matches!(e.kind, SKind::NullOf) { + return SExpr { + kind: SKind::NullOf, + ty: Ty::F64, + nullable: true, + }; + } + let nullable = e.nullable; + SExpr { + kind: SKind::IntToFloat(Box::new(e)), + ty: Ty::F64, + nullable, + } +} diff --git a/src/specializer/ir/gen.rs b/src/specializer/ir/gen.rs index 8f409f1..fcb4feb 100644 --- a/src/specializer/ir/gen.rs +++ b/src/specializer/ir/gen.rs @@ -9,8 +9,8 @@ //! the input source for M-interp's interpreter-vs-codegen fuzzing. use super::{ - BinOp, Block, BlockId, Builder, Col, ColTy, CmpPred, Inst, Lit, Program, RoundMode, StaticTy, - Term, Ty, Value, + BinOp, Block, BlockId, Builder, CmpPred, Col, ColTy, Inst, Lit, NumOp1, Program, RoundMode, + StaticTy, StrOp1, Term, TrimSide, Ty, Value, }; pub struct Rng(u64); @@ -122,14 +122,23 @@ impl Scope { /// Get a value of `ty`, minting a const when none is in scope (or sometimes /// just because — consts are where the literal edge cases enter). -fn ensure(rng: &mut Rng, b: &mut Builder, scope: &mut Scope, insts: &mut Vec, ty: Ty) -> Value { +fn ensure( + rng: &mut Rng, + b: &mut Builder, + scope: &mut Scope, + insts: &mut Vec, + ty: Ty, +) -> Value { if rng.chance(70) { if let Some(v) = scope.pick(rng, ty) { return v; } } let dst = b.fresh(); - insts.push(Inst::Const { dst, lit: rand_lit(rng, ty) }); + insts.push(Inst::Const { + dst, + lit: rand_lit(rng, ty), + }); scope.add(dst, ty); dst } @@ -148,12 +157,19 @@ fn load_all( if c.ty.nullable { let flag = b.fresh(); let dst = b.fresh(); - insts.push(Inst::LoadOpt { flag, dst, col: ci as u32 }); + insts.push(Inst::LoadOpt { + flag, + dst, + col: ci as u32, + }); scope.add(flag, Ty::I1); scope.add(dst, c.ty.ty); } else { let dst = b.fresh(); - insts.push(Inst::Load { dst, col: ci as u32 }); + insts.push(Inst::Load { + dst, + col: ci as u32, + }); scope.add(dst, c.ty.ty); } } @@ -162,13 +178,20 @@ fn load_all( StaticTy::Scalar(ct) if ct.nullable => { let flag = b.fresh(); let dst = b.fresh(); - insts.push(Inst::SloadOpt { static_id: si as u32, flag, dst }); + insts.push(Inst::SloadOpt { + static_id: si as u32, + flag, + dst, + }); scope.add(flag, Ty::I1); scope.add(dst, ct.ty); } StaticTy::Scalar(ct) => { let dst = b.fresh(); - insts.push(Inst::Sload { static_id: si as u32, dst }); + insts.push(Inst::Sload { + static_id: si as u32, + dst, + }); scope.add(dst, ct.ty); } StaticTy::Map { keys, values } => { @@ -184,7 +207,12 @@ fn load_all( scope.add(d, *vt); dsts.push(d); } - insts.push(Inst::Probe { static_id: si as u32, hit, dsts, keys: key_vals }); + insts.push(Inst::Probe { + static_id: si as u32, + hit, + dsts, + keys: key_vals, + }); } } } @@ -194,7 +222,7 @@ fn load_all( fn compute(rng: &mut Rng, b: &mut Builder, scope: &mut Scope, insts: &mut Vec) { let n = rng.below(7); for _ in 0..n { - match rng.below(8) { + match rng.below(12) { 0 => { let ops = [ BinOp::Iadd, @@ -204,6 +232,7 @@ fn compute(rng: &mut Rng, b: &mut Builder, scope: &mut Scope, insts: &mut Vec { @@ -247,7 +282,12 @@ fn compute(rng: &mut Rng, b: &mut Builder, scope: &mut Scope, insts: &mut Vec { @@ -257,7 +297,11 @@ fn compute(rng: &mut Rng, b: &mut Builder, scope: &mut Scope, insts: &mut Vec { - let mode = if rng.chance(50) { RoundMode::Trunc } else { RoundMode::Round }; + let mode = if rng.chance(50) { + RoundMode::Trunc + } else { + RoundMode::Round + }; let a = ensure(rng, b, scope, insts, Ty::F64); let dst = b.fresh(); insts.push(Inst::Ftoi { mode, dst, a }); @@ -270,6 +314,70 @@ fn compute(rng: &mut Rng, b: &mut Builder, scope: &mut Scope, insts: &mut Vec { + let op = if rng.chance(50) { + StrOp1::Upper + } else { + StrOp1::Lower + }; + let a = ensure(rng, b, scope, insts, Ty::Str); + let dst = b.fresh(); + insts.push(Inst::Str1 { op, dst, a }); + scope.add(dst, Ty::Str); + } + 9 => { + let sides = [TrimSide::Both, TrimSide::Lead, TrimSide::Trail]; + let side = sides[rng.below(3) as usize]; + let a = ensure(rng, b, scope, insts, Ty::Str); + let chars = ensure(rng, b, scope, insts, Ty::Str); + let dst = b.fresh(); + insts.push(Inst::Strim { + side, + dst, + a, + chars, + }); + scope.add(dst, Ty::Str); + } + 10 => { + let a = ensure(rng, b, scope, insts, Ty::Str); + // Positions are small fresh consts, not arbitrary scope + // values: the ±2^32 range guard traps, and generated + // programs must stay executable for M-interp fuzzing. + let small = |rng: &mut Rng, b: &mut Builder, insts: &mut Vec| { + let dst = b.fresh(); + insts.push(Inst::Const { + dst, + lit: Lit::I64(rng.below(19) as i64 - 9), + }); + dst + }; + let start = small(rng, b, insts); + scope.add(start, Ty::I64); + let len = if rng.chance(50) { + let l = small(rng, b, insts); + scope.add(l, Ty::I64); + Some(l) + } else { + None + }; + let dst = b.fresh(); + insts.push(Inst::Ssubstr { dst, a, start, len }); + scope.add(dst, Ty::Str); + } + 11 => { + // iabs excluded: it traps on i64::MIN, and generated programs + // must stay executable for M-interp fuzzing. + let op = if rng.chance(50) { + NumOp1::Fabs + } else { + NumOp1::Fround + }; + let a = ensure(rng, b, scope, insts, Ty::F64); + let dst = b.fresh(); + insts.push(Inst::Num1 { op, dst, a }); + scope.add(dst, Ty::F64); + } _ => { let (from, mk): (Ty, fn(Value, Value) -> Inst) = if rng.chance(50) { (Ty::I64, |dst, a| Inst::Itos { dst, a }) @@ -297,9 +405,16 @@ fn stores( let val = ensure(rng, b, scope, insts, c.ty.ty); if c.ty.nullable { let flag = ensure(rng, b, scope, insts, Ty::I1); - insts.push(Inst::StoreOpt { col: ci as u32, flag, val }); + insts.push(Inst::StoreOpt { + col: ci as u32, + flag, + val, + }); } else { - insts.push(Inst::Store { col: ci as u32, val }); + insts.push(Inst::Store { + col: ci as u32, + val, + }); } } } @@ -329,10 +444,16 @@ pub fn gen_program(seed: u64) -> Program { } }; let in_cols: Vec = (0..1 + rng.below(3) as usize) - .map(|i| Col { name: name_of(i, "c", &mut rng), ty: rand_col_ty(&mut rng) }) + .map(|i| Col { + name: name_of(i, "c", &mut rng), + ty: rand_col_ty(&mut rng), + }) .collect(); let out_cols: Vec = (0..1 + rng.below(2) as usize) - .map(|i| Col { name: name_of(i, "o", &mut rng), ty: rand_col_ty(&mut rng) }) + .map(|i| Col { + name: name_of(i, "o", &mut rng), + ty: rand_col_ty(&mut rng), + }) .collect(); let shape = rng.below(3); @@ -345,7 +466,11 @@ pub fn gen_program(seed: u64) -> Program { load_all(&mut rng, &mut b, &mut scope, &mut insts, &in_cols, &statics); compute(&mut rng, &mut b, &mut scope, &mut insts); stores(&mut rng, &mut b, &mut scope, &mut insts, &out_cols); - blocks.push(Block { params: vec![], insts, term: Term::Emit }); + blocks.push(Block { + params: vec![], + insts, + term: Term::Emit, + }); } // Filter: entry branches to a storing block or a skip/trap block. 1 => { @@ -367,15 +492,38 @@ pub fn gen_program(seed: u64) -> Program { }); let mut keep_scope = Scope::new(); let mut keep_insts = Vec::new(); - load_all(&mut rng, &mut b, &mut keep_scope, &mut keep_insts, &in_cols, &statics); - stores(&mut rng, &mut b, &mut keep_scope, &mut keep_insts, &out_cols); - blocks.push(Block { params: vec![], insts: keep_insts, term: Term::Emit }); + load_all( + &mut rng, + &mut b, + &mut keep_scope, + &mut keep_insts, + &in_cols, + &statics, + ); + stores( + &mut rng, + &mut b, + &mut keep_scope, + &mut keep_insts, + &out_cols, + ); + blocks.push(Block { + params: vec![], + insts: keep_insts, + term: Term::Emit, + }); let drop_term = if rng.chance(80) { Term::Skip } else { - Term::Trap { msg: "generated trap".to_string() } + Term::Trap { + msg: "generated trap".to_string(), + } }; - blocks.push(Block { params: vec![], insts: vec![], term: drop_term }); + blocks.push(Block { + params: vec![], + insts: vec![], + term: drop_term, + }); } // Diamond: both arms feed a join param that lands in a store. _ => { @@ -402,16 +550,32 @@ pub fn gen_program(seed: u64) -> Program { blocks.push(Block { params: vec![], insts: arm_insts, - term: Term::Jump { to: BlockId(3), args: vec![v] }, + term: Term::Jump { + to: BlockId(3), + args: vec![v], + }, }); } let param = b.fresh(); let mut join_scope = Scope::new(); join_scope.add(param, join_ty); let mut join_insts = Vec::new(); - load_all(&mut rng, &mut b, &mut join_scope, &mut join_insts, &in_cols, &statics); + load_all( + &mut rng, + &mut b, + &mut join_scope, + &mut join_insts, + &in_cols, + &statics, + ); compute(&mut rng, &mut b, &mut join_scope, &mut join_insts); - stores(&mut rng, &mut b, &mut join_scope, &mut join_insts, &out_cols); + stores( + &mut rng, + &mut b, + &mut join_scope, + &mut join_insts, + &out_cols, + ); blocks.push(Block { params: vec![(param, join_ty)], insts: join_insts, diff --git a/src/specializer/ir/mod.rs b/src/specializer/ir/mod.rs index 2dd8dd4..775512f 100644 --- a/src/specializer/ir/mod.rs +++ b/src/specializer/ir/mod.rs @@ -208,9 +208,7 @@ impl PartialEq for Lit { match (self, other) { (Lit::I1(a), Lit::I1(b)) => a == b, (Lit::I64(a), Lit::I64(b)) => a == b, - (Lit::F64(a), Lit::F64(b)) => { - a.to_bits() == b.to_bits() || (a.is_nan() && b.is_nan()) - } + (Lit::F64(a), Lit::F64(b)) => a.to_bits() == b.to_bits() || (a.is_nan() && b.is_nan()), (Lit::Str(a), Lit::Str(b)) => a == b, _ => false, } @@ -240,6 +238,9 @@ pub enum BinOp { Fsub, Fmul, Fdiv, + /// f64 `%`, IEEE remainder-of-truncated-division (Rust `%`): sign of the + /// dividend, `x % 0.0` is NaN. Never traps (measured DuckDB 1.5.5). + Frem, And, Or, Xor, @@ -252,7 +253,9 @@ impl BinOp { BinOp::Iadd | BinOp::Isub | BinOp::Imul | BinOp::Idiv | BinOp::Irem => { (Ty::I64, Ty::I64) } - BinOp::Fadd | BinOp::Fsub | BinOp::Fmul | BinOp::Fdiv => (Ty::F64, Ty::F64), + BinOp::Fadd | BinOp::Fsub | BinOp::Fmul | BinOp::Fdiv | BinOp::Frem => { + (Ty::F64, Ty::F64) + } BinOp::And | BinOp::Or | BinOp::Xor => (Ty::I1, Ty::I1), } } @@ -268,6 +271,7 @@ impl BinOp { BinOp::Fsub => "fsub", BinOp::Fmul => "fmul", BinOp::Fdiv => "fdiv", + BinOp::Frem => "frem", BinOp::And => "and", BinOp::Or => "or", BinOp::Xor => "xor", @@ -304,6 +308,68 @@ pub enum RoundMode { Round, } +/// One-operand string ops (Str -> Str). Case mapping is SIMPLE (per-codepoint +/// 1:1) to track DuckDB/utf8proc — see the 2026-07-26 builtin-pins spec. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum StrOp1 { + Upper, + Lower, +} + +impl StrOp1 { + pub fn name(self) -> &'static str { + match self { + StrOp1::Upper => "supper", + StrOp1::Lower => "slower", + } + } +} + +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum TrimSide { + Both, + Lead, + Trail, +} + +impl TrimSide { + pub fn name(self) -> &'static str { + match self { + TrimSide::Both => "both", + TrimSide::Lead => "lead", + TrimSide::Trail => "trail", + } + } +} + +/// One-operand numeric ops. `Iabs` traps on i64::MIN (DuckDB: Out of Range); +/// `Fabs` clears the sign bit (abs(-0.0) = +0.0); `Fround` is half away from +/// zero (Rust `f64::round`), total on NaN/inf/huge. +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum NumOp1 { + Iabs, + Fabs, + Fround, +} + +impl NumOp1 { + /// Operand type == result type for all v0 numeric unaries. + pub fn sig(self) -> Ty { + match self { + NumOp1::Iabs => Ty::I64, + NumOp1::Fabs | NumOp1::Fround => Ty::F64, + } + } + + pub fn name(self) -> &'static str { + match self { + NumOp1::Iabs => "iabs", + NumOp1::Fabs => "fabs", + NumOp1::Fround => "fround", + } + } +} + #[derive(Clone, PartialEq, Debug)] pub enum Inst { Const { @@ -368,6 +434,39 @@ pub enum Inst { a: Value, b: Value, }, + /// `supper` / `slower` — Str -> Str, simple case mapping. + Str1 { + op: StrOp1, + dst: Value, + a: Value, + }, + /// `strim.side a, chars` — remove characters in the set `chars` from the + /// given side(s) of `a`. Both operands Str; the empty set is a no-op. + Strim { + side: TrimSide, + dst: Value, + a: Value, + chars: Value, + }, + /// `ssubstr a, start, len` / `ssubstr.rest a, start` — codepoint-window + /// substring with DuckDB's virtual-position arithmetic: negative start + /// counts from the end, start <= 0 consumes length before character 1, + /// a negative length slices BACKWARDS from the resolved start, and + /// offsets/lengths outside ±2^32 trap. `len: None` is the 2-arg SQL + /// form ("rest of the string"), which never length-traps — that is why + /// it is not a sentinel value. + Ssubstr { + dst: Value, + a: Value, + start: Value, + len: Option, + }, + /// `iabs` / `fabs` / `fround` — numeric unaries, operand ty == result ty. + Num1 { + op: NumOp1, + dst: Value, + a: Value, + }, /// Read a NOT NULL input column at the current row. Load { dst: Value, @@ -463,6 +562,10 @@ impl Inst { | Inst::Itos { dst, .. } | Inst::Ftos { dst, .. } | Inst::Sconcat { dst, .. } + | Inst::Str1 { dst, .. } + | Inst::Strim { dst, .. } + | Inst::Ssubstr { dst, .. } + | Inst::Num1 { dst, .. } | Inst::Load { dst, .. } | Inst::Sload { dst, .. } => vec![*dst], Inst::StoiOpt { flag, dst, .. } @@ -477,7 +580,6 @@ impl Inst { Inst::Store { .. } | Inst::StoreOpt { .. } => vec![], } } - } impl Term { @@ -500,6 +602,140 @@ impl Term { } } +impl Inst { + /// Rewrite every value reference (defs and uses) through `m`. + pub fn map_values(&mut self, m: &impl Fn(Value) -> Value) { + match self { + Inst::Const { dst, .. } | Inst::Load { dst, .. } | Inst::Sload { dst, .. } => { + *dst = m(*dst) + } + Inst::Itos { dst, a } + | Inst::Ftos { dst, a } + | Inst::Itof { dst, a } + | Inst::Ftoi { dst, a, .. } + | Inst::Str1 { dst, a, .. } + | Inst::Num1 { dst, a, .. } + | Inst::Not { dst, a } => { + *dst = m(*dst); + *a = m(*a); + } + Inst::Bin { dst, a, b, .. } + | Inst::Cmp { dst, a, b, .. } + | Inst::Strim { + dst, a, chars: b, .. + } + | Inst::Sconcat { dst, a, b } => { + *dst = m(*dst); + *a = m(*a); + *b = m(*b); + } + Inst::Ssubstr { dst, a, start, len } => { + *dst = m(*dst); + *a = m(*a); + *start = m(*start); + if let Some(len) = len { + *len = m(*len); + } + } + Inst::Select { dst, cond, a, b } => { + *dst = m(*dst); + *cond = m(*cond); + *a = m(*a); + *b = m(*b); + } + Inst::StoiOpt { flag, dst, a } | Inst::StofOpt { flag, dst, a } => { + *flag = m(*flag); + *dst = m(*dst); + *a = m(*a); + } + Inst::LoadOpt { flag, dst, .. } | Inst::SloadOpt { flag, dst, .. } => { + *flag = m(*flag); + *dst = m(*dst); + } + Inst::Store { val, .. } => *val = m(*val), + Inst::StoreOpt { flag, val, .. } => { + *flag = m(*flag); + *val = m(*val); + } + Inst::Probe { + hit, dsts, keys, .. + } => { + *hit = m(*hit); + for d in dsts { + *d = m(*d); + } + for k in keys { + *k = m(*k); + } + } + } + } +} + +impl Term { + /// Rewrite every value reference through `m`. + pub fn map_values(&mut self, m: &impl Fn(Value) -> Value) { + match self { + Term::Jump { args, .. } => { + for a in args { + *a = m(*a); + } + } + Term::Brif { + cond, + then_args, + else_args, + .. + } => { + *cond = m(*cond); + for a in then_args.iter_mut().chain(else_args.iter_mut()) { + *a = m(*a); + } + } + Term::Emit | Term::Skip | Term::Trap { .. } => {} + } + } +} + +/// Renumber value ids into canonical form: dense, in definition order as the +/// text format reads (block params, then instruction defs, per block in +/// order). A bijective rename — semantics and verification are unaffected — +/// after which `parse(print(p)) == p` holds exactly. Lowering runs this so +/// every prepared program is canonical even when block-splitting minted ids +/// out of text order. +pub fn canonicalize(p: &mut Program) { + let mut map: std::collections::HashMap = std::collections::HashMap::new(); + let mut next = 0u32; + for b in &p.blocks { + for (v, _) in &b.params { + map.entry(v.0).or_insert_with(|| { + let n = next; + next += 1; + n + }); + } + for inst in &b.insts { + for d in inst.dsts() { + map.entry(d.0).or_insert_with(|| { + let n = next; + next += 1; + n + }); + } + } + } + let f = |v: Value| Value(map[&v.0]); + for b in &mut p.blocks { + for (v, _) in &mut b.params { + *v = f(*v); + } + for inst in &mut b.insts { + inst.map_values(&f); + } + b.term.map_values(&f); + } +} + /// Builds programs with dense, definition-ordered value ids — the canonical /// form for which `parse(print(p)) == p` holds exactly. Lowering (M-lower) /// and the fuzz generator both construct through this. diff --git a/src/specializer/ir/parse.rs b/src/specializer/ir/parse.rs index 607b3ce..544e18b 100644 --- a/src/specializer/ir/parse.rs +++ b/src/specializer/ir/parse.rs @@ -10,8 +10,8 @@ use std::collections::HashMap; use super::{ - Block, BlockId, BinOp, Col, ColTy, CmpPred, Inst, Lit, Program, RoundMode, StaticTy, Term, - Ty, Value, + BinOp, Block, BlockId, CmpPred, Col, ColTy, Inst, Lit, NumOp1, Program, RoundMode, StaticTy, + StrOp1, Term, TrimSide, Ty, Value, }; #[derive(Debug)] @@ -42,16 +42,16 @@ pub fn parse(text: &str) -> Result { #[derive(Clone, PartialEq, Debug)] enum Tok { Ident(String), - Val(String), // %name - Num(String), // integer or float text, optional leading '-' - Str(String), // unescaped content - At, // @ + Val(String), // %name + Num(String), // integer or float text, optional leading '-' + Str(String), // unescaped content + At, // @ Comma, Colon, Dot, Eq, Question, - Arrow, // -> + Arrow, // -> LParen, RParen, LBrace, @@ -413,7 +413,11 @@ impl Parser { self.bump(); Ok(()) } else { - Err(self.err(format!("expected {}, found {}", tok.show(), self.peek().show()))) + Err(self.err(format!( + "expected {}, found {}", + tok.show(), + self.peek().show() + ))) } } @@ -501,7 +505,10 @@ impl Parser { // Resolve labels. let mut label_ids: HashMap = HashMap::new(); for (i, rb) in raw_blocks.iter().enumerate() { - if label_ids.insert(rb.label.clone(), BlockId(i as u32)).is_some() { + if label_ids + .insert(rb.label.clone(), BlockId(i as u32)) + .is_some() + { return Err(ParseError { line: rb.line, msg: format!("block label '{}' defined twice", rb.label), @@ -658,9 +665,7 @@ impl Parser { let name = match self.bump() { Tok::Val(n) => n, other => { - return Err( - self.err(format!("expected a %param, found {}", other.show())) - ) + return Err(self.err(format!("expected a %param, found {}", other.show()))) } }; self.expect(Tok::Colon)?; @@ -852,7 +857,7 @@ impl Parser { Inst::Const { dst: def!(0), lit } } "iadd" | "isub" | "imul" | "idiv" | "irem" | "fadd" | "fsub" | "fmul" | "fdiv" - | "and" | "or" | "xor" => { + | "frem" | "and" | "or" | "xor" => { want_dsts(1, self)?; let op = match opcode.as_str() { "iadd" => BinOp::Iadd, @@ -864,6 +869,7 @@ impl Parser { "fsub" => BinOp::Fsub, "fmul" => BinOp::Fmul, "fdiv" => BinOp::Fdiv, + "frem" => BinOp::Frem, "and" => BinOp::And, "or" => BinOp::Or, _ => BinOp::Xor, @@ -871,7 +877,12 @@ impl Parser { let a = self.use_value()?; self.expect(Tok::Comma)?; let b = self.use_value()?; - Inst::Bin { op, dst: def!(0), a, b } + Inst::Bin { + op, + dst: def!(0), + a, + b, + } } _ if head == "icmp" || head == "fcmp" || head == "scmp" => { want_dsts(1, self)?; @@ -892,7 +903,13 @@ impl Parser { let a = self.use_value()?; self.expect(Tok::Comma)?; let b = self.use_value()?; - Inst::Cmp { pred, ty, dst: def!(0), a, b } + Inst::Cmp { + pred, + ty, + dst: def!(0), + a, + b, + } } "not" => { want_dsts(1, self)?; @@ -906,7 +923,12 @@ impl Parser { let a = self.use_value()?; self.expect(Tok::Comma)?; let b = self.use_value()?; - Inst::Select { dst: def!(0), cond, a, b } + Inst::Select { + dst: def!(0), + cond, + a, + b, + } } "itof" => { want_dsts(1, self)?; @@ -921,7 +943,11 @@ impl Parser { RoundMode::Round }; let a = self.use_value()?; - Inst::Ftoi { mode, dst: def!(0), a } + Inst::Ftoi { + mode, + dst: def!(0), + a, + } } "itos" => { want_dsts(1, self)?; @@ -951,6 +977,69 @@ impl Parser { let b = self.use_value()?; Inst::Sconcat { dst: def!(0), a, b } } + "supper" | "slower" => { + want_dsts(1, self)?; + let op = if opcode == "supper" { + StrOp1::Upper + } else { + StrOp1::Lower + }; + let a = self.use_value()?; + Inst::Str1 { + op, + dst: def!(0), + a, + } + } + "strim.both" | "strim.lead" | "strim.trail" => { + want_dsts(1, self)?; + let side = match opcode.as_str() { + "strim.both" => TrimSide::Both, + "strim.lead" => TrimSide::Lead, + _ => TrimSide::Trail, + }; + let a = self.use_value()?; + self.expect(Tok::Comma)?; + let chars = self.use_value()?; + Inst::Strim { + side, + dst: def!(0), + a, + chars, + } + } + "ssubstr" | "ssubstr.rest" => { + want_dsts(1, self)?; + let a = self.use_value()?; + self.expect(Tok::Comma)?; + let start = self.use_value()?; + let len = if opcode == "ssubstr" { + self.expect(Tok::Comma)?; + Some(self.use_value()?) + } else { + None + }; + Inst::Ssubstr { + dst: def!(0), + a, + start, + len, + } + } + "iabs" | "fabs" | "fround" => { + want_dsts(1, self)?; + let op = match opcode.as_str() { + "iabs" => NumOp1::Iabs, + "fabs" => NumOp1::Fabs, + _ => NumOp1::Fround, + }; + let a = self.use_value()?; + Inst::Num1 { + op, + dst: def!(0), + a, + } + } "load" => { want_dsts(1, self)?; let col = self.col_ref("in", in_cols)?; @@ -959,7 +1048,11 @@ impl Parser { "load.opt" => { want_dsts(2, self)?; let col = self.col_ref("in", in_cols)?; - Inst::LoadOpt { flag: def!(0), dst: def!(1), col } + Inst::LoadOpt { + flag: def!(0), + dst: def!(1), + col, + } } "probe" => { if dst_names.is_empty() { @@ -976,17 +1069,29 @@ impl Parser { for i in 1..dst_names.len() { dsts.push(def!(i)); } - Inst::Probe { static_id, hit, dsts, keys } + Inst::Probe { + static_id, + hit, + dsts, + keys, + } } "sload" => { want_dsts(1, self)?; let static_id = self.static_ref(statics)?; - Inst::Sload { static_id, dst: def!(0) } + Inst::Sload { + static_id, + dst: def!(0), + } } "sload.opt" => { want_dsts(2, self)?; let static_id = self.static_ref(statics)?; - Inst::SloadOpt { static_id, flag: def!(0), dst: def!(1) } + Inst::SloadOpt { + static_id, + flag: def!(0), + dst: def!(1), + } } other => return Err(self.err(format!("unknown opcode '{other}'"))), }; diff --git a/src/specializer/ir/print.rs b/src/specializer/ir/print.rs index 9026858..8d93a96 100644 --- a/src/specializer/ir/print.rs +++ b/src/specializer/ir/print.rs @@ -121,6 +121,23 @@ fn print_inst(s: &mut String, p: &Program, inst: &Inst) { Inst::Sconcat { a, b, .. } => { let _ = write!(s, "sconcat {}, {}", val(*a), val(*b)); } + Inst::Str1 { op, a, .. } => { + let _ = write!(s, "{} {}", op.name(), val(*a)); + } + Inst::Strim { side, a, chars, .. } => { + let _ = write!(s, "strim.{} {}, {}", side.name(), val(*a), val(*chars)); + } + Inst::Ssubstr { a, start, len, .. } => match len { + Some(len) => { + let _ = write!(s, "ssubstr {}, {}, {}", val(*a), val(*start), val(*len)); + } + None => { + let _ = write!(s, "ssubstr.rest {}, {}", val(*a), val(*start)); + } + }, + Inst::Num1 { op, a, .. } => { + let _ = write!(s, "{} {}", op.name(), val(*a)); + } Inst::Load { col, .. } => { let _ = write!(s, "load in.{}", col_name(p, false, *col)); } @@ -196,10 +213,7 @@ fn val(v: Value) -> String { } fn tys(list: &[Ty]) -> String { - list.iter() - .map(|t| t.name()) - .collect::>() - .join(", ") + list.iter().map(|t| t.name()).collect::>().join(", ") } fn col_ty(ct: ColTy) -> String { diff --git a/src/specializer/ir/tests.rs b/src/specializer/ir/tests.rs index 42ba1dd..c8f9e12 100644 --- a/src/specializer/ir/tests.rs +++ b/src/specializer/ir/tests.rs @@ -30,7 +30,11 @@ fn fixtures_verify_and_round_trip() { let printed = print(&p); let p2 = parsed(&printed); assert_eq!(p2, p, "round-trip changed '{name}':\n{printed}"); - assert_eq!(print(&p2), printed, "printing is not a fixpoint for '{name}'"); + assert_eq!( + print(&p2), + printed, + "printing is not a fixpoint for '{name}'" + ); verify(&p2).unwrap_or_else(|_| panic!("canonical form of '{name}' fails verify")); } } @@ -42,13 +46,53 @@ fn fixtures_verify_and_round_trip() { fn fixtures_cover_every_opcode() { let all: String = fixtures::all().iter().map(|(_, t)| *t).collect(); let opcodes = [ - "const.i1", "const.i64", "const.f64", "const.str", "iadd", "isub", "imul", "idiv", - "irem", "fadd", "fsub", "fmul", "fdiv", "and", "or", "xor", "not", "icmp.", "fcmp.", - "scmp.", "select", "itof", "ftoi.trunc", "ftoi.round", "itos", "ftos", "stoi.opt", - "stof.opt", "sconcat", "load in.", "load.opt in.", "store out.", "store.opt out.", - "probe @", "sload @", "sload.opt @", "jump", "brif", "emit", "skip", "trap", + "const.i1", + "const.i64", + "const.f64", + "const.str", + "iadd", + "isub", + "imul", + "idiv", + "irem", + "fadd", + "fsub", + "fmul", + "fdiv", + "and", + "or", + "xor", + "not", + "icmp.", + "fcmp.", + "scmp.", + "select", + "itof", + "ftoi.trunc", + "ftoi.round", + "itos", + "ftos", + "stoi.opt", + "stof.opt", + "sconcat", + "load in.", + "load.opt in.", + "store out.", + "store.opt out.", + "probe @", + "sload @", + "sload.opt @", + "jump", + "brif", + "emit", + "skip", + "trap", ]; - let missing: Vec<&str> = opcodes.iter().filter(|op| !all.contains(**op)).copied().collect(); + let missing: Vec<&str> = opcodes + .iter() + .filter(|op| !all.contains(**op)) + .copied() + .collect(); assert!(missing.is_empty(), "no fixture covers: {missing:?}"); } @@ -103,25 +147,49 @@ entry: ); // ...and the verifier independently rejects the same shape when the // program is constructed through the API (the path lowering will use). - use super::{Block, Col, ColTy, Inst, Program, Term, Ty, Value, BinOp}; + use super::{BinOp, Block, Col, ColTy, Inst, Program, Term, Ty, Value}; let p = Program { statics: vec![], name: "f".into(), - in_cols: vec![Col { name: "a".into(), ty: ColTy { ty: Ty::I64, nullable: false } }], - out_cols: vec![Col { name: "o".into(), ty: ColTy { ty: Ty::I64, nullable: false } }], + in_cols: vec![Col { + name: "a".into(), + ty: ColTy { + ty: Ty::I64, + nullable: false, + }, + }], + out_cols: vec![Col { + name: "o".into(), + ty: ColTy { + ty: Ty::I64, + nullable: false, + }, + }], blocks: vec![Block { params: vec![], insts: vec![ - Inst::Bin { op: BinOp::Iadd, dst: Value(0), a: Value(1), b: Value(1) }, - Inst::Load { dst: Value(1), col: 0 }, - Inst::Store { col: 0, val: Value(0) }, + Inst::Bin { + op: BinOp::Iadd, + dst: Value(0), + a: Value(1), + b: Value(1), + }, + Inst::Load { + dst: Value(1), + col: 0, + }, + Inst::Store { + col: 0, + val: Value(0), + }, ], term: Term::Emit, }], }; let errs = verify(&p).expect_err("use-before-def must not verify"); assert!( - errs.iter().any(|e| e.to_string().contains("used before any definition")), + errs.iter() + .any(|e| e.to_string().contains("used before any definition")), "wrong errors: {:?}", errs.iter().map(|e| e.to_string()).collect::>() ); @@ -438,7 +506,10 @@ fn parser_rejects_bad_escape_and_unterminated_string() { "fn f(in: batch{a: i64}, out: batch{o: str}) {\nentry:\n %x = const.str \"\\q\"\n emit\n}", "unknown escape", ); - assert_parse_rejects("fn f(in: batch{a: i64}, out: batch{o: str}) { \"", "unterminated"); + assert_parse_rejects( + "fn f(in: batch{a: i64}, out: batch{o: str}) { \"", + "unterminated", + ); } #[test] @@ -462,17 +533,19 @@ fn parser_rejects_wrong_dst_count() { // adversarial workflow (4 attack lenses + 2 reviews). /// Build the smallest valid API program shell around custom parts. -fn api_program( - statics: Vec, - name: &str, - blocks: Vec, -) -> Program { +fn api_program(statics: Vec, name: &str, blocks: Vec) -> Program { use super::{Col, ColTy, Ty}; Program { statics, name: name.into(), in_cols: vec![], - out_cols: vec![Col { name: "o".into(), ty: ColTy { ty: Ty::I64, nullable: false } }], + out_cols: vec![Col { + name: "o".into(), + ty: ColTy { + ty: Ty::I64, + nullable: false, + }, + }], blocks, } } @@ -482,8 +555,14 @@ fn store_emit_block() -> super::Block { Block { params: vec![], insts: vec![ - Inst::Const { dst: Value(0), lit: Lit::I64(1) }, - Inst::Store { col: 0, val: Value(0) }, + Inst::Const { + dst: Value(0), + lit: Lit::I64(1), + }, + Inst::Store { + col: 0, + val: Value(0), + }, ], term: Term::Emit, } @@ -499,7 +578,10 @@ fn deep_cfg_verifies_without_crashing() { .map(|i| Block { params: vec![], insts: vec![], - term: Term::Jump { to: BlockId(i + 1), args: vec![] }, + term: Term::Jump { + to: BlockId(i + 1), + args: vec![], + }, }) .collect(); blocks.push(store_emit_block()); @@ -513,13 +595,20 @@ fn deep_cfg_verifies_without_crashing() { fn rejects_empty_map_static_signatures() { use super::{StaticTy, Ty}; for st in [ - StaticTy::Map { keys: vec![], values: vec![Ty::I64] }, - StaticTy::Map { keys: vec![Ty::I64], values: vec![] }, + StaticTy::Map { + keys: vec![], + values: vec![Ty::I64], + }, + StaticTy::Map { + keys: vec![Ty::I64], + values: vec![], + }, ] { let p = api_program(vec![st], "f", vec![store_emit_block()]); let errs = verify(&p).expect_err("empty map signature must not verify"); assert!( - errs.iter().any(|e| e.to_string().contains("at least one key and one value")), + errs.iter() + .any(|e| e.to_string().contains("at least one key and one value")), "wrong errors: {:?}", errs.iter().map(|e| e.to_string()).collect::>() ); @@ -533,7 +622,8 @@ fn rejects_non_identifier_function_name() { let p = api_program(vec![], bad, vec![store_emit_block()]); let errs = verify(&p).expect_err("non-identifier fn name must not verify"); assert!( - errs.iter().any(|e| e.to_string().contains("must be an identifier")), + errs.iter() + .any(|e| e.to_string().contains("must be an identifier")), "'{bad}': wrong errors: {:?}", errs.iter().map(|e| e.to_string()).collect::>() ); @@ -544,24 +634,40 @@ fn rejects_non_identifier_function_name() { /// treat NaNs as one class or non-canonical payloads break the round-trip. #[test] fn non_canonical_nan_payload_round_trips() { - use super::{Block, Inst, Lit, Term, Value, Col, ColTy, Ty}; + use super::{Block, Col, ColTy, Inst, Lit, Term, Ty, Value}; let neg_quiet_nan = f64::from_bits(0xFFF8_0000_0000_0000); let p = Program { statics: vec![], name: "f".into(), in_cols: vec![], - out_cols: vec![Col { name: "o".into(), ty: ColTy { ty: Ty::F64, nullable: false } }], + out_cols: vec![Col { + name: "o".into(), + ty: ColTy { + ty: Ty::F64, + nullable: false, + }, + }], blocks: vec![Block { params: vec![], insts: vec![ - Inst::Const { dst: Value(0), lit: Lit::F64(neg_quiet_nan) }, - Inst::Store { col: 0, val: Value(0) }, + Inst::Const { + dst: Value(0), + lit: Lit::F64(neg_quiet_nan), + }, + Inst::Store { + col: 0, + val: Value(0), + }, ], term: Term::Emit, }], }; verify(&p).expect("NaN const is legal"); - assert_eq!(parsed(&print(&p)), p, "non-canonical NaN payload broke the round-trip"); + assert_eq!( + parsed(&print(&p)), + p, + "non-canonical NaN payload broke the round-trip" + ); } /// Double stores are a lowering bug on ANY path, including trap-terminated @@ -600,7 +706,10 @@ join: ); let errs = verify(&p).expect_err("island + missing store must not verify"); let all: Vec = errs.iter().map(|e| e.to_string()).collect(); - assert!(all.iter().any(|m| m.contains("unreachable block")), "missing island error: {all:?}"); + assert!( + all.iter().any(|m| m.contains("unreachable block")), + "missing island error: {all:?}" + ); assert!( all.iter().any(|m| m.contains("emit without storing out.o")), "store error masked by the island: {all:?}" @@ -626,13 +735,26 @@ fn rejects_duplicate_columns() { use super::{Col, ColTy, Ty}; let mut p = api_program(vec![], "f", vec![store_emit_block()]); p.out_cols = vec![ - Col { name: "o".into(), ty: ColTy { ty: Ty::I64, nullable: false } }, - Col { name: "o".into(), ty: ColTy { ty: Ty::I64, nullable: false } }, + Col { + name: "o".into(), + ty: ColTy { + ty: Ty::I64, + nullable: false, + }, + }, + Col { + name: "o".into(), + ty: ColTy { + ty: Ty::I64, + nullable: false, + }, + }, ]; // Second column now unstored; the duplicate-name error is what matters. let errs = verify(&p).expect_err("duplicate columns must not verify"); assert!( - errs.iter().any(|e| e.to_string().contains("duplicate out column 'o'")), + errs.iter() + .any(|e| e.to_string().contains("duplicate out column 'o'")), "wrong errors: {:?}", errs.iter().map(|e| e.to_string()).collect::>() ); @@ -736,14 +858,20 @@ fn fuzz_round_trip() { let p = gen::gen_program(seed); if let Err(errs) = verify(&p) { let msgs: Vec = errs.iter().map(|e| e.to_string()).collect(); - panic!("generator produced an invalid program (seed {seed}): {msgs:?}\n{}", print(&p)); + panic!( + "generator produced an invalid program (seed {seed}): {msgs:?}\n{}", + print(&p) + ); } let text = print(&p); let p2 = match parse(&text) { Ok(p2) => p2, Err(e) => panic!("canonical text failed to parse (seed {seed}): {e}\n{text}"), }; - assert_eq!(p2, p, "round-trip changed the program (seed {seed}):\n{text}"); + assert_eq!( + p2, p, + "round-trip changed the program (seed {seed}):\n{text}" + ); assert_eq!(print(&p2), text, "printing is not a fixpoint (seed {seed})"); } } diff --git a/src/specializer/ir/verify.rs b/src/specializer/ir/verify.rs index 968dbed..fbe44a7 100644 --- a/src/specializer/ir/verify.rs +++ b/src/specializer/ir/verify.rs @@ -100,13 +100,23 @@ fn check_structure(p: &Program, errs: &mut Vec) { return; } if !p.blocks[0].params.is_empty() { - err(errs, Some(0), None, "entry block cannot have params".to_string()); + err( + errs, + Some(0), + None, + "entry block cannot have params".to_string(), + ); } for (side, cols) in [("in", &p.in_cols), ("out", &p.out_cols)] { let mut seen: HashMap<&str, ()> = HashMap::new(); for c in cols.iter() { if seen.insert(c.name.as_str(), ()).is_some() { - err(errs, None, None, format!("duplicate {side} column '{}'", c.name)); + err( + errs, + None, + None, + format!("duplicate {side} column '{}'", c.name), + ); } } } @@ -126,13 +136,23 @@ fn collect_defs( for (bi, b) in p.blocks.iter().enumerate() { for (v, ty) in &b.params { if def_types.insert(v.0, (*ty, bi)).is_some() { - err(errs, Some(bi), None, format!("%v{} defined more than once", v.0)); + err( + errs, + Some(bi), + None, + format!("%v{} defined more than once", v.0), + ); } } for (ii, inst) in b.insts.iter().enumerate() { for (dst, ty) in dst_types(p, inst) { if def_types.insert(dst.0, (ty, bi)).is_some() { - err(errs, Some(bi), Some(ii), format!("%v{} defined more than once", dst.0)); + err( + errs, + Some(bi), + Some(ii), + format!("%v{} defined more than once", dst.0), + ); } } } @@ -156,9 +176,15 @@ fn dst_types(p: &Program, inst: &Inst) -> Vec<(Value, Ty)> { Inst::Select { dst, .. } => vec![(*dst, Ty::I1)], Inst::Itof { dst, .. } => vec![(*dst, Ty::F64)], Inst::Ftoi { dst, .. } => vec![(*dst, Ty::I64)], - Inst::Itos { dst, .. } | Inst::Ftos { dst, .. } | Inst::Sconcat { dst, .. } => { + Inst::Itos { dst, .. } + | Inst::Ftos { dst, .. } + | Inst::Sconcat { dst, .. } + | Inst::Str1 { dst, .. } + | Inst::Strim { dst, .. } + | Inst::Ssubstr { dst, .. } => { vec![(*dst, Ty::Str)] } + Inst::Num1 { op, dst, .. } => vec![(*dst, op.sig())], Inst::StoiOpt { flag, dst, .. } => vec![(*flag, Ty::I1), (*dst, Ty::I64)], Inst::StofOpt { flag, dst, .. } => vec![(*flag, Ty::I1), (*dst, Ty::F64)], Inst::Load { dst, col } => vec![(*dst, in_col(*col).unwrap_or(Ty::I1))], @@ -166,7 +192,12 @@ fn dst_types(p: &Program, inst: &Inst) -> Vec<(Value, Ty)> { vec![(*flag, Ty::I1), (*dst, in_col(*col).unwrap_or(Ty::I1))] } Inst::Store { .. } | Inst::StoreOpt { .. } => vec![], - Inst::Probe { static_id, hit, dsts, .. } => { + Inst::Probe { + static_id, + hit, + dsts, + .. + } => { let mut v = vec![(*hit, Ty::I1)]; if let Some(StaticTy::Map { values, .. }) = p.statics.get(*static_id as usize) { for (d, ty) in dsts.iter().zip(values.iter()) { @@ -178,7 +209,11 @@ fn dst_types(p: &Program, inst: &Inst) -> Vec<(Value, Ty)> { v } Inst::Sload { static_id, dst } => vec![(*dst, scalar_ty(*static_id))], - Inst::SloadOpt { static_id, flag, dst } => { + Inst::SloadOpt { + static_id, + flag, + dst, + } => { vec![(*flag, Ty::I1), (*dst, scalar_ty(*static_id))] } } @@ -230,7 +265,12 @@ fn want( errs, Some(bi), ii, - format!("{what} %v{} must be {}, got {}", v.0, ty.name(), actual.name()), + format!( + "{what} %v{} must be {}, got {}", + v.0, + ty.name(), + actual.name() + ), ); } } @@ -262,16 +302,33 @@ fn check_block( } Inst::Cmp { ty, a, b: rhs, .. } => { if *ty == Ty::I1 { - err(errs, Some(bi), i, "cmp on i1 is not defined; use xor/not".into()); + err( + errs, + Some(bi), + i, + "cmp on i1 is not defined; use xor/not".into(), + ); } want(&in_scope, def_types, *a, *ty, "operand", bi, i, errs); want(&in_scope, def_types, *rhs, *ty, "operand", bi, i, errs); } - Inst::Not { a, .. } => { - want(&in_scope, def_types, *a, Ty::I1, "operand", bi, i, errs) - } - Inst::Select { dst, cond, a, b: rhs } => { - want(&in_scope, def_types, *cond, Ty::I1, "condition", bi, i, errs); + Inst::Not { a, .. } => want(&in_scope, def_types, *a, Ty::I1, "operand", bi, i, errs), + Inst::Select { + dst, + cond, + a, + b: rhs, + } => { + want( + &in_scope, + def_types, + *cond, + Ty::I1, + "condition", + bi, + i, + errs, + ); let ta = scope_ty(&in_scope, def_types, *a, "operand", bi, i, errs); let tb = scope_ty(&in_scope, def_types, *rhs, "operand", bi, i, errs); if let (Some(ta), Some(tb)) = (ta, tb) { @@ -289,114 +346,184 @@ fn check_block( in_scope.insert(dst.0, ta); } } - Inst::Itof { a, .. } => { - want(&in_scope, def_types, *a, Ty::I64, "operand", bi, i, errs) - } - Inst::Ftoi { a, .. } => { - want(&in_scope, def_types, *a, Ty::F64, "operand", bi, i, errs) - } - Inst::Itos { a, .. } => { - want(&in_scope, def_types, *a, Ty::I64, "operand", bi, i, errs) - } - Inst::Ftos { a, .. } => { - want(&in_scope, def_types, *a, Ty::F64, "operand", bi, i, errs) - } + Inst::Itof { a, .. } => want(&in_scope, def_types, *a, Ty::I64, "operand", bi, i, errs), + Inst::Ftoi { a, .. } => want(&in_scope, def_types, *a, Ty::F64, "operand", bi, i, errs), + Inst::Itos { a, .. } => want(&in_scope, def_types, *a, Ty::I64, "operand", bi, i, errs), + Inst::Ftos { a, .. } => want(&in_scope, def_types, *a, Ty::F64, "operand", bi, i, errs), Inst::StoiOpt { a, .. } | Inst::StofOpt { a, .. } => { want(&in_scope, def_types, *a, Ty::Str, "operand", bi, i, errs) } - Inst::Sconcat { a, b: rhs, .. } => { + Inst::Sconcat { a, b: rhs, .. } | Inst::Strim { a, chars: rhs, .. } => { want(&in_scope, def_types, *a, Ty::Str, "operand", bi, i, errs); want(&in_scope, def_types, *rhs, Ty::Str, "operand", bi, i, errs); } + Inst::Str1 { a, .. } => want(&in_scope, def_types, *a, Ty::Str, "operand", bi, i, errs), + Inst::Ssubstr { a, start, len, .. } => { + want(&in_scope, def_types, *a, Ty::Str, "operand", bi, i, errs); + want( + &in_scope, + def_types, + *start, + Ty::I64, + "operand", + bi, + i, + errs, + ); + if let Some(len) = len { + want(&in_scope, def_types, *len, Ty::I64, "operand", bi, i, errs); + } + } + Inst::Num1 { op, a, .. } => { + want(&in_scope, def_types, *a, op.sig(), "operand", bi, i, errs) + } Inst::Load { col, .. } => match p.in_cols.get(*col as usize) { None => err(errs, Some(bi), i, format!("unknown in column {col}")), - Some(Col { ty, name }) if ty.nullable => { - err(errs, Some(bi), i, format!("in.{name} is nullable: use load.opt")) - } + Some(Col { ty, name }) if ty.nullable => err( + errs, + Some(bi), + i, + format!("in.{name} is nullable: use load.opt"), + ), Some(_) => {} }, Inst::LoadOpt { col, .. } => match p.in_cols.get(*col as usize) { None => err(errs, Some(bi), i, format!("unknown in column {col}")), - Some(Col { ty, name }) if !ty.nullable => { - err(errs, Some(bi), i, format!("in.{name} is not nullable: use load")) - } + Some(Col { ty, name }) if !ty.nullable => err( + errs, + Some(bi), + i, + format!("in.{name} is not nullable: use load"), + ), Some(_) => {} }, Inst::Store { col, val } => match p.out_cols.get(*col as usize) { None => err(errs, Some(bi), i, format!("unknown out column {col}")), - Some(c) if c.ty.nullable => { - err(errs, Some(bi), i, format!("out.{} is nullable: use store.opt", c.name)) - } - Some(c) => { - want(&in_scope, def_types, *val, c.ty.ty, "stored value", bi, i, errs) - } + Some(c) if c.ty.nullable => err( + errs, + Some(bi), + i, + format!("out.{} is nullable: use store.opt", c.name), + ), + Some(c) => want( + &in_scope, + def_types, + *val, + c.ty.ty, + "stored value", + bi, + i, + errs, + ), }, Inst::StoreOpt { col, flag, val } => match p.out_cols.get(*col as usize) { None => err(errs, Some(bi), i, format!("unknown out column {col}")), - Some(c) if !c.ty.nullable => { - err(errs, Some(bi), i, format!("out.{} is not nullable: use store", c.name)) - } + Some(c) if !c.ty.nullable => err( + errs, + Some(bi), + i, + format!("out.{} is not nullable: use store", c.name), + ), Some(c) => { - want(&in_scope, def_types, *flag, Ty::I1, "validity flag", bi, i, errs); - want(&in_scope, def_types, *val, c.ty.ty, "stored value", bi, i, errs); + want( + &in_scope, + def_types, + *flag, + Ty::I1, + "validity flag", + bi, + i, + errs, + ); + want( + &in_scope, + def_types, + *val, + c.ty.ty, + "stored value", + bi, + i, + errs, + ); } }, - Inst::Probe { static_id, dsts, keys, .. } => { - match p.statics.get(*static_id as usize) { - None => err(errs, Some(bi), i, format!("unknown static @{static_id}")), - Some(StaticTy::Scalar(_)) => { - err(errs, Some(bi), i, format!("@{static_id} is a scalar: use sload")) - } - Some(StaticTy::Map { keys: kts, values: vts }) => { - if keys.len() != kts.len() { - err( - errs, - Some(bi), - i, - format!( - "@{static_id} has {} key(s), probe passes {}", - kts.len(), - keys.len() - ), - ); - } else { - for (k, kt) in keys.iter().zip(kts.iter()) { - want(&in_scope, def_types, *k, *kt, "probe key", bi, i, errs); - } - } - if dsts.len() != vts.len() { - err( - errs, - Some(bi), - i, - format!( - "@{static_id} has {} value column(s), probe defines {}", - vts.len(), - dsts.len() - ), - ); + Inst::Probe { + static_id, + dsts, + keys, + .. + } => match p.statics.get(*static_id as usize) { + None => err(errs, Some(bi), i, format!("unknown static @{static_id}")), + Some(StaticTy::Scalar(_)) => err( + errs, + Some(bi), + i, + format!("@{static_id} is a scalar: use sload"), + ), + Some(StaticTy::Map { + keys: kts, + values: vts, + }) => { + if keys.len() != kts.len() { + err( + errs, + Some(bi), + i, + format!( + "@{static_id} has {} key(s), probe passes {}", + kts.len(), + keys.len() + ), + ); + } else { + for (k, kt) in keys.iter().zip(kts.iter()) { + want(&in_scope, def_types, *k, *kt, "probe key", bi, i, errs); } } + if dsts.len() != vts.len() { + err( + errs, + Some(bi), + i, + format!( + "@{static_id} has {} value column(s), probe defines {}", + vts.len(), + dsts.len() + ), + ); + } } - } + }, Inst::Sload { static_id, .. } => match p.statics.get(*static_id as usize) { None => err(errs, Some(bi), i, format!("unknown static @{static_id}")), - Some(StaticTy::Map { .. }) => { - err(errs, Some(bi), i, format!("@{static_id} is a map: use probe")) - } - Some(StaticTy::Scalar(ct)) if ct.nullable => { - err(errs, Some(bi), i, format!("@{static_id} is nullable: use sload.opt")) - } + Some(StaticTy::Map { .. }) => err( + errs, + Some(bi), + i, + format!("@{static_id} is a map: use probe"), + ), + Some(StaticTy::Scalar(ct)) if ct.nullable => err( + errs, + Some(bi), + i, + format!("@{static_id} is nullable: use sload.opt"), + ), Some(_) => {} }, Inst::SloadOpt { static_id, .. } => match p.statics.get(*static_id as usize) { None => err(errs, Some(bi), i, format!("unknown static @{static_id}")), - Some(StaticTy::Map { .. }) => { - err(errs, Some(bi), i, format!("@{static_id} is a map: use probe")) - } - Some(StaticTy::Scalar(ct)) if !ct.nullable => { - err(errs, Some(bi), i, format!("@{static_id} is not nullable: use sload")) - } + Some(StaticTy::Map { .. }) => err( + errs, + Some(bi), + i, + format!("@{static_id} is a map: use probe"), + ), + Some(StaticTy::Scalar(ct)) if !ct.nullable => err( + errs, + Some(bi), + i, + format!("@{static_id} is not nullable: use sload"), + ), Some(_) => {} }, } @@ -411,11 +538,25 @@ fn check_block( // Terminator: cond and branch args are uses in this block's final scope. if let Term::Brif { cond, .. } = &b.term { - want(&in_scope, def_types, *cond, Ty::I1, "branch condition", bi, None, errs); + want( + &in_scope, + def_types, + *cond, + Ty::I1, + "branch condition", + bi, + None, + errs, + ); } for (target, args) in b.term.successors() { match p.blocks.get(target.0 as usize) { - None => err(errs, Some(bi), None, format!("branch to unknown block b{}", target.0)), + None => err( + errs, + Some(bi), + None, + format!("branch to unknown block b{}", target.0), + ), Some(tb) => { if args.len() != tb.params.len() { err( @@ -431,7 +572,16 @@ fn check_block( ); } else { for (arg, (_, pty)) in args.iter().zip(tb.params.iter()) { - want(&in_scope, def_types, *arg, *pty, "branch arg", bi, None, errs); + want( + &in_scope, + def_types, + *arg, + *pty, + "branch arg", + bi, + None, + errs, + ); } } if target.0 == 0 { @@ -480,7 +630,12 @@ fn check_cfg_and_stores(p: &Program, errs: &mut Vec) { } } if cyclic { - err(errs, None, None, "control-flow cycle (v0 CFGs must be acyclic)".to_string()); + err( + errs, + None, + None, + "control-flow cycle (v0 CFGs must be acyclic)".to_string(), + ); return; // the store dataflow below needs a DAG } let reachable: Vec = color.iter().map(|c| *c != 0).collect(); @@ -514,7 +669,10 @@ fn check_cfg_and_stores(p: &Program, errs: &mut Vec) { errs, Some(bi), Some(ii), - format!("out.{} stored more than once on this path", p.out_cols[col].name), + format!( + "out.{} stored more than once on this path", + p.out_cols[col].name + ), ); } } @@ -587,7 +745,9 @@ fn topo_order(p: &Program, n: usize, reachable: &[bool]) -> Vec { } } } - let mut stack: Vec = (0..n).filter(|&i| reachable[i] && indegree[i] == 0).collect(); + let mut stack: Vec = (0..n) + .filter(|&i| reachable[i] && indegree[i] == 0) + .collect(); let mut order = Vec::with_capacity(n); while let Some(b) = stack.pop() { order.push(b); diff --git a/src/specializer/lower.rs b/src/specializer/lower.rs new file mode 100644 index 0000000..a7ced09 --- /dev/null +++ b/src/specializer/lower.rs @@ -0,0 +1,1113 @@ +//! Produce/consume lowering: relational IR -> imperative IR. +//! +//! The central mechanism is the env-threading function builder [`FB`]: +//! CASE and guarded CASTs split the CFG, and the IR's strict block-param SSA +//! (values never cross blocks except as branch args) means every live value +//! must ride the branch when a split happens. `emit` therefore carries a +//! live stack: recursive children push their partial lanes before a sibling +//! is evaluated, and any block transition rewrites the stack in place to the +//! target block's params. Columns are NOT threaded — loads are pure and each +//! block re-loads through its own cache. +//! +//! Null-lane discipline: an SExpr with `nullable == false` lowers to a bare +//! payload register (no flag anywhere); a nullable one carries an `i1` flag, +//! combined with `and` where NULL propagates. Kleene AND/OR are lowered +//! branchless from flag algebra. WHERE keeps a row iff the predicate is +//! TRUE — flag && value. +//! +//! CASE lowers to a condition chain with a join block; branch results are +//! evaluated only on their taken path (a guarded `1/0`-style branch must not +//! trap eagerly — SQL semantics, verified by test). CAST failure is a +//! conditional trap block; TRY_CAST folds the failure into the null lane. +//! +//! ponytail: blocks re-load columns instead of receiving them as branch args +//! — pure loads, identical semantics, simpler lowering. Thread them when the +//! Cranelift backend makes the extra reads measurable. + +use std::collections::HashMap; + +use super::frontend::PrepareError; +use super::ir::{ + BinOp, Block, BlockId, Builder, CmpPred, Col, Inst, Lit, NumOp1, Program, StaticTy, StrOp1, + Term, Ty, Value, +}; +use super::plan::{ArithOp, JoinKind, JoinSpec, Rel, SExpr, SKind, StaticTable}; + +pub fn lower( + rel: &Rel, + joins: &[JoinSpec], + catalog: &[StaticTable], + in_cols: &[Col], + out_cols: Vec, + name: &str, +) -> Result { + let (exprs, filter_pred) = match rel { + Rel::Project { input, exprs } => match input.as_ref() { + Rel::Filter { input: scan, pred } => { + debug_assert!(matches!(scan.as_ref(), Rel::Scan)); + (exprs, Some(pred)) + } + Rel::Scan => (exprs, None), + Rel::Project { .. } => { + return Err(PrepareError::Internal("nested projection".to_string())) + } + }, + _ => { + return Err(PrepareError::Internal( + "plan root is not a projection".to_string(), + )) + } + }; + + let mut fb = FB::new(in_cols, joins); + + // Joins run before WHERE (SQL order), each in FROM order: probe, and for + // INNER skip the row on a miss. A LEFT join's probe is also forced here + // so its key expressions are evaluated (and can trap) for every row that + // reaches it, exactly as the join would — even if no value column is + // ever referenced. Later references re-probe per block (pure, cached), + // same ponytail trade as column re-loads. + for (j, spec) in joins.iter().enumerate() { + let mut live = Vec::new(); + let (valid_hit, _) = fb.emit_probe(j as u32, &mut live)?; + if spec.kind == JoinKind::Inner { + let (keep, _) = fb.create_block(&[]); + let (miss, _) = fb.create_block(&[]); + fb.term(Term::Brif { + cond: valid_hit, + then_to: BlockId(keep as u32), + then_args: vec![], + else_to: BlockId(miss as u32), + else_args: vec![], + }); + fb.switch(miss); + fb.term(Term::Skip); + fb.switch(keep); + } + } + + if let Some(pred) = filter_pred { + let mut live = Vec::new(); + let pl = fb.emit(pred, &mut live)?; + let cond = fb.truthy(pl); + let (keep, _) = fb.create_block(&[]); + let (drop, _) = fb.create_block(&[]); + fb.term(Term::Brif { + cond, + then_to: BlockId(keep as u32), + then_args: vec![], + else_to: BlockId(drop as u32), + else_args: vec![], + }); + fb.switch(drop); + fb.term(Term::Skip); + fb.switch(keep); + } + + let mut live = Vec::new(); + for (ci, (_, e)) in exprs.iter().enumerate() { + debug_assert!(live.is_empty()); + let lane = fb.emit(e, &mut live)?; + let col = ci as u32; + if out_cols[ci].ty.nullable { + let flag = match lane.flag { + Some(f) => f, + // Nullability contract slack (e.g. an infallible TRY_CAST): + // the column is declared nullable, the lane is provably + // valid — store with a constant true flag. + None => fb.const_i1(true), + }; + fb.inst(Inst::StoreOpt { + col, + flag, + val: lane.val, + }); + } else { + debug_assert!(lane.flag.is_none(), "non-nullable column with a flag lane"); + fb.inst(Inst::Store { col, val: lane.val }); + } + } + fb.term(Term::Emit); + + // Map static @j belongs to join j: keyed by the (already promoted) key + // expression types, valued by the join's value columns. + let statics = joins + .iter() + .map(|spec| StaticTy::Map { + keys: spec.keys.iter().map(|k| k.ty).collect(), + values: spec + .val_cols + .iter() + .map(|&c| catalog[spec.table].cols[c as usize].ty.ty) + .collect(), + }) + .collect(); + + fb.finish(name, statics, in_cols, out_cols) +} + +/// A value in the null-lane representation: payload + optional validity. +/// `flag: None` means provably non-NULL (no flag register exists). +#[derive(Clone, Copy)] +struct Lane { + flag: Option, + val: Value, +} + +/// The live stack: lanes (with their payload types) that must survive block +/// transitions triggered while a sibling expression is being emitted. +type Live = Vec<(Lane, Ty)>; + +/// A block under construction. +struct PB { + params: Vec<(Value, Ty)>, + insts: Vec, + term: Option, + cache: HashMap, + /// Probes already emitted in this block: join index -> (valid hit — + /// map hit AND every key flag — and the value-column registers). + probes: HashMap)>, +} + +impl PB { + fn new(params: Vec<(Value, Ty)>) -> PB { + PB { + params, + insts: vec![], + term: None, + cache: HashMap::new(), + probes: HashMap::new(), + } + } +} + +/// Env-threading function builder over the strict block-param SSA IR. +struct FB<'a> { + b: Builder, + blocks: Vec, + cur: usize, + in_cols: &'a [Col], + joins: &'a [JoinSpec], +} + +impl<'a> FB<'a> { + fn new(in_cols: &'a [Col], joins: &'a [JoinSpec]) -> FB<'a> { + FB { + b: Builder::new(), + blocks: vec![PB::new(vec![])], + cur: 0, + in_cols, + joins, + } + } + + fn fresh(&mut self) -> Value { + self.b.fresh() + } + + fn inst(&mut self, i: Inst) { + self.blocks[self.cur].insts.push(i); + } + + fn term(&mut self, t: Term) { + let slot = &mut self.blocks[self.cur].term; + debug_assert!(slot.is_none(), "block terminated twice"); + *slot = Some(t); + } + + fn switch(&mut self, blk: usize) { + self.cur = blk; + } + + fn create_block(&mut self, param_tys: &[Ty]) -> (usize, Vec) { + let params: Vec<(Value, Ty)> = param_tys.iter().map(|t| (self.b.fresh(), *t)).collect(); + let vals = params.iter().map(|(v, _)| *v).collect(); + self.blocks.push(PB::new(params)); + (self.blocks.len() - 1, vals) + } + + fn finish( + self, + name: &str, + statics: Vec, + in_cols: &[Col], + out_cols: Vec, + ) -> Result { + let mut blocks = Vec::with_capacity(self.blocks.len()); + for (i, pb) in self.blocks.into_iter().enumerate() { + let term = pb + .term + .ok_or_else(|| PrepareError::Internal(format!("block b{i} left unterminated")))?; + blocks.push(Block { + params: pb.params, + insts: pb.insts, + term, + }); + } + Ok(Program { + statics, + name: name.to_string(), + in_cols: in_cols.to_vec(), + out_cols, + blocks, + }) + } + + // ------------------------------------------------------ conveniences -- + + fn const_lit(&mut self, lit: Lit) -> Value { + let dst = self.fresh(); + self.inst(Inst::Const { dst, lit }); + dst + } + + fn const_i1(&mut self, v: bool) -> Value { + self.const_lit(Lit::I1(v)) + } + + /// Trapping instructions must never fire on the garbage payload under a + /// false NULL flag — computed garbage is unbounded (`x + MAX + MAX` with + /// x NULL overflows its payload). Mask nullable payloads to the type + /// default right before any trapping instruction. + fn masked(&mut self, l: Lane, ty: Ty) -> Value { + match l.flag { + None => l.val, + Some(f) => { + let d = self.default_of(ty); + let dst = self.fresh(); + self.inst(Inst::Select { + dst, + cond: f, + a: l.val, + b: d, + }); + dst + } + } + } + + fn default_of(&mut self, ty: Ty) -> Value { + self.const_lit(match ty { + Ty::I1 => Lit::I1(false), + Ty::I64 => Lit::I64(0), + Ty::F64 => Lit::F64(0.0), + Ty::Str => Lit::Str(String::new()), + }) + } + + fn bin(&mut self, op: BinOp, a: Value, b: Value) -> Value { + let dst = self.fresh(); + self.inst(Inst::Bin { op, dst, a, b }); + dst + } + + fn not(&mut self, a: Value) -> Value { + let dst = self.fresh(); + self.inst(Inst::Not { dst, a }); + dst + } + + /// The SQL truth of a boolean lane: TRUE iff valid AND value. + fn truthy(&mut self, lane: Lane) -> Value { + match lane.flag { + None => lane.val, + Some(f) => self.bin(BinOp::And, f, lane.val), + } + } + + // -------------------------------------------------- live threading -- + + /// The flattened param type shape of a live stack: per lane, an i1 flag + /// (when present) then the payload. + fn live_types(live: &Live) -> Vec { + let mut tys = Vec::new(); + for (lane, ty) in live { + if lane.flag.is_some() { + tys.push(Ty::I1); + } + tys.push(*ty); + } + tys + } + + fn live_args(live: &Live) -> Vec { + let mut args = Vec::new(); + for (lane, _) in live { + if let Some(f) = lane.flag { + args.push(f); + } + args.push(lane.val); + } + args + } + + /// Rebind the live stack to the params of the block just switched to. + /// The shape (lane count + flag pattern) is invariant across transitions. + fn rebind_live(live: &mut Live, params: &[Value]) { + let mut i = 0; + for (lane, _) in live.iter_mut() { + if lane.flag.is_some() { + lane.flag = Some(params[i]); + i += 1; + } + lane.val = params[i]; + i += 1; + } + debug_assert_eq!(i, params.len()); + } + + // ------------------------------------------------------- expressions -- + + fn emit(&mut self, e: &SExpr, live: &mut Live) -> Result { + match &e.kind { + SKind::Col(idx) => { + if let Some(lane) = self.blocks[self.cur].cache.get(idx) { + return Ok(*lane); + } + let col = &self.in_cols[*idx as usize]; + let lane = if col.ty.nullable { + let flag = self.fresh(); + let val = self.fresh(); + self.inst(Inst::LoadOpt { + flag, + dst: val, + col: *idx, + }); + Lane { + flag: Some(flag), + val, + } + } else { + let val = self.fresh(); + self.inst(Inst::Load { + dst: val, + col: *idx, + }); + Lane { flag: None, val } + }; + self.blocks[self.cur].cache.insert(*idx, lane); + Ok(lane) + } + SKind::StaticCol { join, col } => { + let (valid_hit, dsts) = self.emit_probe(*join, live)?; + let flag = match self.joins[*join as usize].kind { + // INNER: a miss already skipped the row before any + // expression could look — the lane is provably valid. + JoinKind::Inner => None, + JoinKind::Left => Some(valid_hit), + }; + Ok(Lane { + flag, + val: dsts[*col as usize], + }) + } + SKind::Lit(lit) => Ok(Lane { + flag: None, + val: self.const_lit(lit.clone()), + }), + SKind::NullOf => { + let flag = self.const_i1(false); + let val = self.default_of(e.ty); + Ok(Lane { + flag: Some(flag), + val, + }) + } + SKind::IntToFloat(inner) => { + let l = self.emit(inner, live)?; + let dst = self.fresh(); + self.inst(Inst::Itof { dst, a: l.val }); + Ok(Lane { + flag: l.flag, + val: dst, + }) + } + SKind::Arith { op, a, b } => { + let la = self.emit(a, live)?; + live.push((la, a.ty)); + let lb = self.emit(b, live)?; + let (la, _) = live.pop().expect("pushed above"); + let ir_op = match (op, e.ty) { + (ArithOp::Add, Ty::I64) => BinOp::Iadd, + (ArithOp::Sub, Ty::I64) => BinOp::Isub, + (ArithOp::Mul, Ty::I64) => BinOp::Imul, + (ArithOp::Rem, Ty::I64) => BinOp::Irem, + (ArithOp::Add, Ty::F64) => BinOp::Fadd, + (ArithOp::Sub, Ty::F64) => BinOp::Fsub, + (ArithOp::Mul, Ty::F64) => BinOp::Fmul, + (ArithOp::Div, Ty::F64) => BinOp::Fdiv, + (ArithOp::Rem, Ty::F64) => BinOp::Frem, + (op, ty) => { + return Err(PrepareError::Internal(format!( + "arith {op:?} on {} escaped the frontend", + ty.name() + ))) + } + }; + // Integer arithmetic traps (overflow, % edge cases): mask + // nullable payloads so garbage under a false flag can never + // fire the trap. Float ops are total — no masking needed. + let (va, vb) = if e.ty == Ty::I64 { + (self.masked(la, Ty::I64), self.masked(lb, Ty::I64)) + } else { + (la.val, lb.val) + }; + let val = self.bin(ir_op, va, vb); + Ok(Lane { + flag: self.combine_flags(la.flag, lb.flag), + val, + }) + } + SKind::Cmp { pred, a, b } => { + let la = self.emit(a, live)?; + live.push((la, a.ty)); + let lb = self.emit(b, live)?; + let (la, _) = live.pop().expect("pushed above"); + let dst = self.fresh(); + self.inst(Inst::Cmp { + pred: *pred, + ty: a.ty, + dst, + a: la.val, + b: lb.val, + }); + Ok(Lane { + flag: self.combine_flags(la.flag, lb.flag), + val: dst, + }) + } + SKind::Not(inner) => { + let l = self.emit(inner, live)?; + let val = self.not(l.val); + Ok(Lane { flag: l.flag, val }) + } + SKind::And { a, b } => self.kleene(a, b, live, true), + SKind::Or { a, b } => self.kleene(a, b, live, false), + SKind::IsNull { negated, inner } => { + let l = self.emit(inner, live)?; + let val = match l.flag { + None => self.const_i1(*negated), + Some(f) => { + if *negated { + f + } else { + self.not(f) + } + } + }; + Ok(Lane { flag: None, val }) + } + SKind::Case { arms, default } => self.case(e, arms, default.as_deref(), live), + SKind::Cast { inner, trying } => self.cast(e, inner, *trying, live), + SKind::StrCase { upper, a } => { + let l = self.emit(a, live)?; + let dst = self.fresh(); + let op = if *upper { StrOp1::Upper } else { StrOp1::Lower }; + self.inst(Inst::Str1 { op, dst, a: l.val }); + Ok(Lane { + flag: l.flag, + val: dst, + }) + } + SKind::Trim { side, a, chars } => { + let la = self.emit(a, live)?; + live.push((la, Ty::Str)); + let lc = self.emit(chars, live)?; + let (la, _) = live.pop().expect("pushed above"); + let dst = self.fresh(); + self.inst(Inst::Strim { + side: *side, + dst, + a: la.val, + chars: lc.val, + }); + Ok(Lane { + flag: self.combine_flags(la.flag, lc.flag), + val: dst, + }) + } + SKind::Substr { a, start, len } => { + let la = self.emit(a, live)?; + live.push((la, Ty::Str)); + let ls = self.emit(start, live)?; + live.push((ls, Ty::I64)); + let ll = match len { + Some(l) => Some(self.emit(l, live)?), + None => None, + }; + let (ls, _) = live.pop().expect("pushed above"); + let (la, _) = live.pop().expect("pushed above"); + // The range guards trap; mask nullable position payloads. + let start_v = self.masked(ls, Ty::I64); + let len_v = ll.map(|l| self.masked(l, Ty::I64)); + let dst = self.fresh(); + self.inst(Inst::Ssubstr { + dst, + a: la.val, + start: start_v, + len: len_v, + }); + let flag = self.combine_flags(la.flag, ls.flag); + Ok(Lane { + flag: self.combine_flags(flag, ll.and_then(|l| l.flag)), + val: dst, + }) + } + SKind::Abs(a) => { + let l = self.emit(a, live)?; + // iabs traps on i64::MIN; mask the nullable payload. + let (op, av) = if e.ty == Ty::I64 { + (NumOp1::Iabs, self.masked(l, Ty::I64)) + } else { + (NumOp1::Fabs, l.val) + }; + let dst = self.fresh(); + self.inst(Inst::Num1 { op, dst, a: av }); + Ok(Lane { + flag: l.flag, + val: dst, + }) + } + SKind::Round(a) => { + let l = self.emit(a, live)?; + let dst = self.fresh(); + self.inst(Inst::Num1 { + op: NumOp1::Fround, + dst, + a: l.val, + }); + Ok(Lane { + flag: l.flag, + val: dst, + }) + } + SKind::Concat { a, b } => { + let la = self.emit(a, live)?; + live.push((la, Ty::Str)); + let lb = self.emit(b, live)?; + let (la, _) = live.pop().expect("pushed above"); + let dst = self.fresh(); + self.inst(Inst::Sconcat { + dst, + a: la.val, + b: lb.val, + }); + Ok(Lane { + flag: self.combine_flags(la.flag, lb.flag), + val: dst, + }) + } + } + } + + /// Emit (or reuse) join `j`'s probe in the current block. The probe is + /// pure — same keys, same row, same result — so blocks re-probe rather + /// than thread probe lanes through branch args. Returns the valid-hit + /// flag (map hit AND every nullable key's validity: a NULL key never + /// matches, and a garbage payload under a false flag must not spuriously + /// hit) plus the value-column registers. + fn emit_probe(&mut self, j: u32, live: &mut Live) -> Result<(Value, Vec), PrepareError> { + if let Some((valid_hit, dsts)) = self.blocks[self.cur].probes.get(&j) { + return Ok((*valid_hit, dsts.clone())); + } + let spec = &self.joins[j as usize]; + let nkeys = spec.keys.len(); + for key in &spec.keys { + let lane = self.emit(key, live)?; + live.push((lane, key.ty)); + } + let mut keys_valid: Option = None; + let mut key_vals = Vec::with_capacity(nkeys); + for (lane, _) in &live[live.len() - nkeys..] { + key_vals.push(lane.val); + if let Some(f) = lane.flag { + keys_valid = self.combine_flags(keys_valid, Some(f)); + } + } + live.truncate(live.len() - nkeys); + + let spec = &self.joins[j as usize]; + let hit = self.fresh(); + let dsts: Vec = spec.val_cols.iter().map(|_| self.b.fresh()).collect(); + self.inst(Inst::Probe { + static_id: j, + hit, + dsts: dsts.clone(), + keys: key_vals, + }); + let valid_hit = match keys_valid { + None => hit, + Some(f) => self.bin(BinOp::And, f, hit), + }; + self.blocks[self.cur] + .probes + .insert(j, (valid_hit, dsts.clone())); + Ok((valid_hit, dsts)) + } + + /// Branchless Kleene AND/OR from flag algebra. With the lane contract + /// (payloads under a false flag may be garbage), every value read is + /// guarded by its flag. + fn kleene( + &mut self, + a: &SExpr, + b: &SExpr, + live: &mut Live, + is_and: bool, + ) -> Result { + let la = self.emit(a, live)?; + live.push((la, Ty::I1)); + let lb = self.emit(b, live)?; + let (la, _) = live.pop().expect("pushed above"); + + let op = if is_and { BinOp::And } else { BinOp::Or }; + let val = self.bin(op, la.val, lb.val); + let flag = match (la.flag, lb.flag) { + (None, None) => None, + // One side definite: the result is known when the definite side + // decides (false for AND, true for OR), or when the other side + // is valid. + (Some(f), None) => { + let decides = if is_and { self.not(lb.val) } else { lb.val }; + Some(self.bin(BinOp::Or, f, decides)) + } + (None, Some(f)) => { + let decides = if is_and { self.not(la.val) } else { la.val }; + Some(self.bin(BinOp::Or, f, decides)) + } + (Some(fa), Some(fb)) => { + // Known when: both valid, or either side validly decides. + let both = self.bin(BinOp::And, fa, fb); + let da = if is_and { self.not(la.val) } else { la.val }; + let da = self.bin(BinOp::And, fa, da); + let db = if is_and { self.not(lb.val) } else { lb.val }; + let db = self.bin(BinOp::And, fb, db); + let t = self.bin(BinOp::Or, both, da); + Some(self.bin(BinOp::Or, t, db)) + } + }; + Ok(Lane { flag, val }) + } + + /// CASE chain: one condition block per arm, results evaluated only on + /// their taken path, all paths joining with the result lane as params. + fn case( + &mut self, + e: &SExpr, + arms: &[(SExpr, SExpr)], + default: Option<&SExpr>, + live: &mut Live, + ) -> Result { + let res_ty = e.ty; + let res_nullable = e.nullable; + + let mut join_tys = Self::live_types(live); + if res_nullable { + join_tys.push(Ty::I1); + } + join_tys.push(res_ty); + let live_width = Self::live_types(live).len(); + let (join, join_params) = self.create_block(&join_tys); + + let finish_branch = |fb: &mut FB<'a>, lane: Lane, live: &Live| -> Term { + let mut args = Self::live_args(live); + if res_nullable { + let flag = match lane.flag { + Some(f) => f, + None => fb.const_i1(true), + }; + args.push(flag); + } + args.push(lane.val); + Term::Jump { + to: BlockId(join as u32), + args, + } + }; + + for (cond, result) in arms { + let cl = self.emit(cond, live)?; + let keep = self.truthy(cl); + let shape = Self::live_types(live); + let (then_b, then_p) = self.create_block(&shape); + let (else_b, else_p) = self.create_block(&shape); + let args = Self::live_args(live); + self.term(Term::Brif { + cond: keep, + then_to: BlockId(then_b as u32), + then_args: args.clone(), + else_to: BlockId(else_b as u32), + else_args: args, + }); + + self.switch(then_b); + Self::rebind_live(live, &then_p); + let rl = self.emit(result, live)?; + let jump = finish_branch(self, rl, live); + self.term(jump); + + self.switch(else_b); + Self::rebind_live(live, &else_p); + } + + let dl = match default { + Some(d) => self.emit(d, live)?, + None => { + let flag = self.const_i1(false); + let val = self.default_of(res_ty); + Lane { + flag: Some(flag), + val, + } + } + }; + let jump = finish_branch(self, dl, live); + self.term(jump); + + self.switch(join); + Self::rebind_live(live, &join_params[..live_width]); + let tail = &join_params[live_width..]; + Ok(if res_nullable { + Lane { + flag: Some(tail[0]), + val: tail[1], + } + } else { + Lane { + flag: None, + val: tail[0], + } + }) + } + + /// CAST/TRY_CAST lowering. NULL input never traps; CAST failure traps + /// with a conversion message; TRY_CAST failure becomes NULL. + fn cast( + &mut self, + e: &SExpr, + inner: &SExpr, + trying: bool, + live: &mut Live, + ) -> Result { + let from = inner.ty; + let to = e.ty; + let l = self.emit(inner, live)?; + + // Branchless conversions first. + let simple: Option = match (from, to) { + (a, b) if a == b => Some(l), + (Ty::I64, Ty::F64) => { + let dst = self.fresh(); + self.inst(Inst::Itof { dst, a: l.val }); + Some(Lane { + flag: l.flag, + val: dst, + }) + } + (Ty::F64, Ty::I64) if !trying => { + // ftoi.round matches DuckDB CAST rounding; its own range trap + // stands in for DuckDB's conversion error. Nullable payloads + // are masked: computed garbage under a false flag (x * 1e300 + // * 1e300 with x NULL) must not fire the range trap. + let a = self.masked(l, Ty::F64); + let dst = self.fresh(); + self.inst(Inst::Ftoi { + mode: super::ir::RoundMode::Round, + dst, + a, + }); + Some(Lane { + flag: l.flag, + val: dst, + }) + } + (Ty::I64, Ty::Str) => { + let dst = self.fresh(); + self.inst(Inst::Itos { dst, a: l.val }); + Some(Lane { + flag: l.flag, + val: dst, + }) + } + (Ty::F64, Ty::Str) => { + let dst = self.fresh(); + self.inst(Inst::Ftos { dst, a: l.val }); + Some(Lane { + flag: l.flag, + val: dst, + }) + } + (Ty::I1, Ty::Str) => { + let t = self.const_lit(Lit::Str("true".to_string())); + let f = self.const_lit(Lit::Str("false".to_string())); + let dst = self.fresh(); + self.inst(Inst::Select { + dst, + cond: l.val, + a: t, + b: f, + }); + Some(Lane { + flag: l.flag, + val: dst, + }) + } + (Ty::I1, Ty::I64) => { + let one = self.const_lit(Lit::I64(1)); + let zero = self.const_lit(Lit::I64(0)); + let dst = self.fresh(); + self.inst(Inst::Select { + dst, + cond: l.val, + a: one, + b: zero, + }); + Some(Lane { + flag: l.flag, + val: dst, + }) + } + (Ty::I1, Ty::F64) => { + let one = self.const_lit(Lit::F64(1.0)); + let zero = self.const_lit(Lit::F64(0.0)); + let dst = self.fresh(); + self.inst(Inst::Select { + dst, + cond: l.val, + a: one, + b: zero, + }); + Some(Lane { + flag: l.flag, + val: dst, + }) + } + (Ty::I64, Ty::I1) => { + let zero = self.const_lit(Lit::I64(0)); + let dst = self.fresh(); + self.inst(Inst::Cmp { + pred: CmpPred::Ne, + ty: Ty::I64, + dst, + a: l.val, + b: zero, + }); + Some(Lane { + flag: l.flag, + val: dst, + }) + } + (Ty::F64, Ty::I1) => { + // DuckDB: nonzero -> true (measured 2.5::BOOLEAN = true). + let zero = self.const_lit(Lit::F64(0.0)); + let dst = self.fresh(); + self.inst(Inst::Cmp { + pred: CmpPred::Ne, + ty: Ty::F64, + dst, + a: l.val, + b: zero, + }); + Some(Lane { + flag: l.flag, + val: dst, + }) + } + _ => None, + }; + if let Some(lane) = simple { + // TRY_CAST of an infallible conversion: the declared nullability + // is `true`, so surface a flag even though nothing can fail. + if trying && lane.flag.is_none() { + let t = self.const_i1(true); + return Ok(Lane { + flag: Some(t), + val: lane.val, + }); + } + return Ok(lane); + } + + match (from, to) { + // String parses: shared shape, differing parse op. + (Ty::Str, Ty::I64) | (Ty::Str, Ty::F64) => { + let ok = self.fresh(); + let parsed = self.fresh(); + if to == Ty::I64 { + self.inst(Inst::StoiOpt { + flag: ok, + dst: parsed, + a: l.val, + }); + } else { + self.inst(Inst::StofOpt { + flag: ok, + dst: parsed, + a: l.val, + }); + } + if trying { + let flag = match l.flag { + Some(f) => self.bin(BinOp::And, f, ok), + None => ok, + }; + return Ok(Lane { + flag: Some(flag), + val: parsed, + }); + } + // CAST: trap iff the input is a real (non-NULL) string that + // does not parse. + let not_ok = self.not(ok); + let bad = match l.flag { + Some(f) => self.bin(BinOp::And, f, not_ok), + None => not_ok, + }; + let (trap_b, _) = self.create_block(&[]); + let mut cont_tys = Self::live_types(live); + let flag_in_shape = l.flag.is_some(); + if flag_in_shape { + cont_tys.push(Ty::I1); + } + cont_tys.push(to); + let live_width = Self::live_types(live).len(); + let (cont_b, cont_p) = self.create_block(&cont_tys); + let mut args = Self::live_args(live); + if let Some(f) = l.flag { + args.push(f); + } + args.push(parsed); + self.term(Term::Brif { + cond: bad, + then_to: BlockId(trap_b as u32), + then_args: vec![], + else_to: BlockId(cont_b as u32), + else_args: args, + }); + self.switch(trap_b); + self.term(Term::Trap { + msg: format!( + "Conversion Error: could not cast VARCHAR to {}", + if to == Ty::I64 { "BIGINT" } else { "DOUBLE" } + ), + }); + self.switch(cont_b); + Self::rebind_live(live, &cont_p[..live_width]); + let tail = &cont_p[live_width..]; + Ok(if flag_in_shape { + Lane { + flag: Some(tail[0]), + val: tail[1], + } + } else { + Lane { + flag: None, + val: tail[0], + } + }) + } + // TRY_CAST(f64 -> i64): guard the range so ftoi cannot trap; the + // payload rides the branch on the live stack. + (Ty::F64, Ty::I64) => { + debug_assert!(trying); + // Exactly ±2^63; NaN fails both compares -> NULL. A NULL + // input's default payload passes the range check but its + // false flag forces the null path anyway. + let min = self.const_lit(Lit::F64(-9223372036854775808.0)); + let max = self.const_lit(Lit::F64(9223372036854775808.0)); + let ge = self.fresh(); + self.inst(Inst::Cmp { + pred: CmpPred::Ge, + ty: Ty::F64, + dst: ge, + a: l.val, + b: min, + }); + let lt = self.fresh(); + self.inst(Inst::Cmp { + pred: CmpPred::Lt, + ty: Ty::F64, + dst: lt, + a: l.val, + b: max, + }); + let in_range = self.bin(BinOp::And, ge, lt); + let ok = match l.flag { + Some(f) => self.bin(BinOp::And, f, in_range), + None => in_range, + }; + + live.push((l, Ty::F64)); + let live_width = Self::live_types(live).len(); + let mut join_tys = Self::live_types(live); + join_tys.push(Ty::I1); + join_tys.push(Ty::I64); + let (join, join_p) = self.create_block(&join_tys); + let shape = Self::live_types(live); + let (conv_b, conv_p) = self.create_block(&shape); + let (null_b, null_p) = self.create_block(&shape); + let args = Self::live_args(live); + self.term(Term::Brif { + cond: ok, + then_to: BlockId(conv_b as u32), + then_args: args.clone(), + else_to: BlockId(null_b as u32), + else_args: args, + }); + + self.switch(conv_b); + Self::rebind_live(live, &conv_p); + let payload = live.last().expect("pushed above").0.val; + let r = self.fresh(); + self.inst(Inst::Ftoi { + mode: super::ir::RoundMode::Round, + dst: r, + a: payload, + }); + let t = self.const_i1(true); + let mut args = Self::live_args(live); + args.push(t); + args.push(r); + self.term(Term::Jump { + to: BlockId(join as u32), + args, + }); + + self.switch(null_b); + Self::rebind_live(live, &null_p); + let f = self.const_i1(false); + let d = self.const_lit(Lit::I64(0)); + let mut args = Self::live_args(live); + args.push(f); + args.push(d); + self.term(Term::Jump { + to: BlockId(join as u32), + args, + }); + + self.switch(join); + Self::rebind_live(live, &join_p[..live_width]); + live.pop(); + let tail = &join_p[live_width..]; + Ok(Lane { + flag: Some(tail[0]), + val: tail[1], + }) + } + (from, to) => Err(PrepareError::Internal(format!( + "cast {} -> {} escaped the frontend", + from.name(), + to.name() + ))), + } + } + + /// NULL propagation: the result is valid iff every nullable input is. + fn combine_flags(&mut self, a: Option, b: Option) -> Option { + match (a, b) { + (None, None) => None, + (Some(f), None) | (None, Some(f)) => Some(f), + (Some(fa), Some(fb)) => Some(self.bin(BinOp::And, fa, fb)), + } + } +} diff --git a/src/specializer/mod.rs b/src/specializer/mod.rs index 8c3c8d7..ef47be6 100644 --- a/src/specializer/mod.rs +++ b/src/specializer/mod.rs @@ -3,10 +3,84 @@ //! once and invoked millions of times with a small dynamic input relation. //! //! Design: docs/superpowers/specs/2026-07-25-sql-specializer-design.md. -//! Build order (backlog milestone m-7): the imperative IR below (M-ir), then -//! the closure-compiled interpreter oracle (M-interp), the frontend + BTA + -//! lowering (M-lower), the Cranelift backend (M-cranelift), and the generated -//! Python-boundary marshaller (M-boundary). +//! Build order (backlog milestone m-7): imperative IR (M-ir, done), the +//! closure-compiled interpreter oracle (M-interp, done), then this layer — +//! frontend + BTA + lowering (M-lower, in progress) — followed by the +//! Cranelift backend and the generated Python-boundary marshaller. pub mod exec; +pub mod fold; +pub mod frontend; pub mod ir; +pub mod lower; +pub mod plan; + +#[cfg(test)] +mod tests; + +pub use frontend::PrepareError; + +/// How to materialize map static `@N` from the static table it came from: +/// build one entry per table row, keyed by `key_cols` (converted to the +/// declared key types — an int column joined against a float expression +/// becomes f64 here), valued by `val_cols`. Rows with a NULL key are dropped +/// (a NULL never equi-matches); a NULL in a value column is an error. +pub struct StaticSpec { + pub table: String, + pub key_cols: Vec, + pub val_cols: Vec, +} + +/// The output of stage 1: a verified program plus, per map static, the +/// recipe the caller uses to turn its table data into `StaticData`. +pub struct Prepared { + pub program: ir::Program, + pub statics: Vec, +} + +/// STAGE 1 for the v0 ribbon: SQL text + the dynamic table's name and schema +/// + the static-table catalog -> a verified imperative-IR program. The +/// returned program is always verified — a lowering bug becomes +/// [`PrepareError::Internal`], never an executable. +pub fn prepare( + sql: &str, + this_name: &str, + in_cols: &[ir::Col], + statics: &[plan::StaticTable], +) -> Result { + let (rel, joins, out_cols) = frontend::frontend(sql, this_name, in_cols, statics)?; + let mut program = lower::lower(&rel, &joins, statics, in_cols, out_cols, "run")?; + // Block-splitting lowerings mint ids out of text order; renumber so + // every prepared program is exactly canonical (parse(print(p)) == p). + ir::canonicalize(&mut program); + if let Err(errs) = ir::verify::verify(&program) { + let msgs: Vec = errs.iter().map(|e| e.to_string()).collect(); + return Err(PrepareError::Internal(format!( + "lowered program failed verification: {}", + msgs.join("; ") + ))); + } + let specs = joins + .iter() + .map(|j| { + let t = &statics[j.table]; + StaticSpec { + table: t.name.clone(), + key_cols: j + .key_cols + .iter() + .map(|&c| t.cols[c as usize].name.clone()) + .collect(), + val_cols: j + .val_cols + .iter() + .map(|&c| t.cols[c as usize].name.clone()) + .collect(), + } + }) + .collect(); + Ok(Prepared { + program, + statics: specs, + }) +} diff --git a/src/specializer/plan.rs b/src/specializer/plan.rs new file mode 100644 index 0000000..278e2b1 --- /dev/null +++ b/src/specializer/plan.rs @@ -0,0 +1,175 @@ +//! The relational IR: what the frontend produces, what BTA annotates, what +//! lowering consumes. Deliberately skinny — the v0 shape is the +//! scan/filter/project ribbon over the dynamic table; joins and static +//! subtrees grow here at the BTA stretch. + +use super::ir::{CmpPred, Col, Lit, TrimSide, Ty}; + +/// A relational operator tree over the dynamic table. Joins to static +/// tables are not tree nodes: the v0 shape is rigid +/// (project(filter?(join*(scan)))), so the frontend returns them as an +/// ordered [`JoinSpec`] list instead — the tree would only restate the +/// vec's order. +pub enum Rel { + Scan, + Filter { + input: Box, + pred: SExpr, + }, + Project { + input: Box, + exprs: Vec<(String, SExpr)>, + }, +} + +/// A static (prepare-time-known) table's schema, as given to `prepare`. +/// Value-column nullability is deliberately ignored here: arrow schemas +/// default to nullable, so the real check — no NULL in a value column — +/// happens against the data at materialization. +pub struct StaticTable { + pub name: String, + pub cols: Vec, +} + +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum JoinKind { + Inner, + Left, +} + +/// One equi-join to a static table, in FROM-clause order. Join `i` probes +/// map static `@i`; its map layout is `keys[..] -> value columns`, where the +/// key column split comes from the ON clause. +pub struct JoinSpec { + /// Index into the static-table catalog handed to `prepare`. + pub table: usize, + pub kind: JoinKind, + /// Dynamic-side key expressions, one per key column, already promoted + /// to the map's key types. + pub keys: Vec, + /// Static-table columns acting as map keys (indices into `table.cols`), + /// aligned with `keys`. + pub key_cols: Vec, + /// The remaining columns, in table order — the probe's value lanes. + /// ponytail: all non-key columns become map values even if unreferenced; + /// prune to referenced columns when codegen makes the width measurable. + pub val_cols: Vec, +} + +/// A bound, typed scalar expression. `nullable` is the frontend's +/// conservative derivation ("cannot prove non-NULL"), and it is a contract +/// with lowering: an expression lowers to a flag lane IFF `nullable` — the +/// out-column nullability, the CASE join shape, and the store form all key +/// off it. +#[derive(Clone)] +pub struct SExpr { + pub kind: SKind, + pub ty: Ty, + pub nullable: bool, +} + +#[derive(Clone)] +pub enum SKind { + /// Input column, by index into the dynamic table's schema. + Col(u32), + /// Value column `col` (index into the join's `val_cols`) of join `join`. + /// Lowered as a lane of that join's probe: non-nullable under INNER + /// (misses were already skipped), hit-flagged under LEFT. + StaticCol { + join: u32, + col: u32, + }, + Lit(Lit), + /// Typed NULL constant (`ty` is on the SExpr): flag=false, payload + /// default. Produced where context gives the bare NULL literal a type. + NullOf, + /// Arithmetic after type promotion: both sides already the same `Ty` + /// (the frontend inserts `IntToFloat` where DuckDB promotes). + Arith { + op: ArithOp, + a: Box, + b: Box, + }, + /// Comparison after promotion; result i1, NULL-propagating. + Cmp { + pred: CmpPred, + a: Box, + b: Box, + }, + /// i64 -> f64 promotion node, inserted by the frontend. + IntToFloat(Box), + /// 3VL NOT: value negates, NULL stays NULL. + Not(Box), + /// Kleene AND/OR over i1 operands. + And { + a: Box, + b: Box, + }, + Or { + a: Box, + b: Box, + }, + /// IS NULL / IS NOT NULL — result i1, never NULL. + IsNull { + negated: bool, + inner: Box, + }, + /// Searched CASE (the simple form is desugared to `operand = value` + /// conditions at bind). First TRUE condition wins; NULL conditions do + /// not match; missing ELSE yields NULL. + Case { + arms: Vec<(SExpr, SExpr)>, + default: Option>, + }, + /// CAST / TRY_CAST; source is `inner.ty`, target is the SExpr's `ty`. + /// CAST traps on conversion failure (NULL input never traps); TRY_CAST + /// yields NULL instead. + Cast { + inner: Box, + trying: bool, + }, + /// UPPER / LOWER — Str -> Str, NULL-propagating, simple case mapping. + StrCase { + upper: bool, + a: Box, + }, + /// trim/ltrim/rtrim and all TRIM(...) forms. `chars` is always present: + /// the 1-arg SQL form gets a `' '` literal (DuckDB trims ONLY spaces). + Trim { + side: TrimSide, + a: Box, + chars: Box, + }, + /// substr/substring. `len: None` is the 2-arg form ("rest of the + /// string") — kept distinct because DuckDB range-guards an explicit + /// length but never a missing one. All operands NULL-propagate. + Substr { + a: Box, + start: Box, + len: Option>, + }, + /// ABS — I64 or F64; result type = operand type. Traps on i64::MIN. + Abs(Box), + /// ROUND(x) on F64, half away from zero. Integer round is identity and + /// never builds a node. + Round(Box), + /// String concatenation: `||` (always concat in DuckDB, any operands, + /// NULL-propagating) and the NULL-skipping CONCAT() after its per-arg + /// desugar. Both operands are Str by construction. + Concat { + a: Box, + b: Box, + }, +} + +/// SQL-level arithmetic. `Div` is DuckDB's `/` — ALWAYS float division +/// (measured: `5/2 = 2.5 DOUBLE`); the frontend promotes both sides to f64. +/// Integer `%` stays integral (measured: `5%2 -> INTEGER`). +#[derive(Clone, Copy, PartialEq, Eq, Debug)] +pub enum ArithOp { + Add, + Sub, + Mul, + Div, + Rem, +} diff --git a/src/specializer/tests.rs b/src/specializer/tests.rs new file mode 100644 index 0000000..51e225b --- /dev/null +++ b/src/specializer/tests.rs @@ -0,0 +1,1027 @@ +//! End-to-end stretch-1 tests: SQL text -> prepare -> interpreter oracle. +//! Expected values follow the DuckDB pins measured 2026-07-26 (`/` is float +//! division, `%` stays integral, overflow traps). + +use super::exec::interp::compile; +use super::exec::testutil::{batch, c_f64, c_i64, c_str, rows, run_snapshot}; +use super::exec::{KeyBits, ScalarVal, StaticData}; +use super::ir::{parse::parse, print::print, Col, ColTy, Ty}; +use super::plan::StaticTable; +use super::{prepare, PrepareError}; + +fn cols(spec: &[(&str, Ty, bool)]) -> Vec { + spec.iter() + .map(|(n, t, null)| Col { + name: n.to_string(), + ty: ColTy { + ty: *t, + nullable: *null, + }, + }) + .collect() +} + +/// prepare() for the common no-statics case, unwrapped to the program. +fn prep(sql: &str, in_cols: &[Col]) -> Result { + prepare(sql, "__THIS__", in_cols, &[]).map(|p| p.program) +} + +fn stat(name: &str, spec: &[(&str, Ty, bool)]) -> StaticTable { + StaticTable { + name: name.to_string(), + cols: cols(spec), + } +} + +/// prepare + compile + run with static-table map data. +fn run_join( + sql: &str, + in_cols: &[Col], + statics: &[StaticTable], + data: Vec, + input: super::exec::Batch, +) -> Result>, String> { + let p = prepare(sql, "__THIS__", in_cols, statics).map_err(|e| e.to_string())?; + let f = compile(&p.program, data).map_err(|e| e.to_string())?; + run_snapshot(&f, &input).map_err(|e| e.to_string()) +} + +fn run_sql( + sql: &str, + in_cols: &[Col], + input: super::exec::Batch, +) -> Result>, String> { + let p = prep(sql, in_cols).map_err(|e| e.to_string())?; + let f = compile(&p, vec![]).map_err(|e| e.to_string())?; + run_snapshot(&f, &input).map_err(|e| e.to_string()) +} + +#[test] +fn arithmetic_projection_end_to_end() { + let schema = cols(&[("a", Ty::I64, false)]); + let got = run_sql( + "SELECT a + 1 AS x, a * 2 AS y, a % 2 AS m FROM __THIS__", + &schema, + batch(2, vec![c_i64(&[Some(4), Some(7)])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["5", "8", "0"], &["8", "14", "1"]])); +} + +#[test] +fn division_is_float_division() { + // DuckDB pin: 5/2 = 2.5 DOUBLE. + let schema = cols(&[("a", Ty::I64, false)]); + let got = run_sql( + "SELECT a / 2 AS h FROM __THIS__", + &schema, + batch(1, vec![c_i64(&[Some(5)])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["2.5"]])); +} + +#[test] +fn null_propagation_through_arithmetic() { + let schema = cols(&[("a", Ty::I64, false), ("b", Ty::I64, true)]); + let got = run_sql( + "SELECT a + b AS s FROM __THIS__", + &schema, + batch( + 2, + vec![c_i64(&[Some(1), Some(2)]), c_i64(&[Some(10), None])], + ), + ) + .unwrap(); + assert_eq!(got, rows(&[&["11"], &["NULL"]])); +} + +#[test] +fn where_filters_and_null_predicate_drops() { + // SQL WHERE keeps only TRUE: false drops, NULL drops. + let schema = cols(&[("score", Ty::F64, true)]); + let got = run_sql( + "SELECT score FROM __THIS__ WHERE score > 0.5", + &schema, + batch(3, vec![c_f64(&[Some(0.9), Some(0.1), None])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["0.9"]])); +} + +#[test] +fn case_insensitive_bind_preserves_query_spelling() { + let schema = cols(&[("age", Ty::I64, false)]); + let p = prep("SELECT AGE FROM __this__", &schema).unwrap(); + assert_eq!(p.out_cols[0].name, "AGE"); + let f = compile(&p, vec![]).unwrap(); + let got = run_snapshot(&f, &batch(1, vec![c_i64(&[Some(3)])])).unwrap(); + assert_eq!(got, rows(&[&["3"]])); +} + +#[test] +fn unary_minus_and_literals() { + let schema = cols(&[("a", Ty::I64, false)]); + let got = run_sql( + "SELECT -a AS n, 1.5 AS f, 'x' AS s, true AS t FROM __THIS__", + &schema, + batch(1, vec![c_i64(&[Some(3)])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["-3", "1.5", "x", "true"]])); +} + +#[test] +fn integer_overflow_traps_like_duckdb() { + // DuckDB pin: BIGINT overflow is an error, not a wrap. + let schema = cols(&[("a", Ty::I64, false)]); + let err = run_sql( + "SELECT a + 1 AS x FROM __THIS__", + &schema, + batch(1, vec![c_i64(&[Some(i64::MAX)])]), + ) + .unwrap_err(); + assert!( + err.contains("overflow"), + "expected an overflow trap, got: {err}" + ); +} + +#[test] +fn prepared_programs_are_canonical_ir() { + // prepare() output is verified AND round-trips through the text format — + // the Builder assigns definition-ordered ids, so this holds exactly. + let schema = cols(&[("a", Ty::I64, false), ("b", Ty::F64, true)]); + let p = prep( + "SELECT a * 2 AS x, b / a AS r FROM __THIS__ WHERE b > 0.0", + &schema, + ) + .unwrap(); + let text = print(&p); + assert_eq!( + parse(&text).unwrap(), + p, + "prepared program is not canonical:\n{text}" + ); +} + +#[test] +fn column_cache_loads_once_per_block() { + let schema = cols(&[("a", Ty::I64, false)]); + let p = prep("SELECT a + a AS d FROM __THIS__", &schema).unwrap(); + let text = print(&p); + assert_eq!( + text.matches("load in.a").count(), + 1, + "column loaded more than once in one block:\n{text}" + ); +} + +// -------------------------------------------------------- 3VL (stretch 2) -- + +fn c_i1v(vals: &[Option]) -> super::exec::ColData { + super::exec::testutil::c_i1(vals) +} + +/// Kleene truth tables through nullable comparisons: p = (a > 0), q = (b > 0) +/// with NULLs flowing in from the columns. +#[test] +fn kleene_and_or_truth_tables() { + let schema = cols(&[("a", Ty::I64, true), ("b", Ty::I64, true)]); + // rows: (T,T) (T,F) (T,N) (F,N) (N,N) (F,F) + let a = c_i64(&[Some(1), Some(1), Some(1), Some(-1), None, Some(-1)]); + let b = c_i64(&[Some(1), Some(-1), None, None, None, Some(-1)]); + let got = run_sql( + "SELECT (a > 0) AND (b > 0) AS x, (a > 0) OR (b > 0) AS y FROM __THIS__", + &schema, + batch(6, vec![a, b]), + ) + .unwrap(); + assert_eq!( + got, + rows(&[ + &["true", "true"], + &["false", "true"], + &["NULL", "true"], // T AND N = N ; T OR N = T + &["false", "NULL"], // F AND N = F ; F OR N = N + &["NULL", "NULL"], + &["false", "false"], + ]) + ); +} + +#[test] +fn not_and_is_null() { + let schema = cols(&[("b", Ty::I64, true)]); + let got = run_sql( + "SELECT NOT (b > 0) AS n, b IS NULL AS isn, b IS NOT NULL AS notn FROM __THIS__", + &schema, + batch(3, vec![c_i64(&[Some(1), Some(-1), None])]), + ) + .unwrap(); + assert_eq!( + got, + rows(&[ + &["false", "false", "true"], + &["true", "false", "true"], + &["NULL", "true", "false"], + ]) + ); +} + +/// The eager-evaluation hazard: an untaken CASE branch containing a trapping +/// op (`%` by zero) must NOT trap — branches run only on their taken path. +#[test] +fn case_guards_trapping_branches() { + let schema = cols(&[("a", Ty::I64, false), ("b", Ty::I64, false)]); + let got = run_sql( + "SELECT CASE WHEN b <> 0 THEN a % b ELSE -1 END AS r FROM __THIS__", + &schema, + batch( + 2, + vec![c_i64(&[Some(7), Some(9)]), c_i64(&[Some(4), Some(0)])], + ), + ) + .unwrap(); + assert_eq!(got, rows(&[&["3"], &["-1"]])); +} + +#[test] +fn case_forms_null_conditions_and_type_unification() { + let schema = cols(&[("a", Ty::I64, true)]); + // Searched, no ELSE -> NULL; int/float branch unification -> DOUBLE + // (measured: CASE WHEN 1=0 THEN 1/0 ELSE -1 END -> -1.0). + let got = run_sql( + "SELECT CASE WHEN a > 1 THEN 1 WHEN a = 1 THEN 2.5 END AS u, \ + CASE a WHEN 1 THEN 'one' WHEN 2 THEN 'two' ELSE 'many' END AS s \ + FROM __THIS__", + &schema, + batch(4, vec![c_i64(&[Some(1), Some(2), Some(9), None])]), + ) + .unwrap(); + assert_eq!( + got, + rows(&[ + &["2.5", "one"], + &["1.0", "two"], + &["1.0", "many"], + // NULL operand: simple-form conditions are `a = v` -> NULL, + // never TRUE -> ELSE; searched arms also never TRUE -> NULL. + &["NULL", "many"], + ]) + ); +} + +#[test] +fn cast_matrix() { + let schema = cols(&[ + ("s", Ty::Str, false), + ("f", Ty::F64, false), + ("i", Ty::I64, false), + ]); + let input = || { + batch( + 1, + vec![ + super::exec::testutil::c_str(&[Some(" 5")]), + c_f64(&[Some(-2.5)]), + c_i64(&[Some(2)]), + ], + ) + }; + let got = run_sql( + "SELECT s::BIGINT AS a, f::BIGINT AS b, i::BOOLEAN AS c, \ + (i - 2)::BOOLEAN AS d, true::VARCHAR AS e, f::VARCHAR AS g, \ + TRY_CAST('x' AS BIGINT) AS h, TRY_CAST(1e19 AS BIGINT) AS j \ + FROM __THIS__", + &schema, + input(), + ) + .unwrap(); + // ' 5' trims (DuckDB CAST); -2.5 rounds half-away to -3; 2 -> true, + // 0 -> false; TRY_CAST failures -> NULL. + assert_eq!( + got, + rows(&[&["5", "-3", "true", "false", "true", "-2.5", "NULL", "NULL"]]) + ); +} + +#[test] +fn cast_failure_traps_and_null_never_traps() { + let schema = cols(&[("s", Ty::Str, true)]); + // NULL input: CAST(NULL) is NULL, no trap. + let got = run_sql( + "SELECT s::BIGINT AS n FROM __THIS__", + &schema, + batch(1, vec![super::exec::testutil::c_str(&[None])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["NULL"]])); + // Real string that does not parse: trap. + let err = run_sql( + "SELECT s::BIGINT AS n FROM __THIS__", + &schema, + batch(1, vec![super::exec::testutil::c_str(&[Some("abc")])]), + ) + .unwrap_err(); + assert!(err.contains("Conversion Error"), "wrong trap: {err}"); +} + +#[test] +fn null_literal_in_context() { + let schema = cols(&[("a", Ty::I64, false)]); + let got = run_sql( + "SELECT a + NULL AS x, a = NULL AS y, NULL IS NULL AS z, \ + CAST(NULL AS BIGINT) AS w, CASE WHEN a > 0 THEN NULL ELSE 5 END AS v \ + FROM __THIS__", + &schema, + batch(1, vec![c_i64(&[Some(3)])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["NULL", "NULL", "true", "NULL", "NULL"]])); +} + +#[test] +fn where_with_kleene_and_case() { + let schema = cols(&[("a", Ty::I64, true), ("b", Ty::I64, false)]); + // NULL AND true -> NULL -> dropped; CASE inside WHERE. + let got = run_sql( + "SELECT b FROM __THIS__ WHERE (a > 0) AND (CASE WHEN b > 10 THEN true ELSE b > 2 END)", + &schema, + batch( + 4, + vec![ + c_i64(&[Some(1), Some(1), None, Some(1)]), + c_i64(&[Some(20), Some(1), Some(20), Some(3)]), + ], + ), + ) + .unwrap(); + assert_eq!(got, rows(&[&["20"], &["3"]])); +} + +#[test] +fn case_heavy_program_is_canonical() { + // CASE lowering mints join-param ids before later blocks' instructions; + // prepare() canonicalizes, so exact round-trip equality holds anyway. + let schema = cols(&[("a", Ty::I64, true)]); + let p = prep( + "SELECT CASE WHEN a > 0 THEN TRY_CAST(a::VARCHAR AS BIGINT) ELSE a % 2 END AS r \ + FROM __THIS__ WHERE a IS NOT NULL", + &schema, + ) + .unwrap(); + let text = super::ir::print::print(&p); + assert_eq!( + parse(&text).unwrap(), + p, + "prepared program is not canonical:\n{text}" + ); +} + +#[test] +fn bool_column_directly_in_where() { + let schema = cols(&[("ok", Ty::I1, true), ("v", Ty::I64, false)]); + let got = run_sql( + "SELECT v FROM __THIS__ WHERE ok", + &schema, + batch( + 3, + vec![ + c_i1v(&[Some(true), Some(false), None]), + c_i64(&[Some(1), Some(2), Some(3)]), + ], + ), + ) + .unwrap(); + assert_eq!(got, rows(&[&["1"]])); +} + +#[test] +fn unsupported_constructs_are_named_cleanly() { + let schema = cols(&[("a", Ty::I64, false)]); + for (sql, needle) in [ + ("SELECT a FROM __THIS__ JOIN t USING (a)", "USING"), + // Bare aggregates parse as plain function calls; they reject via the + // function arm until the catalogue distinguishes aggregation. + ("SELECT sum(a) FROM __THIS__", "function sum"), + ("SELECT a FROM __THIS__ GROUP BY a", "aggregation"), + ("SELECT length('x') FROM __THIS__", "function"), + ("SELECT * FROM __THIS__", "star expansion"), + ("SELECT a FROM __THIS__ ORDER BY a", "ORDER BY"), + ("SELECT a, a FROM __THIS__", "duplicate output column"), + ("SELECT NULL FROM __THIS__", "NULL literal"), + ("SELECT a FROM other_table", "must be the dynamic table"), + ] { + match prep(sql, &schema) { + Err(PrepareError::Unsupported(msg)) => { + assert!( + msg.contains(needle), + "'{sql}': wanted '{needle}' in '{msg}'" + ) + } + Err(other) => panic!("'{sql}': wrong error kind: {other}"), + Ok(_) => panic!("'{sql}': unexpectedly prepared"), + } + } +} + +#[test] +fn bind_errors_are_not_unsupported() { + let schema = cols(&[("a", Ty::I64, false), ("s", Ty::Str, false)]); + for (sql, needle) in [ + ("SELECT nope FROM __THIS__", "does not exist"), + ("SELECT a + s FROM __THIS__", "numeric"), + ("SELECT a FROM __THIS__ WHERE s", "BOOLEAN"), + ("SELECT a < s FROM __THIS__", "cannot compare"), + ( + "SELECT a FROM __THIS__ JOIN t ON a = a", + "was not provided as a static table", + ), + ] { + match prep(sql, &schema) { + Err(PrepareError::Bind(msg)) => { + assert!( + msg.contains(needle), + "'{sql}': wanted '{needle}' in '{msg}'" + ) + } + Err(other) => panic!("'{sql}': wrong error kind: {other}"), + Ok(_) => panic!("'{sql}': unexpectedly prepared"), + } + } +} + +// ---------------------------------------------------------------- stretch 3: +// statics — equi-joins lower to probes, constants fold at prepare. + +#[test] +fn inner_join_probe_hits_and_misses() { + // k=2 has no build entry: INNER drops the row. + let schema = cols(&[("k", Ty::I64, false)]); + let dim = stat("dim", &[("id", Ty::I64, false), ("name", Ty::Str, false)]); + let data = StaticData::Map(vec![ + (vec![KeyBits::I64(1)], vec![ScalarVal::Str("one".into())]), + (vec![KeyBits::I64(3)], vec![ScalarVal::Str("three".into())]), + ]); + let got = run_join( + "SELECT k, name FROM __THIS__ JOIN dim ON k = dim.id", + &schema, + &[dim], + vec![data], + batch(3, vec![c_i64(&[Some(1), Some(2), Some(3)])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["1", "one"], &["3", "three"]])); +} + +#[test] +fn left_join_miss_and_null_key_give_null_values() { + // LEFT keeps every dynamic row; a miss (k=2) and a NULL key (a NULL + // never equi-matches — DuckDB pin) both yield NULL value columns. + let schema = cols(&[("k", Ty::I64, true)]); + let dim = stat("dim", &[("id", Ty::I64, false), ("name", Ty::Str, false)]); + let data = StaticData::Map(vec![( + vec![KeyBits::I64(1)], + vec![ScalarVal::Str("one".into())], + )]); + let got = run_join( + "SELECT k, name FROM __THIS__ LEFT JOIN dim ON k = dim.id", + &schema, + &[dim], + vec![data], + batch(3, vec![c_i64(&[Some(1), Some(2), None])]), + ) + .unwrap(); + assert_eq!( + got, + rows(&[&["1", "one"], &["2", "NULL"], &["NULL", "NULL"]]) + ); +} + +#[test] +fn inner_join_null_key_drops_the_row() { + let schema = cols(&[("k", Ty::I64, true)]); + let dim = stat("dim", &[("id", Ty::I64, false), ("v", Ty::I64, false)]); + let data = StaticData::Map(vec![(vec![KeyBits::I64(1)], vec![ScalarVal::I64(10)])]); + let got = run_join( + "SELECT v FROM __THIS__ JOIN dim ON k = dim.id", + &schema, + &[dim], + vec![data], + batch(2, vec![c_i64(&[Some(1), None])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["10"]])); +} + +#[test] +fn join_key_promotion_int_dyn_against_float_col() { + // dyn I64 key vs F64 static col: the dynamic side is promoted, the map + // keys stay F64 (canonical bits). + let schema = cols(&[("k", Ty::I64, false)]); + let dim = stat("dim", &[("id", Ty::F64, false), ("v", Ty::I64, false)]); + let data = StaticData::Map(vec![( + vec![KeyBits::F64(1f64.to_bits())], + vec![ScalarVal::I64(10)], + )]); + let got = run_join( + "SELECT v FROM __THIS__ JOIN dim ON k = dim.id", + &schema, + &[dim], + vec![data], + batch(2, vec![c_i64(&[Some(1), Some(2)])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["10"]])); +} + +#[test] +fn join_key_promotion_float_dyn_against_int_col() { + // dyn F64 key vs I64 static col: the map's key type is the expression's + // F64 — the StaticSpec tells the materializer to convert the build side. + let schema = cols(&[("k", Ty::F64, false)]); + let dim = stat("dim", &[("id", Ty::I64, false), ("v", Ty::I64, false)]); + let p = prepare( + "SELECT v FROM __THIS__ JOIN dim ON k = dim.id", + "__THIS__", + &schema, + &[dim], + ) + .unwrap(); + let spec = &p.statics[0]; + assert_eq!( + (spec.table.as_str(), &spec.key_cols[..], &spec.val_cols[..]), + ("dim", &["id".to_string()][..], &["v".to_string()][..]) + ); + let data = StaticData::Map(vec![( + vec![KeyBits::F64(2f64.to_bits())], + vec![ScalarVal::I64(20)], + )]); + let f = compile(&p.program, vec![data]).unwrap(); + let got = run_snapshot(&f, &batch(2, vec![c_f64(&[Some(2.0), Some(2.5)])])).unwrap(); + assert_eq!(got, rows(&[&["20"]])); +} + +#[test] +fn f64_probe_keys_canonicalize_negzero_and_nan() { + // DuckDB pin: for DOUBLE `=`, 0.0 = -0.0 and NaN = NaN. Build keys are + // canonicalized at compile (here -0.0 is stored), probes canonicalize + // the searched value, so every row below hits. + let schema = cols(&[("k", Ty::F64, false)]); + let dim = stat("dim", &[("id", Ty::F64, false), ("v", Ty::I64, false)]); + let data = StaticData::Map(vec![ + ( + vec![KeyBits::F64((-0.0f64).to_bits())], + vec![ScalarVal::I64(1)], + ), + ( + vec![KeyBits::F64((f64::NAN).to_bits() ^ 1)], + vec![ScalarVal::I64(2)], + ), + ]); + let got = run_join( + "SELECT v FROM __THIS__ JOIN dim ON k = dim.id", + &schema, + &[dim], + vec![data], + batch(3, vec![c_f64(&[Some(0.0), Some(-0.0), Some(f64::NAN)])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["1"], &["1"], &["2"]])); +} + +#[test] +fn duplicate_build_keys_are_rejected_at_compile() { + let schema = cols(&[("k", Ty::I64, false)]); + let dim = stat("dim", &[("id", Ty::I64, false), ("v", Ty::I64, false)]); + let data = StaticData::Map(vec![ + (vec![KeyBits::I64(1)], vec![ScalarVal::I64(10)]), + (vec![KeyBits::I64(1)], vec![ScalarVal::I64(11)]), + ]); + let err = run_join( + "SELECT v FROM __THIS__ JOIN dim ON k = dim.id", + &schema, + &[dim], + vec![data], + batch(1, vec![c_i64(&[Some(1)])]), + ) + .unwrap_err(); + assert!(err.contains("duplicate map key"), "got: {err}"); +} + +#[test] +fn join_programs_are_canonical_ir() { + let schema = cols(&[("k", Ty::I64, true), ("x", Ty::I64, false)]); + let dim = stat("dim", &[("id", Ty::I64, false), ("name", Ty::Str, false)]); + let p = prepare( + "SELECT x, name FROM __THIS__ LEFT JOIN dim ON k = dim.id WHERE x > 0", + "__THIS__", + &schema, + &[dim], + ) + .unwrap() + .program; + let text = print(&p); + assert_eq!( + parse(&text).unwrap(), + p, + "join program is not canonical:\n{text}" + ); +} + +#[test] +fn join_shape_errors() { + let schema = cols(&[("k", Ty::I64, false)]); + // A join key column referenced outside its ON clause is a clean unsup. + let dim = stat("dim", &[("id", Ty::I64, false), ("name", Ty::Str, false)]); + match prepare( + "SELECT dim.id FROM __THIS__ JOIN dim ON k = dim.id", + "__THIS__", + &schema, + &[dim], + ) { + Err(PrepareError::Unsupported(msg)) => assert!(msg.contains("ON clause"), "got: {msg}"), + other => panic!("wrong outcome: {:?}", other.err()), + } + // A join where every column is a key has no value columns to produce. + let keys_only = stat("dim", &[("id", Ty::I64, false)]); + match prepare( + "SELECT k FROM __THIS__ JOIN dim ON k = dim.id", + "__THIS__", + &schema, + &[keys_only], + ) { + Err(PrepareError::Unsupported(msg)) => assert!(msg.contains("value columns"), "got: {msg}"), + other => panic!("wrong outcome: {:?}", other.err()), + } +} + +#[test] +fn constant_arithmetic_folds_at_prepare() { + let schema = cols(&[("a", Ty::I64, false)]); + let p = prep("SELECT 1 + 2 * 3 AS x FROM __THIS__", &schema).unwrap(); + let text = print(&p); + assert!( + text.contains("const.i64 7"), + "expected folded constant 7:\n{text}" + ); + assert!( + !text.contains("iadd") && !text.contains("imul"), + "expected no arithmetic:\n{text}" + ); +} + +#[test] +fn dominating_constant_keeps_the_dynamic_side() { + // fold() must not rewrite `false AND ` to false: the dynamic side + // may trap (a % 0) and folding it away would change behavior. + let schema = cols(&[("a", Ty::I64, false)]); + let p = prep("SELECT false AND a % 0 = 0 AS x FROM __THIS__", &schema).unwrap(); + let text = print(&p); + assert!( + text.contains("irem"), + "the trapping rem was folded away:\n{text}" + ); +} + +// ---------------------------------------------------------------- stretch 4: +// the builtin catalogue, per the measured pins in +// docs/superpowers/specs/2026-07-26-stretch4-builtin-pins.md. + +#[test] +fn string_builtins_end_to_end() { + let schema = cols(&[("s", Ty::Str, true)]); + let got = run_sql( + "SELECT upper(s) AS u, lower('AbC') AS l, trim(' x ') AS t, \ + ltrim('xxa', 'x') AS lt, rtrim('a ') AS rt, \ + TRIM(LEADING 'x' FROM 'xax') AS tl, substr('hello', 2, 3) AS sub \ + FROM __THIS__", + &schema, + batch(2, vec![c_str(&[Some("ab"), None])]), + ) + .unwrap(); + assert_eq!( + got, + rows(&[ + &["AB", "abc", "x", "a", "a", "ax", "ell"], + &["NULL", "abc", "x", "a", "a", "ax", "ell"], + ]) + ); +} + +#[test] +fn one_arg_trim_removes_only_spaces() { + // DuckDB pin: 1-arg trim removes ONLY 0x20 — tabs/newlines stay. + let schema = cols(&[("a", Ty::I64, false)]); + let got = run_sql( + "SELECT trim(' \t a \n ') AS t FROM __THIS__", + &schema, + batch(1, vec![c_i64(&[Some(0)])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["\t a \n"]])); +} + +#[test] +fn substr_window_arithmetic_via_sql() { + let schema = cols(&[("a", Ty::I64, false)]); + let got = run_sql( + "SELECT substr('hello', 0, 3) AS a, substr('hello', -2) AS b, \ + substr('hello', -10, 8) AS c, substr('hello', 1, -1) AS d, \ + substring('hello' FROM 2 FOR 2) AS e FROM __THIS__", + &schema, + batch(1, vec![c_i64(&[Some(0)])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["he", "lo", "hello", "", "el"]])); +} + +#[test] +fn concat_operator_always_concats_and_propagates_null() { + // DuckDB pins: 1 || 'x' = '1x' (implicit VARCHAR cast), || propagates + // NULL; CONCAT skips NULLs and never returns NULL. + let schema = cols(&[("n", Ty::I64, true)]); + let got = run_sql( + "SELECT 1 || 'x' AS a, 'a' || NULL AS b, n || '!' AS c, \ + CONCAT('a', NULL, 1) AS d, CONCAT(n, '-') AS e FROM __THIS__", + &schema, + batch(2, vec![c_i64(&[Some(7), None])]), + ) + .unwrap(); + assert_eq!( + got, + rows(&[ + &["1x", "NULL", "7!", "a1", "7-"], + &["1x", "NULL", "NULL", "a1", "-"], + ]) + ); +} + +#[test] +fn abs_and_round_semantics() { + // Pins: abs(i64) traps only on MIN; abs(-0.0) = +0.0; round is half + // away from zero; integer round is identity (type preserved). + let schema = cols(&[("a", Ty::I64, false)]); + let got = run_sql( + "SELECT abs(-5) AS ai, abs(a) AS av, round(2.5) AS r1, \ + round(-2.5) AS r2, round(a) AS ri FROM __THIS__", + &schema, + batch(1, vec![c_i64(&[Some(-3)])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["5", "3", "3.0", "-3.0", "-3"]])); +} + +#[test] +fn abs_min_traps_like_duckdb() { + let schema = cols(&[("a", Ty::I64, false)]); + let err = run_sql( + "SELECT abs(a) AS x FROM __THIS__", + &schema, + batch(1, vec![c_i64(&[Some(i64::MIN)])]), + ) + .unwrap_err(); + assert!(err.contains("overflow"), "got: {err}"); +} + +#[test] +fn int_rem_by_zero_is_null_not_error() { + // DuckDB pin (2026-07-26): 5 % 0 is NULL. MIN % -1 still traps. + let schema = cols(&[("a", Ty::I64, false), ("b", Ty::I64, false)]); + let got = run_sql( + "SELECT a % b AS r FROM __THIS__", + &schema, + batch( + 3, + vec![ + c_i64(&[Some(5), Some(5), Some(-7)]), + c_i64(&[Some(0), Some(3), Some(2)]), + ], + ), + ) + .unwrap(); + assert_eq!(got, rows(&[&["NULL"], &["2"], &["-1"]])); +} + +#[test] +fn float_rem_is_ieee() { + let schema = cols(&[("x", Ty::F64, false), ("y", Ty::F64, false)]); + let got = run_sql( + "SELECT x % y AS r FROM __THIS__", + &schema, + batch( + 2, + vec![ + c_f64(&[Some(-5.5), Some(5.0)]), + c_f64(&[Some(2.5), Some(0.0)]), + ], + ), + ) + .unwrap(); + assert_eq!(got, rows(&[&["-0.5"], &["NaN"]])); +} + +#[test] +fn nan_equals_nan_in_where() { + // DuckDB DOUBLE order: nan = nan is TRUE, nan > 1 is TRUE. + let schema = cols(&[("x", Ty::F64, false)]); + let got = run_sql( + "SELECT x FROM __THIS__ WHERE x = x AND x > 1.0", + &schema, + batch(2, vec![c_f64(&[Some(f64::NAN), Some(0.5)])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["NaN"]])); +} + +#[test] +fn coalesce_binds_lazily_and_unifies() { + let schema = cols(&[("n", Ty::I64, true)]); + // The CAST in the untaken arm must not trap when n is non-NULL. + let got = run_sql( + "SELECT coalesce(n, CAST('nope' AS BIGINT)) AS a, \ + coalesce(NULL, 1, 2) AS b, coalesce(n, 2.5) AS c FROM __THIS__", + &schema, + batch(1, vec![c_i64(&[Some(4)])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["4", "1", "4.0"]])); +} + +#[test] +fn coalesce_taken_null_arm_falls_through_and_bad_cast_traps() { + let schema = cols(&[("n", Ty::I64, true)]); + let err = run_sql( + "SELECT coalesce(n, CAST('nope' AS BIGINT)) AS a FROM __THIS__", + &schema, + batch(1, vec![c_i64(&[None])]), + ) + .unwrap_err(); + assert!( + err.contains("cast") || err.contains("nope") || err.contains("convert"), + "expected a cast trap, got: {err}" + ); +} + +#[test] +fn nullif_compares_promoted_keeps_first_type() { + let schema = cols(&[("x", Ty::F64, false)]); + // nullif(1, 1.0) -> NULL (promoted compare); nullif(nan, nan) -> NULL + // (DuckDB float order); nullif(3, 3.5) -> 3 INTEGER. + let got = run_sql( + "SELECT nullif(1, 1.0) AS a, nullif(x, x) AS b, nullif(3, 3.5) AS c \ + FROM __THIS__", + &schema, + batch(1, vec![c_f64(&[Some(f64::NAN)])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["NULL", "NULL", "3"]])); +} + +#[test] +fn builtin_programs_are_canonical_ir() { + let schema = cols(&[("s", Ty::Str, true), ("x", Ty::F64, false)]); + let p = prep( + "SELECT upper(trim(s)) AS u, substr(s, 2) AS m, s || 'x' AS c, \ + abs(x) AS a, round(x) AS r FROM __THIS__ WHERE x % 2.0 > 0.0", + &schema, + ) + .unwrap(); + let text = print(&p); + assert_eq!( + parse(&text).unwrap(), + p, + "builtin program is not canonical:\n{text}" + ); +} + +// ------------------------------------------------- adversarial-fleet fixes: +// divergences found by the 6-agent differential probe (2026-07-26). + +#[test] +fn null_divisor_rem_is_null_not_trap() { + // The `b = 0` guard alone is NULL for a NULL divisor, which fell through + // to irem on the garbage zero payload. IS NULL now shields it. + let schema = cols(&[("a", Ty::I64, true), ("b", Ty::I64, true)]); + let got = run_sql( + "SELECT a % b AS r FROM __THIS__", + &schema, + batch( + 3, + vec![ + c_i64(&[Some(7), None, Some(7)]), + c_i64(&[None, Some(0), Some(3)]), + ], + ), + ) + .unwrap(); + assert_eq!(got, rows(&[&["NULL"], &["NULL"], &["1"]])); +} + +#[test] +fn traps_never_fire_under_a_null_flag() { + // Computed garbage payloads are unbounded: (NULL + MAX) + MAX would + // overflow its payload lane. Masking forces the default before any + // trapping instruction. A real value still traps like DuckDB. + let schema = cols(&[("a", Ty::I64, true)]); + let sql = "SELECT a + 9223372036854775807 + 9223372036854775807 AS r FROM __THIS__"; + let got = run_sql(sql, &schema, batch(1, vec![c_i64(&[None])])).unwrap(); + assert_eq!(got, rows(&[&["NULL"]])); + let err = run_sql(sql, &schema, batch(1, vec![c_i64(&[Some(1)])])).unwrap_err(); + assert!(err.contains("overflow"), "got: {err}"); +} + +#[test] +fn numeric_conditions_coerce_to_bool() { + // DuckDB pins: WHERE/AND/NOT/CASE take numerics — nonzero is true. + let schema = cols(&[("a", Ty::I64, false)]); + let got = run_sql( + "SELECT a FROM __THIS__ WHERE a % 2 AND 1", + &schema, + batch(4, vec![c_i64(&[Some(1), Some(2), Some(3), Some(4)])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["1"], &["3"]])); + let got = run_sql( + "SELECT CASE WHEN 2 THEN 'a' ELSE 'b' END AS c, NOT 5 AS n FROM __THIS__", + &schema, + batch(1, vec![c_i64(&[Some(0)])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["a", "false"]])); +} + +#[test] +fn trim_default_set_is_unicode_zs() { + // Adversarial census: the 1-arg trim set is exactly the Zs category — + // NBSP and ideographic space go, tab and newline stay. + let schema = cols(&[("s", Ty::Str, false)]); + let got = run_sql( + "SELECT trim(s) AS t FROM __THIS__", + &schema, + batch(2, vec![c_str(&[Some("\u{A0}a\u{3000}"), Some("\ta\n")])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["a"], &["\ta\n"]])); +} + +#[test] +fn substr_negative_length_slices_backwards() { + let schema = cols(&[("a", Ty::I64, false)]); + let got = run_sql( + "SELECT substr('hello', 3, -2) AS a, substr('hello', 6, -5) AS b, \ + substr('hello', 1, -1) AS c FROM __THIS__", + &schema, + batch(1, vec![c_i64(&[Some(0)])]), + ) + .unwrap(); + assert_eq!(got, rows(&[&["he", "hello", ""]])); +} + +#[test] +fn substr_range_guard_traps_like_duckdb() { + let schema = cols(&[("a", Ty::I64, false)]); + let err = run_sql( + "SELECT substr('hello', 4294967296) AS x FROM __THIS__", + &schema, + batch(1, vec![c_i64(&[Some(0)])]), + ) + .unwrap_err(); + assert!(err.contains("offset outside"), "got: {err}"); +} + +#[test] +fn float_to_varchar_matches_duckdb_rendering() { + // Pins: explicit exponent sign, two-digit minimum, lowercase nan. + let schema = cols(&[("x", Ty::F64, false)]); + let got = run_sql( + "SELECT x || '' AS s FROM __THIS__", + &schema, + batch( + 4, + vec![c_f64(&[Some(1e300), Some(1e-5), Some(f64::NAN), Some(2.5)])], + ), + ) + .unwrap(); + assert_eq!(got, rows(&[&["1e+300"], &["1e-05"], &["nan"], &["2.5"]])); +} + +#[test] +fn rowid_and_lateral_alias_reject_cleanly() { + let schema = cols(&[("a", Ty::I64, false)]); + for (sql, needle) in [ + ("SELECT rowid FROM __THIS__", "rowid"), + ("SELECT a % 2 AS k FROM __THIS__ WHERE k = 1", "lateral"), + ] { + match prep(sql, &schema) { + Err(PrepareError::Unsupported(msg)) => { + assert!( + msg.contains(needle), + "'{sql}': wanted '{needle}' in '{msg}'" + ) + } + other => panic!("'{sql}': wrong outcome: {:?}", other.err()), + } + } +} diff --git a/src/types.rs b/src/types.rs index fe37c99..84aba8b 100644 --- a/src/types.rs +++ b/src/types.rs @@ -72,16 +72,29 @@ mod tests { #[test] fn struct_compatibility_is_name_keyed_not_positional() { - let xy = Base::Struct(vec![("x".into(), ft(Base::Int)), ("y".into(), ft(Base::Str))]); - let yx = Base::Struct(vec![("y".into(), ft(Base::Str)), ("x".into(), ft(Base::Int))]); - assert!(compatible(&xy, &yx), "same names+types, reordered, should be compatible"); + let xy = Base::Struct(vec![ + ("x".into(), ft(Base::Int)), + ("y".into(), ft(Base::Str)), + ]); + let yx = Base::Struct(vec![ + ("y".into(), ft(Base::Str)), + ("x".into(), ft(Base::Int)), + ]); + assert!( + compatible(&xy, &yx), + "same names+types, reordered, should be compatible" + ); - let different_names = - Base::Struct(vec![("x".into(), ft(Base::Int)), ("z".into(), ft(Base::Str))]); + let different_names = Base::Struct(vec![ + ("x".into(), ft(Base::Int)), + ("z".into(), ft(Base::Str)), + ]); assert!(!compatible(&xy, &different_names)); - let different_types = - Base::Struct(vec![("x".into(), ft(Base::Str)), ("y".into(), ft(Base::Str))]); + let different_types = Base::Struct(vec![ + ("x".into(), ft(Base::Str)), + ("y".into(), ft(Base::Str)), + ]); assert!(!compatible(&xy, &different_types)); } } diff --git a/tests/test_corpus_replay.py b/tests/test_corpus_replay.py new file mode 100644 index 0000000..c98d1cf --- /dev/null +++ b/tests/test_corpus_replay.py @@ -0,0 +1,150 @@ +"""Corpus replay: the three-outcome contract over tests/corpus/duckdb_mined.jsonl. + +Each case reconstructs its tables in a fresh DuckDB, feeds them to +DuckDBInferFn (driving table = the FROM'd table, all others static), and +classifies: + + match -- engine output equals the mined rows + clean-unsupported -- the engine rejects at BUILD time, naming the limit: + an "unsupported: ..." error, a parse error (DuckDB + dialect beyond sqlparser, e.g. `SELECT * LIKE`, + `COLUMNS(...)`), or the documented v0 static-data + contracts (unique join keys, no NULL in a value + column). Cases whose FROM is a table function + (`range(...)`) have no base tables and can never be + v0 -- also clean. + FAIL -- mismatch, wrong error, or crash + +The gate requires zero FAILs. The match count is the growth ladder: every +construct the engine learns flips cases from clean-unsupported to match +(see scripts/mine_duckdb_corpus.py). +""" + +from __future__ import annotations + +import json +import re +from pathlib import Path + +import duckdb +import pyarrow as pa +from pydantic import create_model + +from sql_transform._interpreter import DuckDBInferFn + +CORPUS = Path(__file__).parent / "corpus" / "duckdb_mined.jsonl" + +# Build-time errors that are documented v0 contract limits, not bugs. +_CLEAN = ("unsupported:", "parse error:", "duplicate map key", "NULL in value column") + +_FROM_RE = re.compile(r"\bFROM\s+([A-Za-z_][A-Za-z0-9_]*)", re.IGNORECASE) + +_PY_OF_ARROW = [ + (pa.types.is_boolean, bool), + (pa.types.is_integer, int), + (pa.types.is_floating, float), + (pa.types.is_string, str), + (pa.types.is_large_string, str), +] + + +def _py_type(t: pa.DataType): + for pred, py in _PY_OF_ARROW: + if pred(t): + return py + return None + + +def _norm_row(row) -> tuple[str, ...]: + # repr keeps int/float/str/bool/None apart and makes NaN self-equal. + return tuple(repr(v) for v in row) + + +def _replay(case: dict) -> tuple[str, str]: + """-> (outcome, detail).""" + con = duckdb.connect() + for stmt in case["setup"]: + try: + con.execute(stmt) + except duckdb.CatalogException as e: + # Miner limitation: a file that drops + re-creates a table via a + # directive the line-parser skips records both CREATEs. Replaying + # the re-create after a drop is exactly what the file did. + m = re.match(r'\s*CREATE\s+TABLE\s+"?([A-Za-z_]\w*)"?', stmt, re.IGNORECASE) + if m and "already exists" in str(e): + con.execute(f'DROP TABLE "{m.group(1)}"') + con.execute(stmt) + else: + return "FAIL", f"setup failed: {stmt[:80]}: {e}" + named = con.execute( + "SELECT schema_name, table_name FROM duckdb_tables()" + ).fetchall() + if not named: + return "unsupported", "FROM is a table function; no base tables, never v0" + + m = _FROM_RE.search(case["sql"]) + by_lower = {t.lower(): (s, t) for s, t in named} + driving = None + if m and m.group(1).lower() in by_lower: + driving = by_lower[m.group(1).lower()][1] + if driving is None: + driving = named[0][1] + + arrow = { + t: con.execute(f'SELECT * FROM "{s}"."{t}"').to_arrow_table() for s, t in named + } + + # Row model from the driving table's schema; unmappable column types are + # the engine's own clean rejection (it sees the same field as unsupported). + fields = {} + for f in arrow[driving].schema: + py = _py_type(f.type) + if py is None: + fields[f.name] = (object, None) # forces the engine's clean error + else: + fields[f.name] = (py | None, None) + model = create_model("Row", **fields) + statics = {t: a for t, a in arrow.items() if t != driving} + + try: + fn = DuckDBInferFn( + case["sql"], row_tables={driving: model}, static_tables=statics + ) + except Exception as e: # noqa: BLE001 -- classification, not control flow + msg = str(e) + if any(n in msg for n in _CLEAN): + return "unsupported", msg + return "FAIL", f"build error: {type(e).__name__}: {msg}" + + try: + rows_in = [model(**r) for r in arrow[driving].to_pylist()] + got = [list(r.model_dump().values()) for r in fn.infer({driving: rows_in})] + except Exception as e: # noqa: BLE001 + return "FAIL", f"run error: {type(e).__name__}: {e}" + + want = case["rows"] + if sorted(map(_norm_row, got)) != sorted(map(_norm_row, want)): + return "FAIL", f"mismatch: got {got!r}, want {want!r}" + return "match", "" + + +def test_corpus_replay_three_outcomes(): + cases = [json.loads(line) for line in CORPUS.open(encoding="utf-8")] + counts = {"match": 0, "unsupported": 0, "FAIL": 0} + fails: list[str] = [] + for i, case in enumerate(cases): + outcome, detail = _replay(case) + counts[outcome] += 1 + if outcome == "FAIL": + fails.append(f"[{i}] {case['source']}: {case['sql']}\n {detail}") + + print( + f"\ncorpus replay: {counts['match']} match, " + f"{counts['unsupported']} clean-unsupported, {counts['FAIL']} FAIL " + f"of {len(cases)}" + ) + assert not fails, ( + f"{len(fails)} corpus FAILs " + f"({counts['match']} match / {counts['unsupported']} unsupported):\n" + + "\n".join(fails[:25]) + ) diff --git a/tests/test_duckdb_interpreter.py b/tests/test_duckdb_interpreter.py index 2465f65..941e80d 100644 --- a/tests/test_duckdb_interpreter.py +++ b/tests/test_duckdb_interpreter.py @@ -1,25 +1,429 @@ -"""DuckDBInferFn is a stub: it parses in the DuckDB dialect and nothing else.""" +"""DuckDBInferFn vs duckdb-python: the specializer's differential oracle. + +`duck_check` runs the same SQL on the same data through both engines and +asserts the outputs agree row-for-row. Stretch-3 surface: the projection / +WHERE spine plus equi-joins to static tables (INNER and LEFT), which lower +to map probes. +""" from __future__ import annotations +from typing import Any + +import duckdb +import pyarrow as pa import pytest -from pydantic import BaseModel +from pydantic import create_model from sql_transform._interpreter import DuckDBInferFn +_PY = {"int": int, "float": float, "str": str, "bool": bool} +_ARROW = { + "int": pa.int64(), + "float": pa.float64(), + "str": pa.string(), + "bool": pa.bool_(), +} -class Row(BaseModel): - age: int +def _row_model(schema: dict[str, str]): + fields: dict[str, Any] = {} + for name, spec in schema.items(): + if spec.endswith("?"): + fields[name] = (_PY[spec[:-1]] | None, None) + else: + fields[name] = (_PY[spec], ...) + return create_model("Row", **fields) -def test_builds_and_infer_raises(): - fn = DuckDBInferFn( - "SELECT age FROM __THIS__", row_tables={"__THIS__": Row}, static_tables={} + +def static(schema: dict[str, str], rows: list[dict[str, Any]]) -> pa.Table: + arrow = pa.schema( + pa.field(n, _ARROW[s.rstrip("?")], nullable=s.endswith("?")) + for n, s in schema.items() + ) + return pa.Table.from_pylist(rows, schema=arrow) + + +def duck_check( + sql: str, + row_schema: dict[str, str], + row_rows: list[dict[str, Any]], + statics: dict[str, pa.Table] | None = None, +) -> None: + statics = statics or {} + model = _row_model(row_schema) + fn = DuckDBInferFn(sql, row_tables={"__THIS__": model}, static_tables=statics) + inputs = [model(**r) for r in row_rows] + got = [r.model_dump() for r in fn.infer({"__THIS__": inputs})] + + con = duckdb.connect() + # Materialize NATIVE tables: duckdb pushes constant filters into + # registered-arrow scans with IEEE NaN semantics, which disagrees with + # its own native-table comparison order (adversarial probe, 2026-07-26). + # The engine follows native-table semantics — the corpus's world. + for name, table in statics.items(): + con.register(f"__arrow_{name}", table) + con.execute(f'CREATE TABLE "{name}" AS SELECT * FROM "__arrow_{name}"') + con.register("__arrow_this", static(row_schema, row_rows)) + con.execute("CREATE TABLE __THIS__ AS SELECT * FROM __arrow_this") + want = con.execute(sql).to_arrow_table().to_pylist() + + # Row order is not part of the contract (a join may reorder); compare as + # multisets of repr'd rows — repr keeps value types apart (1 vs '1' vs + # 1.0) and makes NaN compare equal to itself. + key = lambda r: sorted((k, repr(v)) for k, v in r.items()) # noqa: E731 + assert sorted(map(key, got)) == sorted(map(key, want)), f"{got} != {want}" + + +DIM = static( + {"id": "int", "name": "str"}, + [{"id": 1, "name": "one"}, {"id": 3, "name": "three"}], +) + + +def test_projection_and_where_differential(): + duck_check( + "SELECT a + 1 AS x, a / 2 AS h FROM __THIS__ WHERE a > 0", + {"a": "int"}, + [{"a": 4}, {"a": -1}, {"a": 7}], + ) + + +def test_inner_join_hits_and_misses(): + duck_check( + "SELECT k, name FROM __THIS__ JOIN dim ON k = dim.id", + {"k": "int"}, + [{"k": 1}, {"k": 2}, {"k": 3}], + {"dim": DIM}, + ) + + +def test_left_join_miss_and_null_key(): + duck_check( + "SELECT k, name FROM __THIS__ LEFT JOIN dim ON k = dim.id", + {"k": "int?"}, + [{"k": 1}, {"k": 2}, {"k": None}], + {"dim": DIM}, + ) + + +def test_inner_join_null_key_drops(): + duck_check( + "SELECT name FROM __THIS__ JOIN dim ON k = dim.id", + {"k": "int?"}, + [{"k": 1}, {"k": None}], + {"dim": DIM}, + ) + + +def test_join_key_promotion_int_row_against_float_col(): + duck_check( + "SELECT v FROM __THIS__ JOIN dim ON k = dim.id", + {"k": "int"}, + [{"k": 1}, {"k": 2}], + {"dim": static({"id": "float", "v": "int"}, [{"id": 1.0, "v": 10}])}, + ) + + +def test_join_key_expression_and_where_interaction(): + duck_check( + "SELECT k, name FROM __THIS__ JOIN dim ON k + 1 = dim.id WHERE k >= 0", + {"k": "int"}, + [{"k": 0}, {"k": 2}, {"k": -5}], + {"dim": DIM}, ) - with pytest.raises(NotImplementedError): - fn.infer({"__THIS__": [Row(age=1)]}) + + +def test_duplicate_build_keys_error(): + dup = static({"id": "int", "v": "int"}, [{"id": 1, "v": 10}, {"id": 1, "v": 11}]) + with pytest.raises(ValueError, match="duplicate map key"): + DuckDBInferFn( + "SELECT v FROM __THIS__ JOIN dim ON k = dim.id", + row_tables={"__THIS__": _row_model({"k": "int"})}, + static_tables={"dim": dup}, + ) + + +def test_null_in_value_column_errors(): + holed = static({"id": "int", "v": "int?"}, [{"id": 1, "v": None}]) + with pytest.raises(ValueError, match="NULL in value column"): + DuckDBInferFn( + "SELECT v FROM __THIS__ JOIN dim ON k = dim.id", + row_tables={"__THIS__": _row_model({"k": "int"})}, + static_tables={"dim": holed}, + ) + + +def test_null_key_build_rows_are_dropped(): + # A NULL build key never equi-matches, so the row is dropped rather than + # rejected — probing k=1 still hits the valid entry. + holed = static( + {"id": "int?", "v": "int"}, [{"id": 1, "v": 10}, {"id": None, "v": 99}] + ) + duck_check( + "SELECT v FROM __THIS__ JOIN dim ON k = dim.id", + {"k": "int"}, + [{"k": 1}], + {"dim": holed}, + ) + + +def test_unsupported_is_a_clean_value_error(): + with pytest.raises(ValueError, match="unsupported.*GROUP BY"): + DuckDBInferFn( + "SELECT a FROM __THIS__ GROUP BY a", + row_tables={"__THIS__": _row_model({"a": "int"})}, + static_tables={}, + ) def test_bad_sql_is_a_build_error(): - with pytest.raises(ValueError, match="SQL parse error"): - DuckDBInferFn("SELECT FROM", row_tables={}, static_tables={}) + with pytest.raises(ValueError, match="parse error"): + DuckDBInferFn( + "SELECT FROM", + row_tables={"__THIS__": _row_model({"a": "int"})}, + static_tables={}, + ) + + +def test_unknown_infer_table_is_rejected(): + model = _row_model({"a": "int"}) + fn = DuckDBInferFn( + "SELECT a FROM __THIS__", row_tables={"__THIS__": model}, static_tables={} + ) + with pytest.raises(ValueError, match="unknown table"): + fn.infer({"wrong": [model(a=1)]}) + + +def test_output_model_is_synthesized(): + fn = DuckDBInferFn( + "SELECT a + 1 AS x, b AS y FROM __THIS__", + row_tables={"__THIS__": _row_model({"a": "int", "b": "float?"})}, + static_tables={}, + ) + fields = fn.output_model.model_fields + assert list(fields) == ["x", "y"] + assert fields["x"].annotation is int + assert fields["y"].annotation == float | None + + +# ------------------------------------------------------------- stretch 4: +# builtin catalogue, differential vs duckdb per the measured pins. + + +def test_string_builtins_differential(): + duck_check( + "SELECT upper(s) AS u, lower(s) AS l, trim(s) AS t, ltrim(s, 'a') AS lt, " + "rtrim(s) AS rt, substr(s, 2, 3) AS sub FROM __THIS__", + {"s": "str?"}, + [{"s": " aBc "}, {"s": "abcdef"}, {"s": ""}, {"s": None}], + ) + + +def test_substr_edges_differential(): + duck_check( + "SELECT substr(s, 0, 3) AS a, substr(s, -2) AS b, substr(s, -10, 8) AS c, " + "substr(s, 1, 0) AS d, substr(s, 9) AS e FROM __THIS__", + {"s": "str"}, + [{"s": "hello"}, {"s": "x"}], + ) + + +def test_concat_and_pipes_differential(): + duck_check( + "SELECT n || '!' AS a, 'v=' || n AS b, concat(s, n, 'z') AS c, " + "concat(s) AS d FROM __THIS__", + {"n": "int?", "s": "str?"}, + [{"n": 1, "s": "a"}, {"n": None, "s": None}, {"n": -3, "s": ""}], + ) + + +def test_abs_round_differential(): + duck_check( + "SELECT abs(n) AS an, abs(x) AS ax, round(x) AS rx, round(n) AS rn " + "FROM __THIS__", + {"n": "int?", "x": "float?"}, + [ + {"n": -5, "x": -2.5}, + {"n": 3, "x": 2.5}, + {"n": None, "x": None}, + {"n": 0, "x": -0.4}, + ], + ) + + +def test_rem_by_zero_differential(): + duck_check( + "SELECT a % b AS r FROM __THIS__", + {"a": "int", "b": "int"}, + [{"a": 5, "b": 0}, {"a": 5, "b": 3}, {"a": -7, "b": 2}], + ) + + +def test_float_rem_differential(): + duck_check( + "SELECT x % y AS r FROM __THIS__", + {"x": "float", "y": "float"}, + [{"x": -5.5, "y": 2.5}, {"x": 7.0, "y": 4.0}], + ) + + +def test_coalesce_nullif_differential(): + duck_check( + "SELECT coalesce(n, 9) AS a, coalesce(NULL, n, 9) AS b, " + "nullif(n, 3) AS c, nullif(s, 'x') AS d FROM __THIS__", + {"n": "int?", "s": "str?"}, + [{"n": 3, "s": "x"}, {"n": None, "s": "y"}, {"n": 7, "s": None}], + ) + + +def test_nan_comparison_differential(): + # DuckDB DOUBLE order: NaN = NaN keeps the row. + duck_check( + "SELECT x FROM __THIS__ WHERE x = x", + {"x": "float"}, + [{"x": float("nan")}, {"x": 1.5}], + ) + + +def test_simple_case_mapping_matches_duckdb(): + # Formerly a strict xfail: DuckDB uses utf8proc SIMPLE case maps, Rust + # std only has full maps. src/specializer/exec/casemap.rs now carries the + # measured exception table (see scripts/gen_casemap.py for why it exists + # and why it is dependency-free). + duck_check( + "SELECT upper(s) AS u, lower(s) AS l FROM __THIS__", + {"s": "str"}, + [{"s": "ß"}, {"s": "İ"}, {"s": "ᾀ"}, {"s": "ƛ"}], + ) + + +def test_simple_case_mapping_full_codepoint_census(): + # THE authority on casemap.rs: every Unicode scalar value, chunked into + # long strings (per-codepoint mapping makes one string test them all), + # through both engines. If a duckdb bump shifts utf8proc's tables, this + # fails and scripts/gen_casemap.py regenerates the exception table. + step = 0x8000 + rows = [] + for lo in range(1, 0x110000, step): + s = "".join( + chr(c) + for c in range(lo, min(lo + step, 0x110000)) + if not (0xD800 <= c <= 0xDFFF) + ) + if s: + rows.append({"s": s}) + duck_check("SELECT upper(s) AS u, lower(s) AS l FROM __THIS__", {"s": "str"}, rows) + + +# --------------------------------------------- adversarial-fleet fixes: +# each case below was a measured divergence, now pinned differentially. + + +def test_trim_zs_set_differential(): + duck_check( + "SELECT trim(s) AS t, ltrim(s) AS l, rtrim(s) AS r FROM __THIS__", + {"s": "str"}, + [{"s": "\u00a0a\u3000"}, {"s": "\ta\n"}, {"s": " a "}, {"s": "\u2003a"}], + ) + + +def test_substr_negative_length_differential(): + duck_check( + "SELECT substr(s, st, ln) AS r FROM __THIS__", + {"s": "str", "st": "int", "ln": "int"}, + [ + {"s": "hello", "st": 3, "ln": -2}, + {"s": "hello", "st": 6, "ln": -5}, + {"s": "h\u00e9llo", "st": 4, "ln": -10}, + {"s": "hello", "st": 1, "ln": -1}, + ], + ) + + +def test_float_rendering_differential(): + duck_check( + "SELECT x || '' AS s FROM __THIS__", + {"x": "float"}, + [ + {"x": 1e300}, + {"x": 1e-05}, + {"x": float("nan")}, + {"x": 2.5}, + {"x": 1e16}, + {"x": -1e300}, + ], + ) + + +def test_null_divisor_differential(): + duck_check( + "SELECT a % b AS r FROM __THIS__", + {"a": "int?", "b": "int?"}, + [{"a": 7, "b": None}, {"a": None, "b": 0}, {"a": 7, "b": 3}], + ) + + +def test_numeric_where_differential(): + duck_check( + "SELECT a FROM __THIS__ WHERE a % 2", + {"a": "int"}, + [{"a": 1}, {"a": 2}, {"a": 3}], + ) + + +def test_nan_filter_differential_on_native_tables(): + duck_check( + "SELECT x FROM __THIS__ WHERE x > 0", + {"x": "float?"}, + [{"x": float("nan")}, {"x": 1.0}, {"x": None}, {"x": float("inf")}], + ) + + +# ------------------------------------------------- static-only queries: +# AC #2 — evaluated once at build time by DuckDB, nothing dynamic remains. + + +def test_static_only_query_is_a_constant_emitter(): + dim = static( + {"id": "int", "name": "str"}, + [ + {"id": 1, "name": "one"}, + {"id": 2, "name": "two"}, + {"id": 3, "name": "three"}, + ], + ) + model = _row_model({"a": "int"}) + fn = DuckDBInferFn( + "SELECT name, id * 10 AS x FROM dim WHERE id <> 2 ORDER BY id DESC", + row_tables={"__THIS__": model}, + static_tables={"dim": dim}, + ) + # Input rows are irrelevant; the result is fixed at build time — + # and constructs like ORDER BY work because DuckDB itself evaluated it. + for rows_in in ([], [model(a=1)], [model(a=1), model(a=2)]): + got = [r.model_dump() for r in fn.infer({"__THIS__": rows_in})] + assert got == [ + {"name": "three", "x": 30}, + {"name": "one", "x": 10}, + ] + + +def test_static_only_aggregation_works_via_duckdb(): + dim = static({"v": "int"}, [{"v": 1}, {"v": 2}, {"v": 3}]) + fn = DuckDBInferFn( + "SELECT sum(v) AS s FROM dim", + row_tables={"__THIS__": _row_model({"a": "int"})}, + static_tables={"dim": dim}, + ) + assert [r.model_dump() for r in fn.infer({"__THIS__": []})] == [{"s": 6}] + + +def test_unknown_driving_table_stays_clean_unsupported(): + # Not a static table either -> the original clean unsupported surfaces. + with pytest.raises(ValueError, match="driving relation"): + DuckDBInferFn( + "SELECT x FROM nope", + row_tables={"__THIS__": _row_model({"a": "int"})}, + static_tables={}, + )