Specializer SQL support: workload builtins & predicates wave 1 — corpus 172 → 240 (TASK-47)

backlog/tasks/task-46 - Specializer-SQL-support-SELECT-star-expansion.md

@@ -1,10 +1,10 @@
 ---
 id: TASK-46
 title: 'Specializer SQL support: SELECT * star expansion'
-status: In Progress
+status: Done
 assignee: []
 created_date: '2026-07-26 11:42'
-updated_date: '2026-07-26 11:55'
+updated_date: '2026-07-26 12:20'
 labels: []
 milestone: m-7
 dependencies:
@@ -46,3 +46,9 @@ Measured pins: expansion order is FROM order then declared column order per tabl
 <!-- SECTION:NOTES:BEGIN -->
 Landed in one pass: Binder::expand_star (frontend.rs) expands `*` and `tbl.*` at the single projection site using the existing SKind::Col / static_lane constructors — no IR/backend/boundary changes, exactly as scoped. EXCLUDE handles bare + paren forms case-insensitively (sqlparser 0.62 carries entries as ObjectName; single-part idents only, qualified entries reject by name); unknown-column and duplicate-entry EXCLUDE mirror DuckDB's binder errors; exclude-all falls through to the existing empty-SELECT bind error (same shape as DuckDB's "SELECT list is empty after resolving * expressions"). Star items covering a joined static table reject cleanly naming the join-key column (DuckDB includes the key AND emits duplicate output names there — both unrepresentable; `__THIS__.*` under a join works). ILIKE/EXCEPT/REPLACE/RENAME/COLUMNS() reject by name. CORPUS RESULT (AC #2): 53 -> 172 match of 678 (+119, a 3.2x coverage jump), 0 FAILs, 506 clean-unsupported. New first-blocker head: BETWEEN 31, comma join 30, dynamic-table alias 30, table-function FROM 24, then the builtin catalogue (array_slice 23, contains 18, damerau_levenshtein 17, ...) — the TASK-47 target list confirmed. Gate green: cargo 129 + pytest 627 (13 xfail incl. the pre-existing substr const-fold pin from master).
 <!-- SECTION:NOTES:END -->
+
+## Final Summary
+
+<!-- SECTION:FINAL_SUMMARY:BEGIN -->
+Merged as PR #30 (2026-07-26). Star expansion at the frontend's single projection site tripled corpus coverage: 53 -> 172 match of 678, zero FAILs. All measured edges pinned (order, EXCLUDE case-folding + binder errors, star-over-join rejection naming the join key, macro forms rejected by name). Gate green cargo 129 / pytest 627. Unblocked TASK-47's target list: post-star first-blocker head is BETWEEN 31, comma join 30, dynamic-table alias 30, then the builtin catalogue.
+<!-- SECTION:FINAL_SUMMARY:END -->

backlog/tasks/task-47 - Specializer-SQL-support-workload-builtins-and-predicates-wave-1.md

@@ -1,9 +1,10 @@
 ---
 id: TASK-47
 title: 'Specializer SQL support: workload builtins & predicates wave 1'
-status: To Do
+status: In Progress
 assignee: []
 created_date: '2026-07-26 11:42'
+updated_date: '2026-07-26 12:20'
 labels: []
 milestone: m-7
 dependencies:
@@ -23,9 +24,31 @@ Close the workload ladder measured by the serving-bench scenarios (benchmarks/se
 
 ## Acceptance Criteria
 <!-- AC:BEGIN -->
-- [ ] #1 Each shipped function/predicate has measured DuckDB pins recorded as duck_check tests before its implementation landed (domain edges, NULL, special floats)
-- [ ] #2 Interpreter and cranelift agree byte-identically on all new ops (shared semantic fns; 500-seed differential extended to cover them)
-- [ ] #3 The four serving scenarios' compromises lists re-audited: every gap this wave claims to close is exercised by an upgraded scenario feature or a new duck_check
-- [ ] #4 Corpus replay: predicate/function first-blocker cases flip to match or to a named deeper blocker; zero FAILs; new tally recorded here
-- [ ] #5 mise gate-specializer green
+- [x] #1 Each shipped function/predicate has measured DuckDB pins recorded as duck_check tests before its implementation landed (domain edges, NULL, special floats)
+- [x] #2 Interpreter and cranelift agree byte-identically on all new ops (shared semantic fns; 500-seed differential extended to cover them)
+- [x] #3 The four serving scenarios' compromises lists re-audited: every gap this wave claims to close is exercised by an upgraded scenario feature or a new duck_check
+- [x] #4 Corpus replay: predicate/function first-blocker cases flip to match or to a named deeper blocker; zero FAILs; new tally recorded here
+- [x] #5 mise gate-specializer green
 <!-- AC:END -->
+
+## Implementation Plan
+
+<!-- SECTION:PLAN:BEGIN -->
+Stretch plan (recorded 2026-07-26). Measurement BEFORE implementation, per the builtin-pins discipline; a parallel measurement fleet pins each family against DuckDB 1.5.5 first, and lowering decisions are finalized from those pins.
+1. Measurement fleet (6 families, parallel): log/ln/log2/log10/exp; floor/ceil/trunc + round(x, digits); pow/sqrt (+ the ^ operator); sin/cos; string search (instr/position/strpos/contains/starts_with/ends_with/length); predicates (BETWEEN incl. NULL bounds and NOT, IN-list incl. NULL-element three-valued logic and NOT IN, least/greatest NULL policy). Each family delivers: measured pin table (edges: 0/negative domains, NULL, NaN, +-0.0, +-inf, int vs float overloads, return types, error messages), draft duck_check tests, and a lowering proposal (frontend desugar vs new IR op with the exact Rust semantics fn).
+2. Frontend desugars first (no IR changes where the pins allow): BETWEEN -> >= AND <= under Kleene; IN-list -> OR chain of equalities (three-valued logic makes this exact); NOT variants; least/greatest per measured NULL policy (CASE-chain if NULL-ignoring is expressible, else an IR op). Corpus predicates flip here.
+3. New IR ops for the rest: math unaries/binaries and string-search ops, implemented ONCE as shared semantic functions used by both backends (interp closures + cranelift helpers), fuzz generator extended so the 500-seed differential covers every new op; catalogue entries with the pinned edge/trap behavior.
+4. Re-audit the four serving scenarios' compromises lists (AC #3): upgrade scenario features the wave unblocks (log1p amount/fare/sales, floor decade bins, instr title extraction, IN-list flags), keeping three-way parity green; corpus re-tally into this ticket (AC #4); gate green (AC #5).
+<!-- SECTION:PLAN:END -->
+
+## Implementation Notes
+
+<!-- SECTION:NOTES:BEGIN -->
+Progress (2026-07-26, branch claude/specializer-builtins-wave1). Landed, each oracle-pinned before implementation and green on both backends: (1) BETWEEN/IN as exact K3 desugars with whole-construct f64 unification (corpus 172->179; remaining BETWEEN cases moved to NAMED deeper blockers — COUNT aggregation shapes). (2) Fourteen math ops (ln/log/log2/log10/exp/sqrt/cbrt/sin/cos/tan/floor/ceil/trunc/pow + log(b,x)) — shared semantic fns, safe-masked NULL lowering (log family's trap domain includes the type default; Flogb masks under the COMBINED flag since NULL pre-empts every domain check), fuzz-covered incl. trap agreement; the ^ operator stays cleanly unsupported (sqlparser parses it below *, DuckDB binds it above — measured tree divergence), ** does not parse. (3) String search (instr/strpos/position incl. the needle-first SQL form, contains, starts_with/ends_with + prefix/suffix, length/char_length/len, strlen) — 1-based codepoint positions, empty-needle-matches-all, NULL-strict; the corpus REFUTED the fleet's contains-NULL blanket-error pin, re-measured to the real rule (NULL needle binds iff a non-literal Str haystack anchors resolution). (4) least/greatest as CASE+duck-order composition (NULL-ignoring, first-arg ties, NaN-above-inf) — no IR op. CORPUS: 53 (start of day) -> 172 (star) -> 179 (predicates) -> 237 match of 678, zero FAILs, gate green (cargo 129 / pytest 667). COMPLETE: round/trunc-with-digits landed with the oracle-extracted pow10 table (gen_pow10.py, ulp witnesses k=23/k=126 checked at generation; wrapping i64 semantics; round/trunc non-finite asymmetry preserved) — corpus 240/678, zero FAILs. Scenario re-audit done by a parity-gated 4-agent fleet: 47 famous-solution features restored (titanic 24->31, ames 42->54, fraud 41->57, rossmann 44->56 outputs) — ln(1+x) everywhere, decade buckets, Deotte cents, IN-set flags, domain parsing via starts_with/ends_with/instr, sin/cos cyclical encodings, least/greatest clamps; three-way parity green on all four. Final gate: cargo 129 / pytest 671 / corpus 240 match, 0 FAIL.
+<!-- SECTION:NOTES:END -->
+
+## Final Summary
+
+<!-- SECTION:FINAL_SUMMARY:BEGIN -->
+Wave 1 complete on claude/specializer-builtins-wave1. Shipped, all measured-first: BETWEEN/IN/least/greatest as exact compositions (zero IR); 14 math ops + log(b,x) + round/trunc-with-digits (oracle pow10 table) + 6 string-search ops on both backends via shared semantic fns, fuzz-covered incl. trap agreement. Corpus 53 -> 240 match of 678 across the day (star 172, predicates 179, builtins 240), zero wrong answers at every step. Two measured rejections ship as named unsupported: the ^ operator (sqlparser precedence contradicts DuckDB) and ** (does not parse). One fleet pin was refuted by the corpus and re-measured (contains-NULL overload anchoring). The four serving scenarios regained 47 real famous-solution features under the standing three-way parity gate.
+<!-- SECTION:FINAL_SUMMARY:END -->

benchmarks/bench_serving.py

@@ -22,6 +22,9 @@
 any timing; a scenario that disagrees aborts the bench.
 
 Run: uv run python -m benchmarks.bench_serving [--json out.json]
+IMPORTANT: rebuild the wheel first (uv run --reinstall-package sql-transform
+python -c pass) — a stale wheel inflates ONLY the engine rows and once
+produced a phantom 7x regression (caught by bisection, 2026-07-26).
 """
 
 from __future__ import annotations

benchmarks/serving_scenarios/fraud_txn.py

@@ -3,8 +3,11 @@
 Serving shape of the famous IEEE-CIS Fraud Detection winning solutions:
 uid/frequency/target encodings materialized at prepare time become plain
 LEFT-JOIN lookup tables at serve time; everything else is scalar math over
-the raw transaction row.
-"""
+the raw transaction row — including the wave-1 builtin features the winning
+solutions actually used: log1p amounts via ln(1.0 + x), the Deotte cents
+feature via round(x, 2), amount decade via round(x, -1), cyclical hour/dow
+sin/cos encodings, email-domain group flags via IN / starts_with / ends_with
+/ instr, and C-column outlier clips via least()."""
 
 import math
 import random
@@ -18,12 +21,14 @@
     "IEEE-CIS Fraud Detection (Kaggle 2019, Deotte/Yakovlev winning-solution "
     "serving shape): composite uid (card1 x addr1) + card1/addr1/email-domain "
     "frequency+target encodings as join tables, amt ratio/delta vs card1 and "
-    "uid means, cents feature, D-column NULL flags, hour/day-of-week from "
-    "unix ts, amount buckets, ProductCD one-hots."
+    "uid means, log1p amount, rounded cents feature, amount decade, D-column "
+    "NULL flags, hour/day-of-week from unix ts + cyclical sin/cos encodings, "
+    "amount buckets, ProductCD one-hots, email-domain group/suspicious flags, "
+    "clipped C-columns."
 )
 
 N_INPUT_COLS = 32
-N_OUTPUT_COLS = 41
+N_OUTPUT_COLS = 57
 
 ROW_SCHEMA: dict[str, str] = {
     "txn_id": "int",
@@ -303,7 +308,28 @@ def make_rows(seed: int, n: int) -> list[dict]:
   c1 / (c1 + c2 + c3 + c4 + c5 + c6 + 1) AS c1_share,
   coalesce(dist1, 0.0) AS dist1_filled,
   CASE WHEN m1 IS NULL THEN -1 WHEN m1 THEN 1 ELSE 0 END AS m1_flag,
-  CASE WHEN addr1 IS NULL THEN 0 ELSE 1 END AS addr_known
+  CASE WHEN addr1 IS NULL THEN 0 ELSE 1 END AS addr_known,
+  ln(1.0 + transaction_amt) AS amt_log1p,
+  round(transaction_amt, -1) AS amt_decade,
+  round(transaction_amt - floor(transaction_amt), 2) AS amt_cents_r,
+  sin(2.0 * pi() * (((transaction_dt % 86400) - (transaction_dt % 3600)) / 3600) / 24.0) AS hour_sin,
+  cos(2.0 * pi() * (((transaction_dt % 86400) - (transaction_dt % 3600)) / 3600) / 24.0) AS hour_cos,
+  sin(2.0 * pi() * ((CAST((transaction_dt - (transaction_dt % 86400)) / 86400 AS INTEGER) + 4) % 7) / 7.0) AS dow_sin,
+  cos(2.0 * pi() * ((CAST((transaction_dt - (transaction_dt % 86400)) / 86400 AS INTEGER) + 4) % 7) / 7.0) AS dow_cos,
+  CASE WHEN p_email_domain IN ('protonmail.com', 'protonmail.ch', 'anonymous.com', 'mail.com', 'qq.com')
+       THEN 1 ELSE 0 END AS p_dom_suspicious,
+  CASE WHEN p_email_domain IS NULL THEN 0
+       WHEN starts_with(p_email_domain, 'gmail') THEN 1 ELSE 0 END AS p_is_gmail,
+  CASE WHEN p_email_domain IN ('hotmail.com', 'outlook.com', 'live.com', 'msn.com')
+       THEN 1 ELSE 0 END AS p_is_msft,
+  CASE WHEN p_email_domain IS NULL THEN 0
+       WHEN ends_with(p_email_domain, '.net') THEN 1 ELSE 0 END AS p_dom_net,
+  CASE WHEN p_email_domain IS NULL THEN -1
+       ELSE length(p_email_domain) - instr(p_email_domain, '.') END AS p_dom_suffix_len,
+  least(c1, 50) AS c1_clip,
+  least(c2, 40) AS c2_clip,
+  greatest(coalesce(d1, 0), coalesce(d2, 0), coalesce(d3, 0), coalesce(d4, 0)) AS d_max,
+  ln(1.0 + card1_stats.card1_txn_cnt) AS card1_cnt_log
 FROM __THIS__
 LEFT JOIN card1_stats ON card1 = card1_stats.card1_id
 LEFT JOIN addr1_stats ON addr1 = addr1_stats.addr1_id
@@ -314,6 +340,20 @@ def make_rows(seed: int, n: int) -> list[dict]:
 """
 
 
+def _round_half_away(x: float, n: int) -> float:
+    """SQL round(x, n): scale by 10**n, round half away from zero, unscale.
+    Bit-identical to the engine/DuckDB for this scenario's inputs (probed)."""
+    s = 10.0**n
+    y = x * s
+    return math.copysign(math.floor(abs(y) + 0.5), y) / s
+
+
+_SUSPICIOUS_DOMAINS = frozenset(
+    {"protonmail.com", "protonmail.ch", "anonymous.com", "mail.com", "qq.com"}
+)
+_MSFT_DOMAINS = frozenset({"hotmail.com", "outlook.com", "live.com", "msn.com"})
+
+
 def handcrafted(statics: dict[str, pa.Table]) -> Callable[[dict], dict]:
     """What a competent engineer hand-writes for the same features: hydrate the
     encoding tables into plain dicts once, then a per-row closure."""
@@ -342,6 +382,8 @@ def handcrafted(statics: dict[str, pa.Table]) -> Callable[[dict], dict]:
     def fn(row: dict) -> dict:
         amt = row["transaction_amt"]
         ts = row["transaction_dt"]
+        hour = (ts % 86400) // 3600
+        dow = (ts // 86400 + 4) % 7
 
         c1s = card1_map.get(row["card1"])
         card1_amt_mean, card1_txn_cnt, card1_fraud_rate = (
@@ -405,8 +447,8 @@ def fn(row: dict) -> dict:
             "amt_x_pcd_rate": amt * pcd_fraud_rate
             if pcd_fraud_rate is not None
             else None,
-            "txn_hour": (ts % 86400) // 3600,
-            "txn_dow": (ts // 86400 + 4) % 7,
+            "txn_hour": hour,
+            "txn_dow": dow,
             "is_night": 1 if ts % 86400 < 21600 else 0,
             "amt_bucket": amt_bucket,
             "amt_cents": amt_cents,
@@ -429,6 +471,30 @@ def fn(row: dict) -> dict:
             "dist1_filled": row["dist1"] if row["dist1"] is not None else 0.0,
             "m1_flag": -1 if m1 is None else (1 if m1 else 0),
             "addr_known": 0 if row["addr1"] is None else 1,
+            "amt_log1p": math.log(1.0 + amt),
+            "amt_decade": _round_half_away(amt, -1),
+            "amt_cents_r": _round_half_away(amt - math.floor(amt), 2),
+            "hour_sin": math.sin(2.0 * math.pi * hour / 24.0),
+            "hour_cos": math.cos(2.0 * math.pi * hour / 24.0),
+            "dow_sin": math.sin(2.0 * math.pi * dow / 7.0),
+            "dow_cos": math.cos(2.0 * math.pi * dow / 7.0),
+            # NULL IN (...) is NULL -> CASE falls through to 0, same as .get miss
+            "p_dom_suspicious": 1 if p_dom in _SUSPICIOUS_DOMAINS else 0,
+            "p_is_gmail": 1 if p_dom is not None and p_dom.startswith("gmail") else 0,
+            "p_is_msft": 1 if p_dom in _MSFT_DOMAINS else 0,
+            "p_dom_net": 1 if p_dom is not None and p_dom.endswith(".net") else 0,
+            # instr is 1-based codepoints; domains are ASCII with one dot
+            "p_dom_suffix_len": -1
+            if p_dom is None
+            else len(p_dom) - (p_dom.index(".") + 1),
+            "c1_clip": min(row["c1"], 50),
+            "c2_clip": min(row["c2"], 40),
+            "d_max": max(
+                row["d1"] or 0, row["d2"] or 0, row["d3"] or 0, row["d4"] or 0
+            ),
+            "card1_cnt_log": math.log(1.0 + card1_txn_cnt)
+            if card1_txn_cnt is not None
+            else None,
         }
 
     return fn