Specializer SQL support wave 3: builtin long tail + similarity + VARCHAR subscripts — corpus 265 → 383 (TASK-49)

backlog/tasks/task-48 - Specializer-SQL-support-LIKE-alias-comma-join-wave-2.md

@@ -1,10 +1,12 @@
 ---
 id: TASK-48
-title: 'Specializer SQL support: LIKE, dynamic-table alias, comma-join rewrite (wave 2)'
-status: In Progress
+title: >-
+  Specializer SQL support: LIKE, dynamic-table alias, comma-join rewrite (wave
+  2)
+status: Done
 assignee: []
 created_date: '2026-07-26 15:05'
-updated_date: '2026-07-26 15:05'
+updated_date: '2026-07-26 16:41'
 labels: []
 milestone: m-7
 dependencies:

backlog/tasks/task-49 - Specializer-SQL-support-builtin-long-tail-similarity-string-subscripts-wave-3.md

@@ -0,0 +1,48 @@
+---
+id: TASK-49
+title: >-
+  Specializer SQL support: builtin long tail + similarity + string subscripts
+  (wave 3)
+status: In Progress
+assignee: []
+created_date: '2026-07-26 16:45'
+updated_date: '2026-07-26 17:44'
+labels: []
+milestone: m-7
+dependencies:
+  - TASK-48
+documentation:
+  - docs/superpowers/specs/2026-07-25-sql-specializer-design.md
+  - docs/superpowers/specs/2026-07-26-wave1-builtin-pins.md
+type: feature
+ordinal: 43000
+---
+
+## Description
+
+<!-- SECTION:DESCRIPTION:BEGIN -->
+Census-measured wave (2026-07-26 replay: 265/678 match, zero FAILs). Pure expression kernels riding the wave-1/2 machinery — pins fleet FIRST, shared semantic fn per op, both backends, differential + trap agreement, corpus zero-FAIL gate. Catalogue by measured first-blocker: SIMILARITY (39 cases): levenshtein/editdist3, damerau_levenshtein, jaccard, hamming/mismatches. STRING TAIL (~35): repeat, concat_ws, lpad/rpad, replace, reverse, translate, unicode/ord, strip_accents (oracle-extracted table, casemap playbook), ucase/lcase aliases, bit_length. STRING SUBSCRIPTS (39): array_extract/array_slice/list_slice on VARCHAR (the blocked sources are function/string/*, not lists). MATH TAIL (~10): add/subtract/multiply/divide aliases, mod, fmod, fdiv, nextafter. REJECT BY NAME: aggregates (sum/count/geomean), regexp_* (RE2, own wave), columns(), list-typed subscripts.
+<!-- SECTION:DESCRIPTION:END -->
+
+## Acceptance Criteria
+<!-- AC:BEGIN -->
+- [x] #1 Pins measured BEFORE implementation for all six families (similarity; pad/repeat/replace/translate/reverse/concat_ws; unicode/bit_length/case-aliases; VARCHAR subscripts; math aliases+fmod/fdiv/mod/nextafter; strip_accents oracle table) — JSON pins + spec addendum committed
+- [x] #2 Every op is one shared semantic fn used verbatim by both backends; differential fuzz extended incl. trap agreement
+- [x] #3 Aggregates and regexp family reject cleanly by name (no behavior change, messages recorded)
+- [x] #4 Corpus replay: flips recorded here, zero FAILs, known-divergent list unchanged or justified
+- [x] #5 mise gate-specializer green
+<!-- AC:END -->
+
+## Implementation Notes
+
+<!-- SECTION:NOTES:BEGIN -->
+Pins committed (2622a14) BEFORE implementation. Fleet: 6 agents, 528k tokens, 300+ probes. Headline refutations: similarity is BYTE-based and damerau is the UNRESTRICTED DL variant (not OSA); fmod is FLOORED (divisor sign) while mod/% is C-fmod (dividend sign), // on doubles is PLAIN division with NULL-on-zero; rpad truncation keeps the PREFIX like lpad; hamming('','') is an ERROR; unicode('')=-1 but ascii('')=0 (sole ascii/unicode divergence); VARCHAR out-of-range subscripts give '' where LIST gives NULL and NULL is never an open bound; strip_accents = oracle map (4460 cps) + Hangul compose + context-dependent NUL truncation, DuckDB tables lag Unicode 16 by 57 cps. SCOPE CHANGE: reverse DESCOPED (measured grapheme-cluster/UAX-29 semantics incl. RI pairing = real segmentation machinery for 3 corpus cases; rejects by name, pins retained). concat/concat_ws restricted to VARCHAR args (DuckDB implicit float rendering not modeled). nextafter f64 only (FLOAT overload is f32; f32 columns already reject).
+
+IMPLEMENTATION COMPLETE. Corpus 265 -> 383 of 678, zero FAILs (deterministic, replayed twice); gate green (738 passed, 13 xfailed; cargo 129). New IR: StrOp2 += Levenshtein/Damerau/Jaccard/Hamming, StrOp3 (Replace/Translate), StrOp2i (Repeat/Extract), Spad, Sslice, Sord, StrOp1 += StripAccents, BinOp += Ffloordiv/Ffloormod/Fnextafter, ArithOp::IDiv (// operator + divide()); concat_ws/concat desugar onto Case/Or/Concat with the has-prior-non-null fold — no new IR. Shared semantic fns verbatim in both backends; 500-seed differential incl. trap agreement green. NULL-pre-empts-trap masking: Jaccard/Hamming mask BOTH operands to 'a' under the combined flag ('' is IN their trap domain — the Flogb pattern); Spad masks len->0. Overflow trap texts now DuckDB-verbatim with operand values (test-fleet finding, fixed in-wave; abs text measured too). f32 base tables classify clean-unsupported in the replay (nextafter's f32-grid was the wave's only FAIL class). 57 new oracle tests in 4 files (tests/test_duckdb_wave3_*.py), drafted by a 4-agent fleet, all green incl. NaN-bit assertions (fmod fff8 vs % 7ff8) and the data-dependent Insufficient-padding trap split.
+<!-- SECTION:NOTES:END -->
+
+## Final Summary
+
+<!-- SECTION:FINAL_SUMMARY:BEGIN -->
+Wave 3 complete: 26 builtins across six measured families — similarity (byte-based levenshtein/editdist3, unrestricted damerau, byte-set jaccard, hamming/mismatches with verbatim trap texts), string builders (repeat, lpad/rpad with the data-dependent empty-pad trap, replace, translate, concat_ws/concat via a pure desugar), VARCHAR subscripts (array_extract/list_extract, array_slice/list_slice — '' out-of-range, NULL never an open bound), inspection (unicode/ord/ascii, bit_length, ucase/lcase), math tail (add/subtract/multiply/divide/mod aliases, the // operator, floor-pair fdiv/fmod, bit-exact nextafter), and strip_accents from an oracle-extracted 4460-codepoint table + Hangul jamo composition. reverse descoped by measurement (UAX-29 grapheme clusters for 3 corpus cases). Overflow trap texts brought to DuckDB's verbatim, operand-interpolated forms. Corpus 265 -> 383 of 678, zero FAILs; f32 base tables now classify honestly as clean-unsupported. Gate green: cargo 129, pytest 738 + 13 xfail, 500-seed cross-backend differential incl. trap agreement, 57 new oracle tests.
+<!-- SECTION:FINAL_SUMMARY:END -->

docs/superpowers/specs/2026-07-26-wave3-builtin-pins.md

@@ -0,0 +1,178 @@
+# Wave-3 builtin pins — DuckDB 1.5.5, measured 2026-07-26
+
+The implementation contract for TASK-49. Every claim was MEASURED through the
+vectorized path (table columns, not literals) by a six-family probe fleet; the
+full pin tables (300+ probes with exact reprs, float bit patterns, result
+types, and verbatim error heads) are the JSON files in `pins-wave3/`. Literal
+(constant-fold) spot checks matched the column path in every family unless
+noted. Nothing here is inferred from documentation.
+
+## Similarity — levenshtein, editdist3, damerau_levenshtein, jaccard, hamming, mismatches
+
+- UNIT is raw UTF-8 **bytes** for all six — never codepoints or graphemes.
+  Witnesses: levenshtein('é','e')=2, jaccard('é','è')=1/3 (the two codepoints
+  share lead byte 0xC3 — codepoint-set semantics would give 0.0),
+  hamming('é','e') errors on byte-length mismatch while hamming('é','è')=1.
+- Types: levenshtein/editdist3/damerau_levenshtein/hamming/mismatches →
+  BIGINT; jaccard → DOUBLE. All NULL-strict every arg. All case-sensitive.
+- `editdist3` == `levenshtein` (200-pair sweep, 0 diffs). Plain Levenshtein,
+  empty strings fine: ('','')=0, ('','abc')=3.
+- `damerau_levenshtein` is the **UNRESTRICTED** DL variant, transposition
+  cost 1 — NOT restricted OSA. Witness: ('ca','abc')=2 (OSA gives 3);
+  300-pair sweep 0 diffs vs an unrestricted-DL reference, 3 diffs vs OSA.
+- `jaccard` = |A∩B|/|A∪B| over **single-byte sets**, duplicates ignored:
+  ('ab','ba')=1.0, ('abc','abd')=0.5. Empty string on either side TRAPS:
+  "Invalid Input Error: Jaccard Function: An argument too short!".
+- `hamming` == `mismatches` (identical values AND error texts — the errors
+  say "Mismatch Function" even for hamming). Byte-length mismatch traps
+  "…Strings must be of equal length!"; ANY empty input (both empty too)
+  traps "…Strings must be of length > 0!" — ('','') is an ERROR, not 0.
+- Embedded NUL is an ordinary byte everywhere.
+
+## String builders — repeat, lpad, rpad, replace, translate, concat_ws, concat
+
+- `repeat(s,n)`: n≤0 → '' silently; NULL-strict; multi-byte safe. Huge n
+  deliberately unpinned (OOM risk) — corpus uses small n.
+- `lpad/rpad(s,l,pad)`: l counts **codepoints**. Truncation (l < length(s))
+  keeps the FIRST l codepoints for BOTH lpad and rpad (rpad('abcdef',3,'x')
+  = 'abc' — a naive rpad keeps the suffix; refuted). l≤0 → ''. pad cycles
+  left-to-right cut to (l−length(s)) codepoints, never splitting a
+  codepoint. pad='' traps "Invalid Input Error: Insufficient padding in
+  LPAD." (resp. RPAD) ONLY when growth is needed (l > length(s)); otherwise
+  the prefix returns without error — the trap is data-dependent. NULL-strict
+  all three args.
+- `replace(s,from,to)`: empty needle is a strict NO-OP; leftmost
+  non-overlapping single pass, output not rescanned ('aaa','aa','b'→'ba');
+  byte-sequence match; NULL-strict.
+- `translate(s,from,to)`: per-**codepoint** map; from-chars beyond |to| are
+  DELETED; duplicate in from → FIRST wins; to-extras ignored; to='' deletes
+  every from-char; NULL-strict.
+- `concat_ws(sep, args…)`: NULL args are SKIPPED with their separator;
+  NULL sep → NULL; all args NULL → '' (NOT NULL). Zero value-args is a
+  binder error. `concat(args…)` = same skip-NULL fold joined by '' — and is
+  NOT sugar for `||`, which stays NULL-strict. DuckDB implicitly casts
+  numeric/boolean VALUE args with its own float rendering ('1e+20',
+  '-0.0'); the SEPARATOR never casts. v0 restricts both functions to
+  VARCHAR args — non-VARCHAR args reject by name (we do not model DuckDB
+  float-to-string rendering).
+- `reverse` is **DESCOPED**: measured to operate on UAX-29 grapheme
+  clusters (combining sequences attached, ZWJ emoji one unit, regional-
+  indicator pairs swap as units — real segmentation, not an approximation).
+  Full segmentation machinery for 3 corpus cases fails the cost test;
+  rejects by name citing grapheme semantics. Pins retained for a future
+  wave.
+
+## Inspection + case aliases — unicode, ord, ascii, bit_length, ucase, lcase
+
+- `unicode(s)`/`ord(s)`: FIRST codepoint as INTEGER; multi-char fine
+  ('abc'→97); `unicode('')` = **-1**. `ord` == `unicode` (exhaustive
+  1,112,063-codepoint sweep + 210 strings, 0 diffs).
+- `ascii(s)` == unicode EXCEPT `ascii('')` = **0** — the sole divergence,
+  and ascii does NOT restrict to ASCII (ascii('é')=233).
+- `bit_length(s)` = 8 × strlen(s) exactly (bytes; BIGINT) — pure desugar.
+  `octet_length` does not exist in DuckDB 1.5.5 (binder error).
+- `ucase`/`lcase` == `upper`/`lower`: exhaustive all-codepoint sweep both
+  directions, zero mismatches — pure aliases onto the existing casemap ops.
+
+## VARCHAR subscripts — array_extract, list_extract, array_slice, list_slice, s[i], s[a:b]
+
+- UNIT = **codepoints** (not graphemes: extract(2) of 'e'+U+0301 is the bare
+  combining mark).
+- `array_extract(s,i)` (== `list_extract` == `s[i]`, 220-row sweeps): 1-based;
+  i=0 → ''; negative = from end (-1 last, resolution len+1+i with clamping);
+  out-of-range EITHER direction → **''** (the LIST overload gives NULL — do
+  not copy list semantics); NULL-strict; VARCHAR result.
+- `array_slice(s,a,b)` (== `list_slice` == `s[a:b]`, 220-row sweeps):
+  both-ends-INCLUSIVE, 1-based; negative from-end (-1 = last char); a≤0
+  clamps to start; b>len clamps to end; fully out-of-range or a>b → '';
+  NULL bound → NULL (**NULL is NOT an open bound** — open bounds exist only
+  syntactically: s[:b] ≡ slice(1,b), s[a:] ≡ slice(a,-1), s[:] ≡
+  slice(1,-1)).
+- Step form REJECTS for every step value incl. 1: "Not implemented Error:
+  Slice with steps has not been implemented for string types, …".
+- `substr(s,2,3)`='ell' vs `array_slice(s,2,3)`='el' — (start,LENGTH) vs
+  (begin,end) conventions; do not share lowering.
+
+## Math tail — add, subtract, multiply, divide, mod, fmod, fdiv, nextafter
+
+- `add/subtract/multiply/divide/mod` are EXACT aliases of `+ - * // %`:
+  same values, types, and byte-identical error texts (overflow traps incl.
+  INT64_MIN special cases) — pure frontend desugars onto existing ops.
+  Unary add(x)=x, subtract(x)=-x also exist.
+- `divide`(int,int) = truncating INT division (7,2 → 3); on DOUBLE, `//`
+  and divide() are **plain division** (−7.5//2.0 = −3.75, NOT floor) with
+  divisor==0 → NULL.
+- `mod`/`%` = truncated C-fmod (dividend's sign) on ints AND doubles;
+  INT x%0 → NULL (no trap); DOUBLE x%0.0 → **NaN 7ff8…** (a value, NOT
+  NULL — the fleet's decision line over-generalized from the int probes;
+  re-measured 2026-07-26, correction appended to pins-wave3/math_tail.json.
+  NaN-sign addendum (CI-discovered): the %-by-zero NaN comes from LIBM
+  fmod and its sign is PLATFORM-dependent — 7ff8 on Windows ucrt, fff8 on
+  Linux glibc; both engines use the platform libm so they agree with the
+  oracle per-platform, and the pin is bit AGREEMENT, not a constant.
+  fmod-by-zero is hardware-generated (0·inf under SSE) and stays fff8
+  everywhere); mod(−7.5,2.5) = **−0.0**.
+- `fmod`/`fdiv` are the FLOOR-division pair, always DOUBLE: fdiv =
+  floor(x/y) (±inf on zero divisor); fmod takes the **DIVISOR's** sign
+  (fmod(−7.5,2.5) = +0.0 where mod gives −0.0) and is computed as
+  x − floor(x/y)·y — so fmod(1.0, inf) = NaN (not 1.0 as C fmod);
+  fmod(x,0) = NaN. Computed NaNs surface as fff8… (negative quiet NaN)
+  where mod-by-zero NULLs and propagated NaNs stay 7ff8… — reproduce by
+  computing naturally on x86, verified by bit-exact pin tests.
+- `nextafter` = C nextafter bit-exact, TOTAL (no traps): x==y returns y
+  (nextafter(0.0,−0.0) = −0.0); denormal/inf/NaN edges pinned by bits. Int
+  args promote to DOUBLE. The (FLOAT,FLOAT) overload returns FLOAT (f32
+  nextafterf) — out of v0 scope (f32 columns already reject at binding).
+- VARCHAR never implicitly casts into any of these (binder error) — matches
+  our no-implicit-cast rule.
+
+## strip_accents — oracle-extracted table + Hangul compose
+
+- NOT purely per-codepoint: per-cp map, THEN a canonical-compose pass whose
+  only observable effect is **Hangul jamo composition** (L+V→LV, LV+T→LVT,
+  including precomposed LV + T; formula 0xAC00+(L−0x1100)·588+(V−0x1161)·28
+  +(T−0x11A7), verified over all 399 LV pairs + 200 LVT triples). Wrong-order
+  jamo do not compose.
+- The per-cp map is ORACLE-EXTRACTED (full non-surrogate sweep): 4460
+  changed codepoints; 2450 map to '' (Mn/Mc/Me marks — Indic vowel signs
+  are deleted, not just accents); every non-empty output is exactly ONE
+  codepoint. Includes accent-free rewrites (U+212A→'K', U+2126→Ω,
+  U+2000→U+2002). Compatibility decompositions NOT applied.
+- DuckDB's tables lag Unicode 16 by 57 codepoints — the Rust table is
+  generated from the oracle map (scripts/gen_strip_accents.py, casemap/pow10
+  playbook), NEVER from a host Unicode library.
+- Row-local algorithm (validated 518/518 strings): all-ASCII input →
+  returned VERBATIM (embedded NULs preserved); otherwise truncate at the
+  first NUL (context-dependent NUL quirk — even a map-unchanged non-ASCII
+  char like an emoji triggers the truncating path), apply the map, compose
+  Hangul. Idempotent; total; NULL-strict.
+
+## Implementation addenda (found while building/testing)
+
+- Overflow trap texts are now DuckDB's verbatim, operand values included:
+  "Overflow in addition of INT64 (x + y)!" (sub/mul likewise), "Overflow
+  in division of x / y" for BOTH `//` and `%` on i64::MIN op −1 (no
+  trailing '!'), and "Overflow on abs(x)" (measured this wave). One
+  shared `overflow_msg`/`abs_overflow_msg` feeds both backends.
+- hamming's two traps check EQUAL LENGTH first: hamming('', 'a') raises
+  the equal-length message; only ('','') reaches "length > 0".
+- Documented laxity (same stance as wave-1 round digits): the engine
+  binds lpad/rpad with an i64 length column where DuckDB wants INTEGER;
+  constant integer lengths — the corpus shape — bind in both.
+- concat/concat_ws literal -0.0 renders '0.0' in DuckDB (DECIMAL literal
+  path); the pinned '-0.0' is the DOUBLE column path, which we follow.
+- Corpus replay: f32 base tables classify clean-unsupported — widening
+  to f64 is value-exact but every f32-GRID-sensitive op (nextafter ulp
+  steps, FLOAT→VARCHAR rendering) computes on the wrong grid (2 nextafter
+  cases were the wave's only FAILs before this rule; 3 sources, 5 cases).
+- Unary add(x)=x / subtract(x)=-x exist in DuckDB but are unpinned —
+  they reject by arity until measured.
+
+## Catalogue rejections (do NOT ship)
+
+- Aggregates (`sum`, `count`, `geomean`) reject as aggregate-by-name.
+- `regexp_matches`/`regexp_extract`/`regexp_full_match`: RE2 semantics, own
+  wave.
+- `reverse`: grapheme-cluster semantics (above).
+- Step slicing on VARCHAR, non-VARCHAR args to concat/concat_ws,
+  column-count-changing star macros (`columns(...)`).