Specializer SQL support wave 3: builtin long tail + similarity + VARCHAR subscripts — corpus 265 → 383 (TASK-49) - #34
Merged
claude-agent-ahrzb[bot] merged 6 commits intoJul 26, 2026
Conversation
…ty + string subscripts) created In Progress
…efore implementation (TASK-49) Byte-based similarity (damerau = UNRESTRICTED DL), floored fmod/fdiv vs C-fmod mod, codepoint subscripts with ''-not-NULL out-of-range, oracle- extracted strip_accents map + Hangul compose, reverse descoped (grapheme clusters). Full probe tables in pins-wave3/.
…CHAR subscripts, math tail, strip_accents (TASK-49)
Corpus 265 -> 383 of 678, zero FAILs. Shared semantic fns on both
backends; 500-seed differential green incl. trap agreement.
- Similarity (byte-based, measured): levenshtein==editdist3, UNRESTRICTED
damerau, byte-set jaccard, hamming==mismatches with the shared
Mismatch-Function trap texts; NULL pre-empts the traps via
combined-flag masking to a safe non-empty value (Flogb pattern).
- Builders: repeat, lpad/rpad (codepoint counts, prefix truncation BOTH
sides, data-dependent empty-pad trap masked via len->0), replace,
translate, concat_ws/concat desugar onto Case/Or/Concat (NULL-skipping
with separator; sep never implicitly casts).
- VARCHAR subscripts: array_extract/list_extract, array_slice/list_slice
— codepoint, both-ends-inclusive, out-of-range '' (NOT the list
overload's NULL), NULL never an open bound; step form rejects.
- Inspection: unicode/ord (''->-1) vs ascii (''->0), bit_length =
8*strlen desugar, ucase/lcase aliases.
- strip_accents: oracle-extracted table (scripts/gen_strip_accents.py,
4460 cps, DuckDB's own Unicode vintage) + Hangul jamo compose + the
measured NUL-truncation quirk; generator validates the model on 600
random probes at generation time.
- Math tail: add/subtract/multiply/divide/mod aliases; // operator
(ArithOp::IDiv — truncating on ints, PLAIN division on doubles, zero
divisor -> NULL via the CASE guard); fdiv/fmod floor pair; nextafter
bit-exact. Spec corrected from re-measurement: DOUBLE %-by-zero is NaN
7ff8 (a value), never NULL — the fleet's matrix over-generalized.
- reverse DESCOPED by measurement (UAX-29 grapheme clusters incl.
RI pairing for 3 corpus cases); named rejects for aggregates + regexp.
- Corpus replay: f32 base tables now classify clean-unsupported (engine
is f64-only; the widened grid diverges on nextafter et al).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…trap texts (TASK-49)
Four family test files (similarity / builders / subscripts+inspection /
math tail), fleet-drafted and green against the live oracle: byte-unit
witnesses, unrestricted-DL rows, NaN-bit assertions (fmod fff8 vs % 7ff8),
the data-dependent Insufficient-padding trap split, NULL-pre-empts-trap
rows for jaccard/hamming/lpad, the ascii('')=0 divergence, a ~310-row
strip_accents census, and slice/extract edge grids.
The fleet caught a pre-existing parity gap: overflow trap texts. Now
DuckDB-verbatim with operand values via one shared overflow_msg /
abs_overflow_msg (abs text measured this wave) feeding both backends;
the drafted xfail flipped to positive exact-text assertions.
Gate green: cargo 129, pytest 738 passed + 13 xfailed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…xts (TASK-49) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…bit agreement, not a constant (CI) Windows ucrt says 7ff8, Linux glibc fff8; both engines call the platform libm so they agree per-platform (the cbrt situation again). fmod's NaN is hardware-generated (0*inf under SSE) and stays fff8 everywhere — that constant keeps its pin. Spec addendum updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ahrzb
approved these changes
Jul 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wave 3: builtin long tail + similarity + VARCHAR subscripts — corpus 265 → 383 (TASK-49)
Pins-first as always: a six-family probe fleet measured DuckDB 1.5.5 semantics before any implementation (
docs/superpowers/specs/2026-07-26-wave3-builtin-pins.md+pins-wave3/*.json, 300+ probes).What ships (26 builtins)
levenshtein/editdist3,damerau_levenshtein,jaccard,hamming/mismatches. All raw UTF-8 byte-based (measured); damerau is the unrestricted DL variant (('ca','abc') = 2, not OSA's 3); jaccard is a single-byte set; trap texts verbatim (hamming's errors say "Mismatch Function" — DuckDB's own leak).repeat,lpad/rpad(codepoint counts; truncation keeps the FIRST l codepoints on both sides; the empty-pad trap is data-dependent per row),replace,translate,concat_ws/concat(NULL-skipping-with-separator via a pure desugar onto existing Case/Or/Concat — no new IR).array_extract/list_extract,array_slice/list_slice: codepoints, both-ends-inclusive, out-of-range →''(the LIST overload's NULL is not copied), NULL is never an open bound; step slicing rejects for every step value, matching DuckDB.unicode/ord('' → -1) vsascii('' → 0, the sole divergence),bit_length= 8×strlen desugar,ucase/lcase(exhaustively alias-verified).add/subtract/multiply/divide/modaliases + the//operator (truncating on ints, plain division on doubles, zero divisor → NULL), floor-pairfdiv/fmod(fmod takes the divisor's sign — not C fmod, which is what%is), bit-exactnextafter.strip_accents— oracle-extracted 4460-codepoint table (scripts/gen_strip_accents.py, casemap playbook — DuckDB's Unicode tables lag Unicode 16 by 57 cps, so host libraries are the wrong spec twice) + formulaic Hangul jamo composition + the measured context-dependent NUL truncation.Honest scope calls
reversedescoped by measurement: it's UAX-29 grapheme-cluster-based (ZWJ families, regional-indicator pairing) — real segmentation machinery for 3 corpus cases. Rejects by name; pins retained for a future wave.%was refuted by re-measurement (NaN7ff8, never NULL) — correction recorded in the pins with probes.Verification
fmod→fff8…vs%→7ff8…), NULL-pre-empts-trap rows, and a ~310-row strip_accents census.abstext measured this wave).🤖 Generated with Claude Code