Add mask functions to allow gandiva evaluation.

[lriggs]

DX-117112: HiveMaskConvertlets bypasses Gandiva mask functions, forcing Java execution

Why

Dremio's HiveMaskConvertlets rewrites every SQL MASK* call into mask_internal before
function resolution. mask_internal had no native implementation, so all string masking ran in Java and every MASK* split reported Split Evaluated in Gandiva: false.
Gandiva's existing mask / mask_first_n / mask_last_n / mask_show_first_n /
mask_show_last_n were unreachable dead code.

Redirecting the convertlet to those existing natives was the obvious fix and turns out to be
wrong: they do not compute the same thing as Hive. This PR makes the native functions match
Hive and adds the one signature Dremio actually emits.

Changes

1. Only Lu, Ll and Nd are masked

Both mask families now mask uppercase letters, lowercase letters and decimal digits, and pass
every other Unicode general category through — matching Hive's GenericUDFMaskBase and
Dremio's MaskTransformer.

• gdv_mask_first_n_utf8_int32 / gdv_mask_last_n_utf8_int32 dropped an undocumented
  case 10 (Nl) and now use named UTF8PROC_CATEGORY_* enums instead of bare integers.
• mask_utf8_utf8_utf8_utf8 dropped Lt, Lo, Nl and No.

This is a bug fix, not a preference. Three pieces of evidence:

• The two families disagreed with each other, and both contradictions were pinned by
  passing tests: gdv_function_stubs_test.cc asserted mask_first_n("世界您", 4) leaves CJK
  untouched, while projector_test.cc asserted mask("A的Ççd-123") masks 的 to x.
• The only written statement of intent anywhere in the tree is ARROW-17070's commit message —
  "'Masking' according to Hive specification (a-z : x, A-Z : X, 0-9 : n)" — which the code
  did not implement. docs/ has nothing, and neither the registrations nor the declarations
  carry comments.
• Provenance explains the drift: three commits, two authors, seven months apart, no shared
  spec. ARROW-14482 added the windowed family with the bare-integer switch; ARROW-17070 added
  the show_* wrappers citing Hive but delegating to it; ARROW-17121 added mask() the next
  day and deliberately expanded to Lt/Lo/Nl/No. {1,2,9,10} is exactly the adjacent
  initcap helper's category set minus LT, which is the likely origin of the stray Nl.

Lo is the widest-reaching: CJK, Japanese, Korean, Hebrew, Arabic, Thai, Devanagari. Cased
scripts (Cyrillic, Greek, accented Latin) were already correct and are unaffected.

2. An empty replacement argument means "use the default", not "delete"

Also matching getCharArg. This additionally fixes an under-allocation: the output buffer was
sized max(upper_len, lower_len, num_len) * data_len, which is 0 when all three arguments
are empty even though pass-through characters still get written. SimpleArena::Allocate(0)
returns the arena cursor without advancing it, so those bytes landed in space the next row's
allocation would reuse — silent cross-row corruption in a batch rather than a crash, which is
why the previous test asserted the truncated ":)" and passed.

3. New: mask with otherChar, and native mask_internal

mask(utf8, utf8, utf8, utf8, utf8) -> utf8
    text, upper, lower, digit, other

mask_internal(utf8, utf8, int32, utf8, utf8, utf8, utf8) -> utf8
              text, mode, char_count, upper, lower, digit, other

The 1–4 argument mask overloads now delegate to a shared mask_impl with a null other,
meaning pass-through, so their behavior is unchanged apart from change 1.

mask_internal is a single entry point for all five Hive modes (FULL, FIRST_N, LAST_N,
SHOW_FIRST_N, SHOW_LAST_N) with caller-supplied replacements. It implements the full
getCharArg contract: an empty argument takes the per-class default, an argument parsing to
-1 leaves that class unmasked, and anything longer is truncated to its first character.
Character counts are in codepoints.

This is what lets Dremio vectorize the shapes that actually matter. Apache Ranger's built-in
mask types generate mask_show_last_n(col, 4, 'x', 'x', 'x', -1, '1') and
mask_show_first_n(...) — custom replacement characters, which the (utf8, int32) natives
with hardcoded X/x/n cannot express.

Behavior changes to existing functions

expression	before	after
mask('Dž') — Lt	X	Dž
mask('世'), mask('ㅏ') — Lo	x	世, ㅏ
mask('Ⅷ') — Nl	n	Ⅷ
mask('½'), mask('²') — No	n	½, ²
mask_first_n('Ⅷ', 1) — Nl	n	Ⅷ
mask(s, '', '', '')	deletes matched characters	masks with X/x/n

No change for ASCII, or for any Lu/Ll/Nd character in any script.

Four expectations in projector_test.cc were updated accordingly (TestMaskAll,
TestMaskUpperLower, TestMaskUpper, TestMaskDefault), plus one in
gdv_function_stubs_test.cc for the empty-argument change. Those five are the entire blast
radius — every other mask assertion was already Hive-compatible.

Tests

gdv_function_stubs_test.cc:

• TestMaskInternalModes — all five modes, empty input, unknown-mode error, case sensitivity
• TestMaskInternalCharCountBoundaries — INT32_MIN, negative, 0, at length, past length,
  INT32_MAX, across all four windowed modes
• TestMaskInternalReplacementArguments — custom replacements, the -1 spelling and its
  -01 variant, -2 as an ordinary character, multi-character truncation, empty-means-default,
  and the Ranger MASK_SHOW_LAST_4 shape
• TestMaskInternalUnicode — the category matrix, codepoint-counted windows, multi-byte
  replacements, truncated UTF-8
• TestMaskOtherChar — the new 5-argument overload through both the ASCII fast path and the
  utf8proc path, including mask('王小明', …, '*') => "***"
• TestMaskUnicodeCategories — one case per disputed category in both families, plus a
  cross-family consistency assertion that mask and mask_first_n(…, len) now agree

tests/projector_test.cc:

• TestMaskInternal — end to end in the shape an engine emits: text column plus literal mode,
  char_count and replacements. This is what exercises the generated call, so it is the real
  check on the 15-parameter stub mapping.
• TestMaskOtherChar — the 5-argument overload end to end
• TestMaskNullInput — closes a gap: none of the six existing mask projector tests had a
  single false in its validity vector, so kResultNullIfNull was untested

187 gandiva-internals-test and 249 gandiva-projector-test cases pass.

cmake --build cpp/debug --target gandiva-internals-test gandiva-projector-test -j 8
cpp/debug/debug/gandiva-internals-test --gtest_filter='TestGdvFnStubs.*Mask*'
cpp/debug/debug/gandiva-projector-test --gtest_filter='TestProjector.*Mask*'

Notes for reviewers

Known divergence from the Java implementation. Character counts are in codepoints, while
CharSequenceWrapper counts UTF-16 code units. This is visible only when a char_count
boundary falls inside a surrogate pair — mask_first_n('🌍Abc', 2) gives 🌍Xbc here and
🌍Abc in Java. FULL mode has no char_count and is unaffected. Documented in the stub.

Not covered. mask_internal for int32, int64 and date. Those keep running in Java,
including Ranger's MASK_DATE_SHOW_YEAR. The int path is a handful of integer divisions and
the date path three field assignments, so the vectorization win is much smaller than for
strings; worth adding on evidence rather than speculatively.

Unrelated pre-existing issue this does not address. Because only Lu/Ll/Nd are masked
and the other default is pass-through, mask('王小明') returns the input unchanged. Ranger's
built-in "Redact" policy is plain mask(col), so a redaction policy over uncased-script data
provides no protection. That is Hive-compatible behavior and is tracked separately; changing it
here would have meant diverging from Hive in a masking function, which is worse than a
documented gap.

Dremio-side change required

Only the function-list-gandiva.csv golden, which gains two lines:

mask,GandivaFunctionHolder [functionName=mask  returnType=varchar  parameters=[varchar  varchar  varchar  varchar  varchar]]
mask_internal,GandivaFunctionHolder [functionName=mask_internal  returnType=varchar  parameters=[varchar  varchar  int32  varchar  varchar  varchar  varchar]]

No convertlet change, no new support key and no new Java functions: mask_internal is already
what the convertlet emits and already has a Java implementation, so GandivaPushdownSieve
starts pushing it down on signature match alone, and every degraded path — constant folding, a
Gandiva-less build, exec.disabled.gandiva-functions, oversized CASE — still falls back to
the existing Java mask_internal.

[akravchukdremio]

🟡 Verdict: Minor issues found — 0 must fix, 1 should fix, 2 nits

This PR looks very good overall — the Hive-semantics alignment is well-evidenced, the new mask_internal is carefully bounds-checked, and the test coverage is unusually thorough. There are a couple of items I'd want to check (inline comment below); it is OK to resolve them as-is without code changes if you consider them out of scope or intentional — a short reply is enough.

🟡 Should fix:

• [Correctness] mask_impl output-size computation can overflow int32 for large text × long replacement — same int64 + INT32_MAX guard as in gdv_fn_mask_internal would close it (cpp/src/gandiva/gdv_function_stubs.cc:576-580, see inline)

🔵 Nits (no code change requested):

• The documented codepoint-vs-UTF-16 divergence is understated: it is visible not only at char_count window boundaries. Verified against Dremio's MaskTransformer.java (which iterates UTF-16 code units and classifies surrogate halves as "other"), FULL mode also diverges:

  • mask_internal('😀','FULL',-1,'X','x','n','*') → Java "**" vs native "*" (each surrogate unit takes the otherChar in Java)
  • mask_internal('𐐀','FULL',-1,'X','x','n','-1') (U+10400 is Lu) → Java leaves it unchanged (surrogates = "other" = unmasked), native masks to "X"

  In every divergent case the native path masks at least as much as Java, so there is no exposure risk — suggestion is only to widen the "Known divergence" note (stub comment + PR description say FULL mode is unaffected) and optionally pin the two cases above with one-line tests so the behavior is chosen rather than accidental.

• Enhancement idea for a follow-up: since mode and the replacement args are always literals (the Java side already requires @Param(constant = true)), a MaskInternalHolder mirroring ToDateHolder (+ kNeedsFunctionHolder) would hoist the per-row mode/replacement parsing to plan-build time — removes the per-row parsing overhead on short strings and turns an invalid mode literal into a plan-time error instead of a row-level runtime error.

Observation on pre-existing code (informational only, outside this PR's changes): the utf8proc_iterate calls in the older mask loops (mask_impl's utf8proc loop and gdv_mask_first_n/gdv_mask_last_n) pass the full data_len rather than the remaining length — the over-read your own comment in mask_internal_count_chars describes — and mask_impl's loop has no char_len < 0 check. The new gdv_fn_mask_internal handles both correctly. Could be a small follow-up ticket; no action requested here.

Change walkthrough (5 files)

File	Change
cpp/src/gandiva/function_registry_string.cc	Registers mask_internal (7 args) and 5-arg mask with otherChar
cpp/src/gandiva/gdv_function_stubs.cc	Lu/Ll/Nd-only masking per Hive; shared mask_impl; new gdv_fn_mask_internal; LLVM mappings (verified to match the C signatures)
cpp/src/gandiva/gdv_function_stubs.h	Declarations for the two new exports
cpp/src/gandiva/gdv_function_stubs_test.cc	Six new test suites; empty-argument expectations updated
cpp/src/gandiva/tests/projector_test.cc	Three new end-to-end tests; four expectation updates for the intentional narrowing

---

Generated with AI-assisted analysis

[akravchukdremio]

🟡 SHOULD FIX · Correctness · High confidence

Int32 overflow in the output-size bound

max_repl * data_len is computed in 32-bit signed math and wraps negative for a large text value combined with a long replacement argument (both are user-suppliable SQL expressions). SimpleArena::Allocate then moves its cursor by a huge negative offset and returns a garbage pointer that the masking loops write through — heap corruption instead of a clean error.

Suggest the same guard gdv_fn_mask_internal already uses below: compute the bound in int64_t, compare against std::numeric_limits<int32_t>::max(), and set a context error + return nullptr when exceeded.

OK to resolve as-is if you prefer to track it separately.