Describe the bug
Combining DictionaryArrays whose dictionaries were built independently requires
merging their values. merge_dictionary_values deduplicates, so the merged
dictionary holds only the distinct referenced values; the MutableArrayData
fallback concatenates them and can exceed what the key type addresses even when
the distinct values fit comfortably.
Since #10675 this surfaces as a DictionaryKeyOverflowError rather than a panic,
which is the right behaviour for a genuine overflow. But in the two cases below
the distinct values do fit, and the error is avoidable.
1. View-typed dictionary values are never merged.
should_merge_dictionary_values matches primitives and offset-based byte arrays.
Utf8View/BinaryView are neither, so it returns early with
should_merge = false and such dictionaries always take the non-deduplicating
path. Were it to reach the merge, get_masked_values would hit
unimplemented!() — it has no arm for the view layouts either.
The two are indistinguishable to a caller: identical data merges as Utf8 and
fails as Utf8View.
2. The interner alone can overflow the key type.
Interner is best-effort by design — a hash collision evicts the previous
occupant, so one value may be handed several keys. In practice this leaves ~42%
duplicates, enough to overflow a UInt16 key at realistic cardinalities. This
affects Utf8 dictionaries too; it is simply less visible there.
To Reproduce
// (1) identical data, only the value layout differs
let utf8 = || {
let v: StringArray = (0..200).map(|i| Some(format!("v{i}"))).collect();
DictionaryArray::<UInt8Type>::new(UInt8Array::from_iter_values(0..200), Arc::new(v))
};
let view = || {
let v: StringViewArray = (0..200).map(|i| Some(format!("v{i}"))).collect();
DictionaryArray::<UInt8Type>::new(UInt8Array::from_iter_values(0..200), Arc::new(v))
};
concat(&[&utf8(), &utf8()]); // Ok(400)
concat(&[&view(), &view()]); // Err("Dictionary key bigger than the key type")
// (2) 4 dictionaries over the same distinct values, UInt16 keys, Utf8 values
// (so the merge path is actually taken)
4 x 20000 distinct -> Ok, merged dict = 28445 (+42%)
4 x 40000 distinct -> Ok, merged dict = 57433 (+43%)
4 x 60000 distinct -> Err(DictionaryKeyOverflowError)
Expected behavior
Both should succeed with a merged dictionary holding one key per distinct value:
200 in the first case, 60000 in the second. DictionaryKeyOverflowError should
be reserved for a genuine overflow — more distinct values than the key type can
address.
Additional context
Encountered in a column store that declares dictionary-encoded columns as
Dictionary(UInt16, Utf8View). Every data source builds its own dictionary, so
any query combining batches hits this: a sort-preserving merge, a hash join's
build side, or plain CoalesceBatchesExec — the last of which means it
reproduces with a single partition.
Follow-up to #10674 / #10675, which made this an error instead of a panic.
I have a fix and can open a PR.
Describe the bug
Combining
DictionaryArrays whose dictionaries were built independently requiresmerging their values.
merge_dictionary_valuesdeduplicates, so the mergeddictionary holds only the distinct referenced values; the
MutableArrayDatafallback concatenates them and can exceed what the key type addresses even when
the distinct values fit comfortably.
Since #10675 this surfaces as a
DictionaryKeyOverflowErrorrather than a panic,which is the right behaviour for a genuine overflow. But in the two cases below
the distinct values do fit, and the error is avoidable.
1. View-typed dictionary values are never merged.
should_merge_dictionary_valuesmatches primitives and offset-based byte arrays.Utf8View/BinaryVieware neither, so it returns early withshould_merge = falseand such dictionaries always take the non-deduplicatingpath. Were it to reach the merge,
get_masked_valueswould hitunimplemented!()— it has no arm for the view layouts either.The two are indistinguishable to a caller: identical data merges as
Utf8andfails as
Utf8View.2. The interner alone can overflow the key type.
Interneris best-effort by design — a hash collision evicts the previousoccupant, so one value may be handed several keys. In practice this leaves ~42%
duplicates, enough to overflow a
UInt16key at realistic cardinalities. Thisaffects
Utf8dictionaries too; it is simply less visible there.To Reproduce
Expected behavior
Both should succeed with a merged dictionary holding one key per distinct value:
200 in the first case, 60000 in the second.
DictionaryKeyOverflowErrorshouldbe reserved for a genuine overflow — more distinct values than the key type can
address.
Additional context
Encountered in a column store that declares dictionary-encoded columns as
Dictionary(UInt16, Utf8View). Every data source builds its own dictionary, soany query combining batches hits this: a sort-preserving merge, a hash join's
build side, or plain
CoalesceBatchesExec— the last of which means itreproduces with a single partition.
Follow-up to #10674 / #10675, which made this an error instead of a panic.
I have a fix and can open a PR.