Nulls are not handle correctly
nulls() vs logical_nulls()
nulls() - Returns physical nulls
Returns the raw null buffer as stored in Arrow format
Only shows nulls encoded directly in the array's null buffer
Fast: Direct buffer access, no computation
May not show all "logically" null values
logical_nulls() - Returns computed logical nulls
Computes which values are actually null from all sources
For most types: equivalent to nulls()
Different for special cases:
DictionaryArray with nullable values
RunArray with nullable values
UnionArray where selected values are null
NullArray (all values are logically null, but no physical buffer)
Example:
// DictionaryArray with values ["a", NULL, "b"]
// The dictionary has a null, but the keys might not reference it
// nulls() - might say no nulls (if the null value isn't referenced by any key)
// logical_nulls() - correctly reports which keys actually point to null
// NullArray
// nulls() - returns None (no buffer)
// logical_nulls() - returns a buffer indicating all are null
For your use case:
Nulls are not handle correctly
nulls() vs logical_nulls()
nulls() - Returns physical nulls
Returns the raw null buffer as stored in Arrow format
Only shows nulls encoded directly in the array's null buffer
Fast: Direct buffer access, no computation
May not show all "logically" null values
logical_nulls() - Returns computed logical nulls
Computes which values are actually null from all sources
For most types: equivalent to nulls()
Different for special cases:
DictionaryArray with nullable values
RunArray with nullable values
UnionArray where selected values are null
NullArray (all values are logically null, but no physical buffer)
Example:
// DictionaryArray with values ["a", NULL, "b"]
// The dictionary has a null, but the keys might not reference it
// nulls() - might say no nulls (if the null value isn't referenced by any key)
// logical_nulls() - correctly reports which keys actually point to null
// NullArray
// nulls() - returns None (no buffer)
// logical_nulls() - returns a buffer indicating all are null
For your use case: