Skip to content

Commit e154cdf

Browse files
authored
Fix masked array deserialization overflow for integer dtypes
NumPy masked arrays default to a fill value of 999999, which cannot be represented by small integer dtypes such as uint8 or uint16. During distributed deserialization this raises a TypeError when reconstructing the masked array and may cause tasks to hang indefinitely. Ensure that fill values are cast safely to the target dtype, falling back to NumPy's default fill value when necessary.
1 parent 4fb4814 commit e154cdf

File tree

1 file changed

+8
-0
lines changed

1 file changed

+8
-0
lines changed

distributed/protocol/numpy.py

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -216,4 +216,12 @@ def deserialize_numpy_maskedarray(header, frames):
216216
if pickled_fv:
217217
fill_value = pickle.loads(fill_value)
218218

219+
# Ensure fill_value is compatible with dtype
220+
if fill_value is not None:
221+
try:
222+
fill_value = np.array(fill_value, dtype=data.dtype).item()
223+
except (OverflowError, ValueError, TypeError):
224+
fill_value = np.ma.default_fill_value(data.dtype)
225+
219226
return np.ma.masked_array(data, mask=mask, fill_value=fill_value)
227+

0 commit comments

Comments
 (0)