fix: keep None in an array wherever it sits

[shcheklein]

list[int | None] fails to save when the first element is None:

class Scores(dc.DataModel):
    vals: list[int | None]

chain.map(s=lambda: Scores(vals=[1, None]), output=Scores)   # ok
chain.map(s=lambda: Scores(vals=[None, 2]), output=Scores)   # UdfError

convert_type decided how to handle a whole array by looking at val[0]. A
leading 1 matched the item type, so the array was stored untouched and the
None rode along with it. A leading None did not match, so every element was
sent to a converter — and None has nothing to convert, so it raised.

Each element now answers for itself, and a None survives only where
item_type.dc_nullable says one belongs.

What that changes

Read back after a save, main against this PR:

list[int | None]	main	here
[1, None]	[1, None]	[1, None]
[None, 2]	UdfError	[None, 2]
[None, None]	UdfError	[None, None]

list[float | None]	main	here
[1.0, None]	[1.0, nan]	[1.0, None]
[None, 2.0]	UdfError	[None, 2.0]

nan in that first row is the read side of the same bug: TypeReadConverter.float
maps None to nan, which is right for a float column and wrong for a nullable
element. Scalar Optional[float] already returns None, so arrays were the
outlier.

A model nested inside a dict item was converted in one order and not the other:

list[dict[str, Item] | None]	main	here
[{"a": Item(n=1)}, None]	TypeError: Object of type Item is not JSON serializable	[{'a': Item(n=1)}, None]
[None, {"a": Item(n=1)}]	[None, {'a': Item(n=1)}]	[None, {'a': Item(n=1)}]

And a None reached a non-nullable column unchecked, again depending on
where it sat:

dc.read_values(x=[[1, None]], output={"x": list[int]}).to_values("x")
# main: [[1, None]]   -- stored into Array(Int64)
# here: UdfError

dc.read_values(x=[[None, 2]], output={"x": list[int]}).to_values("x")
# main: UdfError
# here: UdfError

On disk

Only arrays that contain a None change, and only the order that was already
inconsistent with itself:

list[dict | None]	before	after
[{"k": 1}, None]	[{"k":1},null]	unchanged
[None, {"k": 1}]	["null","{\"k\":1}"]	[null,{"k":1}]

An array without a None takes exactly the path it took before, so nothing else
moves. Checked by writing twelve annotations with both versions and diffing the
stored column — list[int], list[str], list[dict], list[Model],
list[list[int]], tuple[int, ...], tuple[str, ...], tuple[Model, Model],
tuple[int, int] and the rest are byte-identical, and so are mixed arrays like
({"k": 1}, 2).

Earlier revisions of this PR did not hold that line: one moved
tuple[int, ...] from ["1","2"] to [1,2], another moved ({"k": 1}, 2) from
[{"k":1},2] to [{"k":1},"2"]. Both made merge on such a column return
nothing across old and new datasets, since the two hydrate the same but compare
as different in SQL.

Still unfixed here, exactly as on main: a model beside a plain value in an
array with no None, ({"a": Item(n=2)}, 1), is left raw and fails to write.
Same defect as the row above reached without a None; it belongs with the rest
of the mixed-array behaviour in #1968.

Converting every element costs about 10× more per array on a microbenchmark and
nothing measurable on a write: 4000 rows of 512-dim embeddings ran 2.83 s
against 2.81 s, JSON encoding and IO dominating. Arrays are also walked once to
find out whether a None is present, which the old code never did — that is
what keeps a stray None out of a non-nullable column.

Left for #1968

All of these are how main behaves today and are untouched here, because fixing
them changes the stored shape of arrays that already round-trip — which breaks
comparison between datasets written either side of the change. They are written
up under An array's stored shape is decided by one of its
elements:

• A model beside a plain value is left raw. ({"a": Item(n=2)}, 1) raises
  TypeError: Object of type Item is not JSON serializable; reverse the order and
  it writes. Same defect as the None case fixed here, reached without a None.
• Element order picks the layout of a mixed array. ({"k": 1}, 2) stores
  [{'k': 1}, 2], (2, {"k": 1}) stores ['2', '{"k":1}'].
• A tuple stores unlike the list of the same values. list[int] stores
  [1,2]; tuple[int, ...] stores ["1","2"]. This is what
  #1963 runs into.
• Non-finite floats become null in loosely typed fields — dict or Any,
  not list[float].
• A nested list cannot hold a None at the outer level.
  list[list[int] | None] writes [[1, 2], [3]] but not [None, [1, 2]]: only a
  scalar item is made nullable, and an array has nowhere to keep the null. Since
  list[Model | None] resolves now, list[list[Model | None] | None] reaches the
  same wall, where it used to stop earlier for want of an inner type. Rejecting
  the annotation outright would take the working all-present case with it.
• A root model element reads back as its bare value. list[RootModel[int]]
  returns [1, 2], not [R(1), R(2)], because the reader rebuilds a model only
  from a mapping. The Model | None spelling is refused here rather than stored
  that way.

First of the steps found while reviewing #1963 — that PR's order-dependent
nullable failure is this same bug, reached through a different annotation.

[cloudflare-workers-and-pages]

Deploying datachain with    Cloudflare Pages

Latest commit:	1d9b362
Status:	✅  Deploy successful!
Preview URL:	https://5be407d5.datachain-2g6.pages.dev
Branch Preview URL:	https://fix-array-conversion-element.datachain-2g6.pages.dev

View logs

[codecov]

Codecov Report

❌ Patch coverage is 91.57895% with 8 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/datachain/lib/convert/python_to_sql.py	91.42%	3 Missing and 3 partials ⚠️
src/datachain/lib/convert/flatten.py	71.42%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!