A model and a plain value serialize by different rules

[shcheklein]

Whether a value can be stored, and what it turns into, depends on whether a
Pydantic model happens to wrap it. Two converters take signals to storage and
they do not agree:

• lib/convert/flatten.py dumps in Pydantic's python mode and hands the
  result to datachain.json, this project's encoder.
• data_storage/warehouse.py::_to_jsonable dumps in Pydantic's JSON mode,
  where Pydantic settles every value itself.

Which one runs is decided by the shape holding the model, not by anything the
user chose. Instrumenting the Pydantic branch of _to_jsonable during real
writes:

field	converter
list[Model]	flatten
dict[str, Model]	flatten
plain model signal	flatten
tuple[Model, Model]	_to_jsonable

So a fixed-length tuple of models serializes by different rules than a list of
the same models.

What actually differs

The encoder refuses types JSON mode writes. With the type declared on the
field, as models do:

declared type	JSON mode	datachain.json
timedelta	"P3D"	TypeError
PurePosixPath / Path	"/a/b"	TypeError
IPv4Address	"1.2.3.4"	TypeError
plain enum	"red"	TypeError
set[int]	[1, 2]	TypeError
bytes	"ab"	TypeError
Decimal("1.20")	"1.20"	1.2

JSON mode rewrites non-finite floats to null, in loosely typed fields.
A field declared list[float] keeps them; one declared dict or Any does not,
because Pydantic serializes its contents as arbitrary JSON and applies
ser_json_inf_nan there. So the same model loses data in one container and not
the other:

class Loose(BaseModel):
    name: str
    meta: dict

# list[Loose]        -> meta={'s': [nan, 1.5]}    flatten, preserved
# tuple[Loose, Loose] -> meta={'s': [None, 1.5]}  JSON mode, nan gone

There is no per-call escape — neither model_dump nor SchemaSerializer accepts
inf_nan_mode, and ser_json_inf_nan is model config, which belongs to the
caller. Note this is a property of the JSON string step, not of
model_dump(mode="json"), which returns the floats untouched.

Numpy scalar types that subclass a builtin never reach the encoder.
Pydantic serializes them directly, so np.bytes_(b"ab") is stored as "ab"
inside a model while the encoder refuses bytes outside one. np.complex128
is worse: written as "1+2j" on Pydantic 2.13 and refused on 2.11, so the
behaviour varies across the supported range.

JSON mode merges colliding dict keys before anything can see them:

model holds : {1: 'first', '1': 'second'}   2 entries
python mode : {1: 'first', '1': 'second'}   2 entries
json mode   : {'1': 'second'}               1 entry

An array's stored shape is decided by one of its elements

The same split shows up a level down, in warehouse.py::convert_type. An
Array(JSON) column asks a single element whether the array is already in
storage form, and that one answer picks the physical layout for all of it. So
element order decides what gets written:

({"k": 1}, 2)   ->  [{'k': 1}, 2]          objects, left as they are
(2, {"k": 1})   ->  ['2', '{"k":1}']       each element encoded as a string

Consequences visible on main today:

• A model beside a plain value is never converted.
  ({"a": Item(n=2)}, 1) raises TypeError: Object of type Item is not JSON serializable, because the leading dict makes the array look converted and the
  nested model is left raw. Reverse the order and it writes.
• A tuple stores unlike the list of the same values. list[int] is
  Array(Int64) and stores [1,2]; tuple[int, ...] is Array(JSON) and
  stores ["1","2"]. This is what fix: keep the element type of a variadic tuple #1963 runs into: correcting the element type
  reinterprets bytes already written that way.

#1970 fixes the part of this that involves a None and deliberately leaves the
rest, since changing the layout of arrays that already round-trip breaks
comparison between datasets written either side of the change — the same
migration problem as the type table above.

Why this is one issue and not several

Every item above is the same fact — a model is converted by different rules than
a plain value — and they end together, when the warehouse converter stops using
JSON mode and both paths share one conversion. Fixing them piecemeal has already
failed repeatedly: each PR that moved a shape from one converter to the other
broke whatever only the other converter knew (#1960 paths and serializers, #1964
numpy, #1965 paths and Decimal).

What it blocks

• fix: refuse a dict whose keys collide as JSON instead of losing a value #1943 cannot drop its two Pydantic xfails while a live model's colliding keys
  are merged inside model_dump before the check runs.
• feat: encode the remaining types Pydantic's JSON mode knows #1966 was closed rather than merged; the encoder gaps in the first table are
  its content and are needed before either path can be made the only one.

Not blocked by this

#1967 makes numpy writable in the warehouse path and refuses what it cannot
store faithfully, rather than storing something else. It does not close anything
above.