fix: normalize models inside collections before they reach the warehouse

[shcheklein]

The same model is stored two different ways depending on the container it sits in.

class Item(dc.DataModel):
    n: int

class InList(dc.DataModel):
    items: list[Item]

class InTuple(dc.DataModel):
    items: tuple[Item, ...]

What lands in the column:

list[Item]        [{"n":1}]        an object
tuple[Item, ...]  ["{\"n\":1}"]    a string holding JSON

Why

A model inside a list is replaced by its dump before it reaches storage. Inside a tuple, a mapping value, or a UDF's returned collection it is not — it arrives live and the warehouse converts it there instead, by different rules.

Two places decide this and neither did it completely:

• _flatten_fields_values chose from the runtime type of the value and knew only list and dict. Its dict branch converted a value that was a model, but not one that was a list of them.
• flatten_value, which UDF outputs go through, converted nothing unless the whole annotation was a model — so a UDF returning list[Item] handed over live models.

Both now share one normalizer that walks by declared type.

What that buys, beyond consistency

in a tuple[Model, ...]	before	after
numpy field	fails to save	works
redacting when_used="json" serializer	persists '***', original lost	stores the real value
Decimal	"1.20"	1.2, same as a list
Path field	works	fails, as in every other shape

The serializer row is the one worth reading twice: a serializer written to hide a value in an API response was deciding what went into the database, and the original was unrecoverable.

Deciding from the annotation

• A type that cannot hold a model is returned untouched, so list[int] is never copied — and this holds per element, so tuple[Item, list[int]] converts the model and leaves the list alone.
• Optional is unwrapped before arguments are read, so list[Item] | None and None | list[Item] behave identically.
• Elements are converted one at a time rather than from whatever the first one is, which tuple[Item, int] needs.
• An erased annotation — Any, object, a bare container — says nothing about its contents, so it is walked rather than assumed model-free.

Compatibility

tuple[Model, ...] columns written before this will not compare equal to ones written after — filter, distinct, subtract, merge. Reads are unaffected: verified by writing such a column on main and reading it here, where both spellings load as the declared tuple.

Relationship to #1943

Independent — neither needs the other. But together they close two of that PR's strict xfails: a colliding mapping inside a live model reaches model_dump(mode="json"), which merges the keys before anything can inspect them. Normalizing first means it arrives as a plain dict and the existing check sees both keys. UDF-returned list[Model] and dict[str, Model] are covered too. Whichever lands second should flip those markers.

[cloudflare-workers-and-pages]

Deploying datachain with    Cloudflare Pages

Latest commit:	0dc3227
Status:	✅  Deploy successful!
Preview URL:	https://7758a407.datachain-2g6.pages.dev
Branch Preview URL:	https://fix-flatten-model-collection.datachain-2g6.pages.dev

View logs

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!