Validation accepts annotations that storage cannot honour

[shcheklein]

Umbrella for the validation/storage contract. Consolidates #1917, #1920, #1921, #1923, #1927, #1933, #1934 and #1935.

is_chain_type, or the model-field path that bypasses it, accepts an annotation. The storage layer then cannot honour it, and the failure surfaces somewhere else — at write, at read, or only after a reload in a fresh process. Nothing checks that validation and storage agree about what is supported.

Use cases

Accepted, then the write fails

• bytes inside a concrete list/dict — element, key or value, including nested (was is_chain_type accepts bytes inside concrete list/dict trees, but writing them fails #1921)
• unparameterized sequences: list, typing.List, Sequence — TypeError/IndexError from the SQL layer, via output= and via model fields, which never consult is_chain_type (was Unparameterized sequence annotations pass validation but fail in the SQL layer #1923)
• list[Enum] — likely fixed by Map enum signals by member value type instead of builtin str #1929; recheck when it lands (was list[Enum] model fields pass validation but fail to write #1934)
• Iterable[T] model fields — pydantic leaves a ValidatorIterator in the field, which the write refuses. Generic to T, not File-specific (was Iterable model fields fail to store; Iterable[File] setup values also miss catalog attachment #1920)
• dict[str, bytes] and other bytes-in-container shapes

Accepted, then the read is wrong or impossible

• mapping keys that decode to the wrong type are kept silently; unhashable ones crash (was Mapping keys that decode to the wrong type are kept silently; unhashable ones crash #1927)
• Literal keys with non-string members cannot be read back through the whole-model route at all (was Literal keys with non-string members cannot be read back through the whole-model route #1933)
• Iterable[File] setup values get no catalog, so the files are unreadable (was Iterable model fields fail to store; Iterable[File] setup values also miss catalog attachment #1920)

Accepted, then it vanishes

• inferred and name-only UDF outputs skip validation entirely, so an unsupported type is accepted and then disappears from a loaded schema (was Inferred and name-only UDF outputs skip validation and can vanish from loaded schemas #1917)
• Literal, enum and parameterised-alias specialisations are dropped from the persisted schema, so a clean-process reload degrades them to Any (was Literal, enum and parameterised-alias annotations do not survive a clean-process schema reload #1935)

Accepted nowhere, though storage could manage it

• is_chain_type rejects an enum at the root, so output={"k": Species} and a top-level dict[Species, int] raise UdfSignatureError — while the same enum works as a model field. The same disagreement pointing the other way.

This list is very likely incomplete

That is the main reason for one ticket rather than eight.

Every instance was found one at a time, over four review rounds, and in three of those the newly-found case was one a previous fix had introduced. Each was discovered because somebody named a specific spelling — Key[str], Literal[1] | bool, type A0[T] = A1[T], a model defining __value__. None came from a systematic sweep, because there isn't one.

The surface is combinatorial: Annotated, NewType, PEP-695 aliases (bare, subscripted, defaulted, recursive), Literal (string, numeric, enum-valued, mixed), enums (str-mixin, int-mixin, plain, mixed-value), unions and Optional around any of those, nested in list/dict/tuple/a model, in key or value position, across four read routes and the write path.

Assume more instances exist. A change that ticks only the boxes above should be treated as incomplete.

What would actually close this

Not another round of point fixes:

• one authority on "can this annotation be stored", consulted by is_chain_type and by model-field validation, so the two cannot drift;
• a round-trip property test over generated annotations — declare, write, read by every route, reload in a fresh process, compare. This is the only thing that finds the cases nobody has thought to name, and it is the part that matters most;
• for anything storage cannot honour, rejection at schema-build time naming the field, instead of a TypeError from the SQL layer or a silent degradation to Any.

A ~12-line detector was prototyped that catches every exotic annotation above (28 cases, no false positives on anything we want to keep), if loud rejection is the preferred direction.

Related

• Read path: values do not come back as their declared type #1918 — the read path returning the wrong type. Different fix location.
• dict signals with mixed str/non-str key types silently lose data on save #1914 — write-time data loss on ordinary mixed keys.

🤖 AI-assisted (Claude Code) — verify before relying on it.