feat(codegen): struct/list containers — TASK-29 Phase B (Tasks 1–2 of 4) (#4)

* docs: TASK-29 Phase B plan — codegen struct/list containers

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(codegen): struct/list column passthrough — enable codegen for container output (TASK-29B-1)

Move struct and list column projection from deferred (UnsupportedInCodegen) to committed
surface. The output model + marshalling infrastructure already handles struct/list
Pydantic models. This unlocks struct/list passthrough for codegen, matching native
engine parity. DataFusion oracle validates.

Changes:
- sql_transform/_codegen/engine.py: Remove the guard block rejecting struct/list containers
- tests/test_codegen_coverage.py: Move struct/list passthroughs from _DEFERRED to _COMMITTED
- sql_transform/_codegen/engine_test.py: Update test to verify struct columns now pass through

Test results: 495 passed (+3), 11 skipped (-3)

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>

* fix(codegen): drop unused import, reformat test, correct _to_native docstring (TASK-29B-1 review)

- Remove unused UnsupportedInCodegen import from engine_test.py (F401)
- Fix stray blank lines in engine_test.py via ruff format
- Update _to_native docstring: struct columns now reach projected output via passthrough,
  not only via JOIN keys; removed false claim about UnsupportedInCodegen rejection

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(codegen): struct/list construction (named_struct/struct/array) — TASK-29 Phase B

Adds StructExpr/ListExpr IR: named_struct(...), struct(...)/STRUCT(...), and
array(...)/make_array(...)/[...] now build a Python dict/list at runtime
instead of raising UnsupportedInCodegen. _DEFERRED_FUNCS shrinks to
("unnest",).

Also tightens the Func emit path in engine.py: an unrecognized function name
(notably a transformer call, __tfm_N__) now raises UnsupportedInCodegen
instead of a generic ValueError -- this path was previously unreachable
because named_struct short-circuited before descending into its arguments;
completing named_struct exposed it. Updated plan_test.py's stale
"named_struct defers" assertion accordingly and added a focused unit test for
the new construction IR.

* fix(codegen): widen mixed-numeric list elements + narrow unknown-func defer to __tfm_ (TASK-29B-2 review)

- infer_type's ListExpr arm now unifies element types via _common_base
  (same helper COALESCE uses), so make_array(int, float) types as
  list[float] and pydantic widens the int element at model_validate
  (measured: no emission-side change needed).
- _emit_expr's Func branch narrows the UnsupportedInCodegen defer to
  __tfm_ transformer placeholders only; every other unknown function
  name (e.g. foobar(x), invalid SQL) keeps the original hard ValueError
  instead of being silently skipped by the differential harness.

* refactor(codegen): drop dead struct/array Anonymous branches (TASK-29B review)

struct(...) parses as exp.Struct and array(...) as exp.Array, both handled
by earlier _convert_expr arms; only make_array reaches the Anonymous arm.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(codegen): cover positional/named struct + make_array; fix stale comments (PR#4 review)

Auto-review flagged three committed codegen construction shapes with no value
test: positional struct(a,b) -> c0/c1, named struct(a AS x,...) (exp.PropertyEQ),
and make_array(...). All three match the DataFusion oracle on codegen but native
has no dispatch for them, so they xfail_on_native -- same parity-gap pattern as
the mixed-numeric widening test. Native gaps want their own tickets.

Also fixed two stale comments: plan.py claimed named_struct parses to exp.Struct
(it's exp.Anonymous), and test_transformer_case claimed codegen defers on
named_struct (it defers on the __tfm_0__ transformer call; named_struct is
supported now).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: ahrzb

docs/superpowers/plans/2026-07-22-codegen-deferred-phase-b-containers.md

@@ -94,6 +94,12 @@ def _emit_expr(e: Any, env: dict) -> str:
     if isinstance(e, cp.Func):
         fn = _BUILTINS.get(e.name)
         if fn is None:
+            if e.name.startswith("__tfm_"):
+                # A transformer placeholder call: codegen has no transformer
+                # support (native-only, TASK-34) -- deferred surface, not an error.
+                raise UnsupportedInCodegen(
+                    f"{e.name}() is not supported in codegen yet"
+                )
             raise ValueError(f"Unknown function: {e.name}")
         return f"{fn}({', '.join(_emit_expr(a, env) for a in e.args)})"
     if isinstance(e, cp.Case):
@@ -104,6 +110,11 @@ def _emit_expr(e: Any, env: dict) -> str:
                 f"else {out})"
             )
         return out
+    if isinstance(e, cp.StructExpr):
+        items = ", ".join(f"{name!r}: {_emit_expr(v, env)}" for name, v in e.fields)
+        return f"{{{items}}}"
+    if isinstance(e, cp.ListExpr):
+        return f"[{', '.join(_emit_expr(x, env) for x in e.items)}]"
     raise UnsupportedInCodegen(f"cannot compile {type(e).__name__}")
 
 
@@ -173,9 +184,8 @@ def looked_up(inner: dict, i: int) -> None:
 
 def _to_native(v: Any) -> Any:
     """Recursively unwrap a row's Pydantic struct/list value into plain
-    dicts/lists. Struct columns can't reach a projected output (rejected as
-    UnsupportedInCodegen below), but a JOIN ON key CAN reference one -- and
-    two structurally-identical struct columns from different row tables are
+    dicts/lists for struct/list passthrough or JOIN key normalization.
+    Two structurally-identical struct columns from different row tables are
     two different synthesized Pydantic classes, so BaseModel.__eq__ (which
     checks type identity) would wrongly call them unequal. Plain dicts
     compare structurally, matching DataFusion/rt.val_eq's semantics."""
@@ -238,12 +248,6 @@ def __init__(
         schemas = validation.effective_schemas
 
         inferred = [(alias, cp.infer_type(e, schemas)) for alias, e in plan.projection]
-        for alias, ft in inferred:
-            if cp.is_container(ft.base):
-                raise UnsupportedInCodegen(
-                    f"column '{alias}' is a struct/list, which codegen does not "
-                    "support yet"
-                )
 
         if output_model is None:
             self.output_model = create_model(

sql_transform/_codegen/engine.py

@@ -10,7 +10,7 @@
 import pytest
 from pydantic import BaseModel
 
-from sql_transform._codegen import CodegenFn, UnsupportedInCodegen
+from sql_transform._codegen import CodegenFn
 
 
 class Row(BaseModel):
@@ -115,15 +115,18 @@ def test_int_division_by_zero_raises_value_error():
         fn.infer({"t": [Row(a=1)]})
 
 
-def test_container_columns_are_deferred_not_silently_wrong():
+def test_container_columns_struct_passthrough():
     class Inner(BaseModel):
         x: int
 
     class WithStruct(BaseModel):
         s: Inner
 
-    with pytest.raises(UnsupportedInCodegen):
-        CodegenFn("SELECT s AS out FROM t", {"t": WithStruct}, {})
+    # Struct columns now pass through (no longer deferred)
+    fn = CodegenFn("SELECT s AS out FROM t", {"t": WithStruct}, {})
+    result = fn.infer(t=[WithStruct(s=Inner(x=42))])
+    assert len(result) == 1
+    assert result[0].out.x == 42
 
 
 def test_generated_source_is_available_for_debugging():

sql_transform/_codegen/engine_test.py

@@ -189,6 +189,16 @@ class Cast:
     target: str
 
 
+@dataclass
+class StructExpr:
+    fields: list  # list[tuple[str, expr]] -- key order significant
+
+
+@dataclass
+class ListExpr:
+    items: list  # list[expr]
+
+
 @dataclass
 class Case:
     arms: list  # list[tuple[cond, result]]
@@ -263,7 +273,7 @@ class Plan:
     exp.Abs: "abs",
     exp.Round: "round",
 }
-_DEFERRED_FUNCS = ("named_struct", "struct", "unnest", "make_array")
+_DEFERRED_FUNCS = ("unnest",)
 
 # Measured: sqlglot spreads variadic args differently PER FUNCTION, so each needs
 # its own extraction. Concat  -> this=None, args all in .expressions
@@ -348,10 +358,13 @@ def _convert_expr(e: exp.Expression) -> Any:
             return BinaryOp(op, _convert_expr(e.this), _convert_expr(e.expression))
     if isinstance(e, exp.Cast):
         return Cast(_convert_expr(e.this), _cast_target(e.to.sql()))
-    if isinstance(e, (exp.Struct, exp.Array)):
-        raise UnsupportedInCodegen(
-            "struct/list construction is not supported in codegen yet"
-        )
+    if isinstance(e, exp.Struct):
+        # Authored struct(...) form: positional `struct(a, b)` (bare exprs) or
+        # named `struct(a AS x, ...)` (exp.PropertyEQ). named_struct(...) is a
+        # separate shape -- it parses as exp.Anonymous, handled below.
+        return _convert_struct(e.expressions)
+    if isinstance(e, exp.Array):
+        return ListExpr([_convert_expr(a) for a in e.expressions])
     if isinstance(e, exp.Unnest):
         # Measured: UNNEST(...) parses as its own exp.Unnest class, not
         # exp.Anonymous -- the _DEFERRED_FUNCS name-matching branch below never
@@ -379,12 +392,44 @@ def _convert_expr(e: exp.Expression) -> Any:
             return Func(name, [_convert_expr(a) for a in extract(e)])
     if isinstance(e, exp.Anonymous):
         name = e.name.lower()
+        if name == "named_struct":
+            return _named_struct(e.expressions)
+        # struct(...) parses as exp.Struct and array(...) as exp.Array (handled
+        # above); only make_array reaches the Anonymous arm.
+        if name == "make_array":
+            return ListExpr([_convert_expr(a) for a in e.expressions])
         if name in _DEFERRED_FUNCS:
             raise UnsupportedInCodegen(f"{name}() is not supported in codegen yet")
         return Func(name, [_convert_expr(a) for a in e.expressions])
     raise ValueError(f"Unsupported expression: {e.sql()}")
 
 
+def _named_struct(args: list) -> StructExpr:
+    if len(args) % 2 != 0:
+        raise ValueError("named_struct expects (key, value) pairs")
+    fields = []
+    for i in range(0, len(args), 2):
+        key = args[i]
+        if not (isinstance(key, exp.Literal) and key.is_string):
+            raise ValueError("named_struct field names must be string literals")
+        fields.append((key.this, _convert_expr(args[i + 1])))
+    return StructExpr(fields)
+
+
+def _convert_struct(exprs: list) -> StructExpr:
+    # STRUCT(...) authored form: sqlglot wraps each field as exp.PropertyEQ
+    # (aliased 'name := value') or a bare expr (positional c0, c1, ...).
+    fields = []
+    for i, item in enumerate(exprs):
+        if isinstance(item, exp.PropertyEQ):
+            fields.append((item.this.name, _convert_expr(item.expression)))
+        elif isinstance(item, exp.Alias):
+            fields.append((item.alias, _convert_expr(item.this)))
+        else:
+            fields.append((f"c{i}", _convert_expr(item)))
+    return StructExpr(fields)
+
+
 def _convert_literal(e: exp.Literal) -> Any:
     if e.is_string:
         return e.this
@@ -722,6 +767,12 @@ def _validate_expr(e: Any, resolved, row_schemas, static_schemas, used) -> None:
     elif isinstance(e, Func):
         for a in e.args:
             _validate_expr(a, resolved, row_schemas, static_schemas, used)
+    elif isinstance(e, StructExpr):
+        for _, v in e.fields:
+            _validate_expr(v, resolved, row_schemas, static_schemas, used)
+    elif isinstance(e, ListExpr):
+        for x in e.items:
+            _validate_expr(x, resolved, row_schemas, static_schemas, used)
 
 
 _STR_FUNCS = frozenset({"upper", "lower", "trim", "substr", "substring"})
@@ -763,6 +814,22 @@ def infer_type(e: Any, schemas: dict) -> FieldType:
         nullable = (not e.has_else) or any(t.nullable for t in branch_types)
         base = _common_base(branch_types)
         return FieldType(base, nullable)
+    if isinstance(e, StructExpr):
+        fields = tuple((name, infer_type(v, schemas)) for name, v in e.fields)
+        return FieldType(StructBase(fields), False)
+    if isinstance(e, ListExpr):
+        elem_types = [infer_type(x, schemas) for x in e.items]
+        if not elem_types:
+            elem = FieldType(OTHER, True)
+        else:
+            # Unify element bases the same way COALESCE does (numeric int/float
+            # -> float), so e.g. make_array(int_col, float_col) types as
+            # list<float> -- matching DataFusion, which widens the int element
+            # at runtime (measured: make_array(1, 2.5) -> [1.0, 2.5]).
+            elem = FieldType(
+                _common_base(elem_types), any(t.nullable for t in elem_types)
+            )
+        return FieldType(ListBase(elem), False)
     if isinstance(e, Func):
         return _function_type(e.name, [infer_type(a, schemas) for a in e.args])
     raise UnsupportedInCodegen(f"cannot infer the type of {type(e).__name__}")

sql_transform/_codegen/plan.py

@@ -236,11 +236,19 @@ def test_build_plan_not_and_logic():
     )
 
 
-def test_build_plan_defers_containers():
+def test_build_plan_defers_unnest():
     with pytest.raises(cp.UnsupportedInCodegen):
         cp.build_plan("SELECT unnest(a) AS x FROM t")
-    with pytest.raises(cp.UnsupportedInCodegen):
-        cp.build_plan("SELECT named_struct('k', a) AS x FROM t")
+
+
+def test_build_plan_constructs_struct_and_list():
+    # TASK-29 Phase B: named_struct/array construction, previously deferred.
+    plan = cp.build_plan("SELECT named_struct('k', a) AS x FROM t")
+    assert plan.projection == [("x", cp.StructExpr([("k", cp.Column(None, "a"))]))]
+    plan = cp.build_plan("SELECT array(a, a) AS x FROM t")
+    assert plan.projection == [
+        ("x", cp.ListExpr([cp.Column(None, "a"), cp.Column(None, "a")]))
+    ]
 
 
 def _schemas():

sql_transform/_codegen/plan_test.py

@@ -85,6 +85,16 @@
             "s": static({"k": "int", "v": "float"}, [{"k": 1, "v": 1.0}]),
         },
     ),
+    ("SELECT s AS x FROM t", {"t": rows({"s": "struct{x:int}"}, [{"s": {"x": 1}}])}),
+    ("SELECT l AS x FROM t", {"t": rows({"l": "list[int]"}, [{"l": [1]}])}),
+    (
+        "SELECT named_struct('a', x, 'b', y) AS s FROM t",
+        {"t": rows({"x": "int", "y": "int"}, [{"x": 1, "y": 2}])},
+    ),
+    (
+        "SELECT array(x, y) AS l FROM t",
+        {"t": rows({"x": "int", "y": "int"}, [{"x": 1, "y": 2}])},
+    ),
 ]
 
 
@@ -100,8 +110,6 @@ def test_committed_surface_is_never_deferred(query, tables):
 
 _DEFERRED = [
     ("SELECT unnest(l) AS x FROM t", {"t": rows({"l": "list[int]"}, [{"l": [1]}])}),
-    ("SELECT s AS x FROM t", {"t": rows({"s": "struct{x:int}"}, [{"s": {"x": 1}}])}),
-    ("SELECT l AS x FROM t", {"t": rows({"l": "list[int]"}, [{"l": [1]}])}),
     # A struct in a comparison must defer, not silently mis-evaluate: DataFusion
     # supports it, codegen does not yet, so it must raise (not return a wrong bool).
     (

tests/test_codegen_coverage.py

@@ -29,3 +29,13 @@ def test_int_mod_by_zero_rejected_by_both():
         "SELECT a % b AS r FROM t",
         {"t": rows({"a": "int", "b": "int"}, [{"a": 5, "b": 0}])},
     )
+
+
+def test_unknown_function_rejected_by_both():
+    # foobar() is invalid SQL -- DataFusion errors -- and is NOT a `__tfm_`
+    # transformer placeholder, so codegen must raise a hard error too (not
+    # UnsupportedInCodegen, which the harness would skip as a deferred surface).
+    check_both_raise(
+        "SELECT foobar(x) AS y FROM t",
+        {"t": rows({"x": "int"}, [{"x": 1}])},
+    )

tests/test_diff_errors.py

@@ -21,6 +21,72 @@ def test_list_construct():
     )
 
 
+# The three construction shapes below (positional struct, named struct(...),
+# make_array) are supported on codegen and match the DataFusion oracle, but
+# native has no dispatch for them at all -- so they xfail_on_native, the same
+# parity-gap pattern as test_list_construct_mixed_numeric_widens. Codegen
+# correctness is the assertion; the native gaps want their own tickets.
+def test_struct_construct_positional(xfail_on_native):
+    # struct(a, b) names fields positionally c0, c1 (matches DataFusion).
+    xfail_on_native("native has no struct(...) construction dispatch (expr_build.rs)")
+    check(
+        "SELECT struct(a, b) AS s FROM t",
+        {"t": rows({"a": "int", "b": "int"}, [{"a": 1, "b": 2}])},
+        expect=[{"s": {"c0": 1, "c1": 2}}],
+    )
+
+
+def test_struct_construct_named(xfail_on_native):
+    # struct(a AS x, b AS y) parses as exp.PropertyEQ -> explicit field names.
+    xfail_on_native("native has no struct(...) construction dispatch (expr_build.rs)")
+    check(
+        "SELECT struct(a AS x, b AS y) AS s FROM t",
+        {"t": rows({"a": "int", "b": "int"}, [{"a": 1, "b": 2}])},
+        expect=[{"s": {"x": 1, "y": 2}}],
+    )
+
+
+def test_make_array_construct(xfail_on_native):
+    # make_array(a, b) is a DataFusion builtin; native doesn't recognize it as a
+    # function (only the bracket literal [a, b] reaches native's list path).
+    xfail_on_native("native has no make_array function dispatch (expr_build.rs)")
+    check(
+        "SELECT make_array(a, b) AS l FROM t",
+        {"t": rows({"a": "int", "b": "int"}, [{"a": 1, "b": 2}])},
+        expect=[{"l": [1, 2]}],
+    )
+
+
+def test_list_construct_mixed_numeric_widens(xfail_on_native):
+    # DataFusion widens the int element to match the float element (list<double>).
+    # Locks in infer_type's ListExpr arm unifying element bases via _common_base
+    # (like COALESCE), so the output model types the list as list[float] and
+    # pydantic coerces the runtime int element to float.
+    #
+    # Bracket literal `[x, y]` (not make_array(...)): native's function dispatch
+    # doesn't recognize "make_array"/"array" as a builtin at all (a separate,
+    # pre-existing gap -- expr_build.rs's convert_function has no such case,
+    # unlike Python's ANONYMOUS-function branch), so it isn't a usable surface
+    # to compare both engines against the oracle. The bracket form reaches the
+    # same Expr::List / ListExpr construction path in both engines.
+    #
+    # xfail_on_native: measured -- native's unify_list_element_types
+    # (src/types.rs) is exact-equality-only, the same bug class just fixed here
+    # in codegen's infer_type. Native still emits [1, 2.5] (un-widened) instead
+    # of DataFusion's [1.0, 2.5]. A native-side fix is out of scope here; flag
+    # as a parity bug for its own ticket rather than fixing inline.
+    xfail_on_native(
+        "native does not widen mixed int/float list elements "
+        "(types.rs unify_list_element_types is exact-equality-only) -- "
+        "separate parity bug, own ticket"
+    )
+    check(
+        "SELECT [x, y] AS l FROM t",
+        {"t": rows({"x": "int", "y": "float"}, [{"x": 1, "y": 2.5}])},
+        expect=[{"l": [1.0, 2.5]}],
+    )
+
+
 def test_struct_input_roundtrip():
     check(
         "SELECT s FROM t",

tests/test_diff_types.py

@@ -57,8 +57,9 @@ def test_transformer_in_nonfirst_case_branch_parity_native():
 
 
 def test_transformer_in_case_defers_on_codegen():
-    # Codegen has no transformer support and already defers named_struct; the
-    # construct must defer loudly (UnsupportedInCodegen), never silently mishandle.
+    # Codegen has no transformer support, so the __tfm_0__(...) call defers
+    # (named_struct itself is supported now); the construct must defer loudly
+    # (UnsupportedInCodegen), never silently mishandle.
     model = synthesize_this_model(pa.schema([("g", pa.int64()), ("age", pa.float64())]))
     with pytest.raises(UnsupportedInCodegen):
         CodegenFn(_SQL, row_tables={"__THIS__": model}, static_tables={})