Skip to content

docs: guidance on typing untyped nulls and empty collections - #1140

Closed
nielspardon wants to merge 2 commits into
substrait-io:mainfrom
nielspardon:docs/untyped-null-typing-guidance
Closed

docs: guidance on typing untyped nulls and empty collections#1140
nielspardon wants to merge 2 commits into
substrait-io:mainfrom
nielspardon:docs/untyped-null-typing-guidance

Conversation

@nielspardon

@nielspardon nielspardon commented Jul 22, 2026

Copy link
Copy Markdown
Member

Motivation

Substrait deliberately has no dedicated null or "bottom" type: a null is a value of a nullable concrete type, and an empty list/map carries a concrete element type. The type system docs already state this ("null is considered to be a special value of a nullable type, rather than the only value of a special null type") and that the type system is strict with no implicit coercion.

What the docs don't yet say is how a producer should satisfy that rule when its source models untyped nulls or empty collections with a dedicated null/unknown type — e.g. Spark NullType, ClickHouse Nothing, Apache Arrow Null. This is a recurring question, and real consumers have hit it: DataFusion's Substrait producer, for example, added typed-null handling (folding an explicit cast into a typed null, and mapping a bare Arrow Null to a concrete type) rather than introducing a bottom type — see apache/datafusion#15854 and apache/datafusion#18414.

This PR adds a short, non-normative note documenting the intended handling, without changing the type system:

  • resolve the type from surrounding context where possible;
  • otherwise assign a widely-supported concrete type (e.g. i32) for portability;
  • it is guidance, not a requirement — any concrete type is valid.

It also explicitly distinguishes this case from the unbound type added in #1081 (which marks a type that is not yet known in a partially bound plan), since the two concepts now sit side by side in the docs and are easy to conflate.

Context

This came out of the Apache Gluten ↔ Substrait fork-consolidation effort. Gluten carries a forked Type.Nothing message to represent Spark's NullType / ClickHouse's Nothing (Gluten PR apache/gluten#791). Rather than upstream a bottom type — which would conflict with Substrait's explicitly-typed-null design and strict no-coercion rule — the resolution is for producers to emit typed nulls / typed empty collections, which this guidance documents. See Gluten's SubstraitModifications.md.

Changes

  • site/docs/types/type_system.md: add a non-normative note, "Untyped nulls and empty collections."

Docs-only; no proto or spec-behavior change.

🤖 Generated with AI


This change is Reviewable

Substrait has no null/bottom type: nulls are typed and empty collections
carry a concrete element type. Add a non-normative note telling producers
to resolve an untyped source value (a bare NULL, or an empty array/map
from a source that models these with a null/unknown type) to a concrete
type -- from context where possible, otherwise a widely-supported type
such as i32 for portability. Distinguishes this from the unbound type.
@nielspardon nielspardon added the apache-gluten Upstreaming features from the Apache Gluten Substrait fork label Jul 22, 2026
@nielspardon nielspardon added the apache-gluten Upstreaming features from the Apache Gluten Substrait fork label Jul 22, 2026

@bvolpato bvolpato left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Non-blocking wording suggestion left inline.

Comment thread site/docs/types/type_system.md Outdated
…ections

Address non-blocking review feedback: use the nullable form i32? for a
bare NULL, and spell out collection type parameters (LIST<i32> for an
empty list, MAP<i32, i32> for an empty map; a map carries key and value
types, not a single element type).
@nielspardon

Copy link
Copy Markdown
Member Author

Superseded by #1147.

Per the community sync discussion, the agreed direction is to introduce a dedicated null type rather than document a producer convention for picking a concrete type, so this guidance no longer reflects the intended handling. The useful half of it — resolve the type from surrounding context where possible, and reserve the untyped form for values that genuinely carry no type — is carried over into the producer guidance in #1147.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

apache-gluten Upstreaming features from the Apache Gluten Substrait fork

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants