Introducing the VARIANT Type in Velox

[baibaichen]

1. Motivation

The VARIANT type provides an efficient binary representation for dynamic semi-structured data. Engines such as Databricks, Snowflake, Spark, DuckDB, and StarRocks, together with table formats such as Apache Iceberg and Apache Paimon, already support Variant. It has become the de facto semi-structured type in lakehouse systems.

Parquet Variant and its Shredding encoding preserve type information, typed_value, and column statistics. This behavior differs fundamentally from the traditional JSON-as-VARCHAR full-text parsing path. From the perspectives of both performance and structural consistency, Velox needs a dedicated VARIANT type to represent this upstream physical format.

2. Technical Background and Upstream Ecosystem

2.1 Parquet Variant Format and Shredding

Parquet Variant is not a simple wrapper around JSON text. It is a well-defined binary format. A Variant value consists of two parts: metadata and value. The former stores the field-name dictionary. The latter stores the recursive encoding of types and values. With this format, the reader can resolve child fields by path directly, without falling back to text parsing.

In addition, Parquet defines the Shredding specification. Shredding splits frequently accessed fields out of the Variant blob into independent strongly typed columns. This changes the physical representation from a single value blob to a composite layout of metadata + remainder value + typed columns. That change directly affects the structural design of VARIANT columns in Velox.

2.2 How Spark Uses It

Spark 4.1.x uses two execution paths for Variant.

The first is the optimized path. When a query uses only path-based access such as variant_get, Spark's optimization rule PushVariantIntoScan rewrites those accesses into projections of specific child fields and pushes them down into the Parquet reader. The reader then decodes the Variant value and returns it as a Struct. This effectively eliminates the Variant type at scan time, so the rest of execution uses only structured columns.

The second is the full-return path. Some functions, such as to_json, semantically require the complete Variant value. In that case, even when the Variant field has already been split into shredded columns, Spark must still re-encode typed_columns together with the remainder value to reconstruct the metadata and value of the Variant.

For vectorized engines such as Gluten + Velox, directly adopting Spark's strategy would require the full-return path to re-encode shredded columns. That introduces significant overhead. Therefore, the Velox design must also preserve efficient full-Variant return.

2.3 Velox's Existing Type Extension Mechanisms

Velox's type system includes physical types and the logical, custom, and complex types built on top of them. A physical type is a fixed built-in scalar type such as BOOLEAN, INTEGER, BIGINT, VARCHAR, or VARBINARY. It determines the in-memory layout. Each physical type corresponds to one C++ type used as the template parameter of FlatVector<T>. Complex types include the three built-in composite types ARRAY, MAP, and ROW. Velox provides three extension paths:

Path	Mechanism	Examples
Logical Type	Adds semantics on top of an existing Physical Type	DATE (INTEGER), DECIMAL (BIGINT/HUGEINT)
Custom Type	Reuses the storage of an existing Physical Type	JSON (VARCHAR), GEOMETRY (VARBINARY), IPPREFIX (ROW(HUGEINT, TINYINT))
OPAQUE	Used when no suitable Physical Type exists	std::shared_ptr<void>

Custom Type also supports custom comparison and hash semantics through providesCustomComparison, but only for primitive fixed-width physical types.

3. VARIANT Type Design

None of the three extension mechanisms described above can satisfy the requirements of VARIANT:

1. Logical Type and VARBINARY-based Custom Type can store only a single blob. They cannot represent the multi-column structure of metadata + value + shreddedColumns, which would effectively forfeit Shredding.

2. ROW-based Custom Type, such as IPPREFIX, requires the number of children to be fixed at construction time. In addition, RowVector::copy() assumes that children are aligned by index. In contrast, shredded columns in VARIANT are dynamic at runtime.

3. OPAQUE stores values row by row as std::shared_ptr<void>. It therefore cannot be vectorized.

Therefore, Velox must introduce VARIANT as a built-in type. The core of the design is the new TypeKind::VARIANT enum together with the corresponding VariantType and VariantVector implementations.

Note

If VARIANT did not require Shredding, and consisted only of the two fixed VARBINARY columns metadata and value, then modeling it as ROW(VARBINARY, VARBINARY) through a Custom Type would be sufficient, much like IPPREFIX. However, Shredding is the key performance feature of the Parquet Variant specification. Dropping it would forfeit an order-of-magnitude performance improvement.

3.1 Core Definitions

VARIANT is added as a first-class member of the Velox type system.

// ===== Type.h =====
/// Simple enum with type category.
enum class TypeKind : int8_t {
  VARIANT = 36,
  INVALID = 37
};

The corresponding type traits, type class, and vector class are defined as follows.

// ===== TypeTraits =====
template <>
struct TypeTraits<TypeKind::VARIANT> {
  static constexpr TypeKind typeKind = TypeKind::VARIANT;
  static constexpr const char* name = "VARIANT";
  // ... other traits
};

// ===== Type (velox/type/Type.h) =====
class VariantType : public TypeBase<TypeKind::VARIANT> {
  public:
    static std::shared_ptr<const VariantType> create() {
      static const auto instance = std::make_shared<const VariantType>();
      return instance;
    }

    /// Zero-schema type: no children, no static structure.
    uint32_t size() const override { return 0; }
    const TypePtr& childAt(uint32_t) const override {
      VELOX_FAIL("VariantType has no children");
    }

    bool isOrderable() const override { return false; }
    bool isComparable() const override { return false; }
};
inline VariantTypePtr VARIANT() { return VariantType::create(); }

// ===== Vector (velox/vector/ComplexVector.h) =====
class VariantVector : public BaseVector {
  // Core data members:
  VectorPtr metadata_;    // VARBINARY, metadata
  VectorPtr value_;       // VARBINARY, value (or remainder value)
  // Optional Shredding acceleration columns:
  std::vector<std::string> shreddedPaths_;
  std::vector<TypePtr> shreddedTypes_;
  std::vector<VectorPtr> shreddedColumns_;
};

The corresponding Encoding is:

// ===== VectorEncoding.h =====
enum class Simple {
  // ... other types
  VARIANT, // new
};

Type constraints: both isComparable() and isOrderable() return false. A VARIANT value cannot participate in ORDER BY, GROUP BY, join keys, DISTINCT, INTERSECT, EXCEPT, or REPARTITION BY. This is not a temporary MVP compromise. It follows naturally from the absence of general comparison semantics for semi-structured types. Spark 4.1 also disallows VARIANT in all of these operations through SPARK-47569, SPARK-48224, and SPARK-47822.

Zero-schema principle: VariantType does not expose childAt() or names(). It is fundamentally different from ROW<a:int, b:string>. For semi-structured data, hierarchy and field names exist only inside the metadata and value stored in column data, following a schema-on-read model. Path visibility, strongly typed fast paths, and shredded schema all belong to the runtime state of VariantVector, not to the static Type definition.

3.2 Binary Compatibility with Parquet Variant Encoding

The binary encoding of Parquet Variant originated in the Spark Variant specification and was later adopted formally by Parquet. The Iceberg Variant proposal compared several alternatives, including BSON, Amazon ION, PostgreSQL JSONB, and the Spark encoding, and ultimately selected the Spark encoding as the standard Variant encoding for Iceberg. Its core advantages are the internal field-name dictionary and offset arrays. These structures support binary search and O(1) jumps, and they eliminate the need to build an additional index by traversal. BSON and ION do not provide this structure.

The value representation is also more compact. Integers can be as small as 1 byte rather than 4 bytes in BSON. Short strings avoid an extra 4-byte length prefix. Array keys are encoded implicitly rather than explicitly, unlike BSON. Spark, Snowflake, DuckDB, and Iceberg have already adopted this encoding. By aligning with it, Velox can directly reuse upstream codec libraries and avoid introducing a private format.

Accordingly, Velox keeps its in-memory representation binary-compatible with the Parquet Variant encoding:

• Avoid dual protocol maintenance: Velox does not need to maintain a separate internal object-tree representation outside the Parquet Variant encoding. This substantially reduces round-trip conversion cost and compatibility risk in the read/write path.
• Unify semantic boundaries: reader output, execution-layer operations, and writer output all use the same binary specification.
• Minimize I/O and CPU cost: the reader writes metadata and value byte streams into memory through zero-copy or minimal conversion, and later operators extract paths directly from those two columns.

3.3 Why This Must Be a Built-in Type

TypeKind::VARIANT

BaseVector::create() dispatches vector creation based on TypeKind. If it reused TypeKind::ROW, create() would produce a RowVector rather than a VariantVector. A dedicated TypeKind::VARIANT ensures that the factory constructs the correct vector type. The reason VariantVector cannot reuse RowVector is described next.

VariantVector: Why It Cannot Inherit from RowVector

The RowVector constructor in ComplexVector.h imposes a hard constraint:

RowVector(..., std::vector<VectorPtr> children, ...)
    : BaseVector(pool, type, VectorEncoding::Simple::ROW, ...),
      childrenSize_(children.size()),
      children_(std::move(children)) {
  VELOX_CHECK_LE(children_.size(), type->size());
}

children_.size() <= type->size() means that the number of children must be known at construction time and cannot exceed the number of fields declared by RowType. For VariantVector:

• VariantType is intentionally schema-free, as described in Section 3.1, so type->size() cannot reflect shredded columns.
• shreddedColumns_ are attached dynamically by the reader from different Parquet row groups, and different batches may expose different path sets.
• If VariantType returned a dynamic size(), then two VariantVector instances with different shredded columns would also have different Type instances, even though execution should still treat them as the same logical type.

Even if the construction constraint were bypassed, RowVector::copy() would still assume that source and target children correspond one by one by index, as shown in ComplexVector.cpp:187:

for (auto i = 0; i < childrenSize_; ++i) {
  BaseVector::ensureWritable(rows, type()->childAt(i), pool_, children_[i]);
  children_[i]->copy(rowSource->childAt(i), rows, toSourceRow);
}

When two VariantVector instances have different shredded schemas, for example, target has {event_type, event_ts} while source has {event_type, user_id}, the meaning of children_[2] is already different. A positional, column-by-column copy() is therefore semantically incorrect.

Note

VariantVector::copy() requires its own merge logic. When source and target shredded schemas differ, it must align paths by union, arbitrate types through compatible promotion, fall back to binary form when types are incompatible, and then append column by column. When the schemas match, it can take a fast path and copy column by column directly with overhead comparable to RowVector::copy().

Comparison: why IPPrefixType can inherit from RowVector

Dimension	IPPrefixType	Variant
Child structure	Fixed two-field ROW(HUGEINT, TINYINT) known at compile time	metadata + value + runtime-dynamic shredded columns
type->size()	Always = 2	Cannot be known statically
copy() semantics	Positional child copy is always correct because both sides have identical children	Positional child copy is wrong because batches may have different shredded columns
SQL field access	SELECT data.ip, data.prefix FROM t, user reads fields directly by name	SELECT variant_get(data, '$.event_type', 'string') FROM t, user accesses through functions

Alternative considered: In the proof-of-concept stage, VariantVector inherited from RowVector and reused VectorEncoding::ROW. That approach reduced the amount of override work, but it introduced two problems. RowVector::copy() cannot handle mismatched shredded schemas. In addition, is_row_kind() would return true, causing VARIANT to enter ROW code paths incorrectly and leading to invalid type conversions. The final design therefore makes VariantVector inherit directly from BaseVector.

VectorEncoding::VARIANT

Once VariantVector inherits from BaseVector, it also needs its own encoding for two reasons:

• Reusing ROW encoding would cause undefined behavior. The codebase contains many paths that do case VectorEncoding::Simple::ROW: and then cast directly to RowVector*. For example, in FieldReference.cpp:

  // FieldReference.cpp:40-42
  case VectorEncoding::Simple::ROW:
    return static_cast<const RowVector*>(&vector);

  If VariantVector reused VectorEncoding::ROW, those paths would cast it to RowVector*, which is undefined behavior.

• Dedicated dispatch path: VectorEncoding::VARIANT allows all BaseVector virtual functions such as slice, copy, estimateFlatSize, serialization, and hashing to take dedicated Variant-specific branches rather than patching those paths with isVariant() checks.

4. Implementation Plan

Note

When submitting PRs, keep each PR as small and self-contained as possible so reviewers can reason about it quickly. The phases below are logical groupings. A single phase may still be split into multiple PRs.

Phase 1: Foundational Type and Vector Support (3 PRs)

• PR 1: TypeKind + VariantType (minimum compilable change): add the TypeKind::VARIANT enum, TypeTraits, and the VariantType class; add a VARIANT branch to VELOX_DYNAMIC_TYPE_DISPATCH_IMPL without falling through to ROW; and fix global switch(typeKind) compilation issues.
• PR 2: VariantVector + BaseVector dispatch: add VectorEncoding::VARIANT, the VariantVector class inheriting from BaseVector, factory creation, a DecodedVector branch, and VELOX_NYI stubs for all serialization, row container, and bridge paths. This PR must land together with the BaseVector inheritance change to avoid falling through to ROW code paths and triggering undefined behavior.
• PR 3: variant_get: first try the fast path by querying shreddedColumn(path), and fall back to decoding the binary representation in metadata + value.

Later Phases

• Parquet Reader + Shredding: add VariantColumnReader to read metadata + value and attach shredded columns.
• Serialization: serialize VARIANT as metadata + value + shreddedColumns. The shredded columns are part of the value and must participate in serialization.
• Merge: implement the Parquet write path, schema-merge logic, boundary tests, and performance benchmarks.
• Variant functions + Gluten integration: implement the 10 Variant SQL functions from Spark 4.1 and wire up Substrait type mapping.

5. Risks and Known Limitations

• Known limitation: VARIANT cannot participate in GROUP BY, join keys, ORDER BY, DISTINCT, and similar operations. This is a Spark 4.1 semantic restriction, not a Velox-specific limitation.
• Global switch maintenance: adding a new TypeKind and VectorEncoding affects roughly 20 encoding switches and even more typeKind switches, creating a long maintenance tail.
• BaseVector override effort: around 15 virtual functions require new implementations, and copy() and copyRanges() are particularly complex because they include merge logic.

5.1 Serialization Boundary Strategy: Preserve Shredded State

At Exchange and Spill boundaries, this design preserves shredded state instead of reconstructing a complete metadata + value blob each time:

	Preserve shredded state (chosen here)	Reconstruct full blob at boundaries
Fast path retention	Shredding acceleration still works after exchange or spill	Lost; all later access falls back to binary navigation
Serialization complexity	Higher: dynamic schema must be encoded	Lower: fixed two-column VARBINARY layout
Reconstruction cost	None	O(variant_size * shredded_fields) per row

The rationale is straightforward. The entire purpose of Shredding is to avoid binary decoding. If every exchange or spill reconstructs a full blob, the fast path enabled by shredded columns survives only within a single pipeline. That outcome discards most of the performance benefit.

Version compatibility: In Phase 1, all serialization paths throw VELOX_NYI for VARIANT. In Phase 2, the design adopts a versioned wire format. Receivers that do not support shredded serialization automatically fall back to full-blob form.

6. Appendix: Understanding VARIANT Binary Encoding Through variant_get

This section walks through a concrete example of the full execution path of variant_get and explains the internal layout of the Parquet Variant binary encoding.

6.1 Example Data

{
  "owner": "Alice",
  "contacts": [
    {
      "age": 30,
      "name": "Bob",
      "phones": [
        {"number": "+1-555-1000", "type": "home"}
      ]
    }
  ]
}

Query: variant_get(data, '$.contacts[0].phones[0].number', 'string')

6.2 Binary Layout of VARIANT

A Variant value consists of two parts: Metadata, the per-row field-name dictionary, and Value, the per-row recursive binary tree with a header byte. Object nodes in Value refer to the Metadata dictionary through field_id. Fields are stored in dictionary order to support binary search, and the offset array enables O(1) jumps to any child node.

The diagram below shows the complete binary layout of the example data. The red arrows show the extraction path for $.contacts[0].phones[0].number:

The full path costs 3 dictionary binary searches, 3 object header parses, 2 array header parses, and 5 offset reads, for a total of about 16 ops/row. Each operation is simple pointer arithmetic that reads a 1-byte to 4-byte integer. The process requires no memory allocation and no string copying.

[baibaichen]

cc: @Yuhta @mbasmanova @pedroerp @rui-mo
Looking forward to your thoughts and insights, thank you.

[zhanglistar]

Looks great! Velox should support this data type.

[mbasmanova]

Thanks for proposing this extension to Velox!

A few questions that would help us evaluate the design:

What queries benefit?

It would be great to include a few representative queries and workloads that would benefit from this, ideally with some performance numbers comparing shredding vs. binary navigation. This would help us understand the practical impact and prioritize the work.

Similarity to subfield pruning — could this be a custom type on top of ROW?

Shredding seems closely related to the existing subfield pruning mechanism in Velox, where the planner collects which parts of a complex type (ROW, MAP, ARRAY) are accessed and tells the reader what to materialize instead of returning everything. Shredding is conceptually the same — the planner identifies which paths within the semi-structured data are accessed and the reader materializes those as typed columns. See also #16968 for related work on extraction pushdown into scan specs.

This suggests the shredded schema could be determined at query planning time, based on which fields the query accesses — similar to Spark's PushVariantIntoScan. If the schema is fixed per query:

• type->size() would be known statically
• RowVector::copy() would work because source and target have the same schema
• Different Parquet row groups that lack a shredded column would populate it from the binary blob during reading
• The existing subfield pruning infrastructure could potentially be reused or extended, rather than building a parallel mechanism from scratch

If this approach works, it would avoid adding a new TypeKind and VectorEncoding, which as you note touches ~20+ switch statements and requires ~15 new virtual function implementations. It would be worth exploring whether the ROW-based approach is viable before committing to a new built-in type.

Naming

Worth noting that VARIANT may cause confusion — in C++, std::variant is a type-safe union, and Velox already uses variant in velox/type/Variant.h as a runtime-typed value holder. These are quite different from what this proposal describes, which is a binary-encoded semi-structured data type. It might be worth considering a name that avoids this ambiguity.

[baibaichen]

Thank you, @mbasmanova, for the detailed feedback. The two remaining questions are addressed below.

Both questions depend on the mechanics of Parquet Variant shredding, so the discussion begins with a brief overview.

Background: Shredding is a write-time decision

Shredding is distinct from subfield pruning or column-extraction pushdown (#16968). Those are query-time optimizations. Shredding is a write-time decision. The Parquet writer extracts selected Variant fields into typed_value columns, as defined in the spec. These columns exist in the file independently of any specific query.

For example, a writer may shred event_type and event_ts:

optional group event (VARIANT) {
  required binary metadata;
  optional binary value;               -- remainder (un-shredded fields only)
  optional group typed_value {
    required group event_type {
      optional binary value;
      optional binary typed_value (STRING);
    }
    required group event_ts {
      optional binary value;
      optional int64 typed_value (TIMESTAMP);
    }
  }
}

Key implications:

• Different files or row groups can expose different shredded column sets.
• The value blob contains only unshredded fields. Shredded columns are stored alongside it.
• Queries that require the complete Variant such as to_json or SELECT data force a ROW(metadata, value) design to reconstruct the full value by merging shredded columns back into value. VariantVector avoids this reconstruction step.

Spark's PushVariantIntoScan (apache/spark#49235) communicates the query requirements to the reader. It does not perform shredding itself. The reader must still reconcile those requirements with the shredded columns that physically exist in each file.

1. What queries benefit?

Shredded columns are standard typed Parquet columns. As a result, they inherit the usual columnar benefits, including statistics, encoding, and projection pruning.

Micro-benchmarks on 1M TPC-H lineitem rows (16-field Variant, i9-13900KF, RelWithDebInfo, GCC 13):

Operation	Binary nav	Shredded	Speedup
Single field extract (l_shipdate)	14.13 ms	0.18 ms	43x
Multi-field extract (3x DOUBLE)	20.36 ms	0.44 ms	24x
Field extract + filter	14.20 ms	0.19 ms	41x

Shredded columns also enable row-group skipping through column statistics and dictionary or RLE encoding. Binary blobs do not provide these benefits.

Spark 4.1 query pattern mapping:

Scenario	Example	Path
Extract shredded field	variant_get(data, '$.event_type', 'string')	return struct: direct typed-column read
Filter on shredded field	WHERE variant_get(...) = 'login'	return struct: with row-group skipping
Non-shredded field	variant_get(data, '$.rare_field', 'string')	return struct: binary navigation decode at scan
Return full Variant	SELECT data, to_json(data)	return variant: pass-through
schema_of_variant_agg	Infer schema	return variant: traverse all data
variant_explode	Expand object or array	return variant
IS NULL	data IS NOT NULL	Null-bitmap check

Scope: This design targets Spark 4.1 and Gluten. Presto is planning Variant support. Iceberg V3 reading is scheduled for September 2026 (prestodb/presto#27198), and Delta Variant is tracked in prestodb/presto#26966. The Delta Variant protocol uses the same encoding and plans shredding separately. This design supports both through optional shreddedColumns_.

2. Could this be a custom type on top of ROW?

Variant is inherently semi-structured. Its only fixed physical schema is ROW(metadata, value).

The issue is that a ROW(metadata, value) design requires the reader to reconstruct the full value at scan time by merging shredded columns back into value. Shredded columns cannot pass the scan boundary as independent columns. As a result, downstream variant_get falls back to binary navigation on the reconstructed blob. This loses the 24x to 43x performance advantage and adds reconstruction cost. Queries that do not eliminate Variant at scan time, including to_json, SELECT data, schema_of_variant_agg, variant_explode, and UDFs, incur this penalty with no opportunity to recover the shredding benefit downstream.

VariantVector avoids this limitation. It carries shredded columns alongside metadata + value through the full pipeline as runtime state rather than as part of the type definition. variant_get uses the fast path through shreddedColumn(). to_json traverses value and shredded columns directly. No reconstruction is required. If a shredded column is missing because different files expose different shredded sets, the implementation falls back to binary navigation with no additional overhead. TypeKind::VARIANT provides a single type identity for all Variant values, regardless of which columns are physically shredded.

Cost acknowledgment

This design introduces approximately 20 switch statements and 15 virtual overrides. However, a ROW-based approach would still require isVariant() guards throughout the codebase to distinguish a ROW that semantically represents a Variant. We explored that approach in the proof of concept. The prototype began with RowVector inheritance and later moved to BaseVector after these issues became apparent.

[baibaichen]

Hi @mbasmanova, let me address the naming question first — I'll follow up on the other two questions tomorrow.

Here are a couple of alternatives I've considered:

Name	Rationale
SEMISTRUCTURED	Precisely describes what the type represents
BINARY_VARIANT	Preserves the upstream "Variant" lineage while disambiguating

My initial preference is SEMISTRUCTURED, because the Parquet Variant spec itself defines its purpose as:

allows representation of semi-structured data (e.g. JSON) in a form that can be efficiently queried by path.

So SEMISTRUCTURED captures the essence of the type most accurately.

However, I do have a concern with this choice: every other engine and lakehouse format in the ecosystem calls it VARIANT — Spark, Snowflake, DuckDB, Iceberg all use this name. Using a different name in Velox would be inconsistent and potentially confusing.

So perhaps the cleanest path is to keep TypeKind::VARIANT for the new type and instead rename the existing Variant class in velox/type/Variant.h to something like DynamicValue. Would that be worth considering?

[mbasmanova]

Thanks for the benchmarks and detailed response. A few follow-up questions:

Different files having different shredded columns — You emphasize this as a key reason for runtime-dynamic shredded schemas. But if event_type is shredded in some files and not others, then variant_get(data, '$.event_type', 'string') uses the fast path for some row groups and falls back to binary navigation for others. For shredding to deliver the 24-43x speedup consistently, the same columns need to be shredded across all files. And if that's the case, the shredded schema is effectively known at query planning time — which is what we suggested.

Subfield pruning is the same mechanism — You note that shredding is a write-time decision while subfield pruning is a query-time optimization. But the reader-side mechanism is the same: the planner tells the reader what columns to materialize. The reader reconciles that with what's physically available in each file. This is exactly how subfield pruning works today for ROW/MAP/ARRAY — some files may have different nested structures, and the reader handles it.

Field access doesn't need Variant at all — For variant_get(data, '$.event_type', 'string'), the planner can rewrite this to read event_type as a regular typed column directly — no Variant type needed in the execution pipeline. This is what Spark's PushVariantIntoScan does. The Variant is eliminated at scan time.

Full Variant queries — For queries that need the complete data (to_json, SELECT data), the reader reconstructs the full blob at read time and returns it as VARBINARY. The reconstruction cost happens once at the scan boundary. The reader already knows whether the query accesses specific fields (via scan spec), so it can choose the right strategy.

In neither case does the execution pipeline need to carry shredded columns alongside the blob.

[baibaichen]

@mbasmanova Thanks for the follow-up.

1. Shredded schema across files

You're right that consistent shredding is needed for consistent speedup. In practice this is often the case within a single write job. However, the Parquet spec does not require cross-file consistency, and Spark's implementation confirms this: each task independently infers its shredding schema from the first 4096 rows / 64MB it sees (InferVariantShreddingSchema.scala), with fields below 10% occurrence dropped and a 300 physical-column cap. Different tasks, different data partitions, different shredding schemas. This means schema heterogeneity across files is a real scenario, not just a theoretical one.

2. PushVariantIntoScan — agreed

When it fires, Variant is eliminated at scan time.

3. When it doesn't fire — the real question

You suggest reconstructing the full blob at scan time and returning VARBINARY. Spark does exactly this. But this is an architectural constraint, not an optimal design choice — Spark's InternalRow cannot carry dynamically varying columns across rows, the same limitation as Velox's RowVector. So blob reconstruction is the pragmatic workaround.

I've spoken with folks at Databricks privately. Their recommendation is to avoid materializing the full Variant blob whenever possible, because every subsequent field access must decode the binary format per row — exactly the cost that shredded typed vectors eliminate. The advantage Velox has as a columnar engine is that we can define VariantVector to keep shredded columns as typed vectors, avoiding materialization entirely.

However, I'm open to starting with reconstructing as Phase 1 to unblock basic support, then using benchmarking to decide whether to introduce Variant as a built-in type.

[baibaichen]

@mbasmanova Does it make sense to you if we begin with a custom type PR as Phase 1, mainly to unblock basic support, and defer the built-in Variant decision until after we have benchmarking data?

However, I'm open to starting with reconstructing as Phase 1 to unblock basic support, then using benchmarking to decide whether to introduce Variant as a built-in type.

[mbasmanova]

I don't see a point in carrying heterogeneously shredded data through the execution pipeline. If the scan produces data whose shape keeps changing from batch to batch, query performance will be terrible — expressions assume a stable schema.

The query knows which columns it needs. The scan should produce exactly that — using shredded columns when available, falling back to binary parsing when not. The shredding is a reader-level optimization, invisible to the rest of the pipeline. This is how subfield pruning already works for ROW/MAP/ARRAY.

What am I missing?

[baibaichen]

Hi @mbasmanova Thanks for the explanation

I see — carrying heterogeneous shredded data through the pipeline is the wrong boundary. Confirmed by reading Spark's PushVariantIntoScan — it rewrites the plan so every access pattern, including the full-variant pass-through, becomes a dedicated field in a STRUCT produced by the scan. For example, the following SQL:

SELECT
  variant_get(data, '$.user.id',    'bigint') AS uid,
  variant_get(data, '$.event.type', 'string') AS etype,
  to_json(data)                               AS raw
FROM t

is rewritten as:

Relation [data: variant]
  -> Relation [data: struct<0: bigint, 1: string, 2: variant>]

variant_get(data, '$.user.id',    'bigint') -> GetStructField(data, 0)
variant_get(data, '$.event.type', 'string') -> GetStructField(data, 1)
to_json(data)                               -> to_json(GetStructField(data, 2))
                                               -- field 2 is a full Variant

So the reader's contract is "produce one column per requested access pattern, with stable types". Shredding reconstruction, when needed for the full-variant slot, happens inside the reader. Nothing heterogeneous leaks into the pipeline. I'll align Velox's design with this model:

• VARIANT as a custom type over ROW(VARBINARY metadata, VARBINARY value)
• Parquet VariantColumnReader consumes required Subfields and rewrites each variant column into a ROW whose fields are the requested access patterns — typed columns for path + target-type requests, and a full (metadata, value) Variant for full-blob requests
• variant_get operates on the blob.

Plan to drop the RFC and send small PRs. Let me know if you'd rather see a short design note first.

[mbasmanova]

Thanks for aligning on the design.

One question: why do we need VARIANT as a custom type over ROW(VARBINARY metadata, VARBINARY value)? If the planner rewrites all variant_get calls into struct field accesses and the reader produces typed columns, then the execution pipeline never sees a Variant. The only case where the pipeline needs the raw data is to_json(data) or SELECT data, and for that a single VARBINARY (the reconstructed blob) would suffice. Who would navigate the metadata + value pair at runtime?

[baibaichen]

Some runtime consumers still need to navigate (metadata, value) per row:

1. The native Parquet writer. Variant on disk is two byte streams in a variant group, so the writer needs them as two distinct buffers to hand to the column writers. A single spliced VARBINARY would force the writer to un-splice every row, and introduce a Velox-private framing format into a path that's supposed to be Parquet-native.
2. variant_get isn't always eliminated. PushVariantIntoScan only rewrites it when the path is a foldable constant:

  case g@VariantGet(v@StructPathToVariant(fields), path, _, _, _)
      if path.foldable =>
    GetStructField(rewriteAttribute(v), fields(RequestedVariantField(g)))

When the path is dynamic, the rewrite doesn't fire and variant_get runs per row, navigating both buffers.

ROW(VARBINARY metadata, VARBINARY value) keeps both cheap and stays aligned with how Iceberg models it.

WDYT?

[mbasmanova]

Thanks for the explanation. A few more design questions:

Package dependency. Up until now, custom types in Velox have been defined by function packages and only understood by functions. Operators and readers/writers do not know about custom types — they see only the underlying physical type. Here, the VARIANT custom type would need to be understood by the Parquet reader/writer in addition to Spark functions. Having Parquet depend on Spark or Spark functions depend on Parquet would be undesirable. This likely needs its own package to define the type, separate from both.

Data lifecycle. How does VARIANT data get created in the first place? In Spark, there are functions like parse_json(string) and to_variant_object(struct/array/map) that convert data into Variant. These would also need to be implemented in Velox for the write path to work. Worth thinking about the full lifecycle — creation, read, transform, write — and which components need to be built.

Relationship to JSON. Spark's existing JSON functions (get_json_object, json_object_keys, json_array_length) operate on plain VARCHAR and do not work with VARIANT. VARIANT comes with its own separate set of functions (variant_get, is_variant_null, variant_explode, schema_of_variant, etc.). Worth keeping in mind that these are two completely independent function sets for what is conceptually similar data.

We are generally aligned on adding Variant support to Velox and appreciate the effort you've put into this proposal. Please go ahead with prototyping and share the design as it emerges — we'd love to follow along and provide feedback as these questions get resolved.

[mbasmanova]

CC: @PingLiuPing @xiaoxmeng

[PingLiuPing]

@baibaichen Thanks for the proposal.

Support column stats is one of the key benefit of variant type, could you help to explain how variant column stats are been collected especially for non-shredding column? This could help us understand/review the potential VariantColumnReader. Does Spark currently calculate them during writing variant column?

And it seems parquet C++ does not support reading and writing variant type yet.

[baibaichen]

On stats:

For an unshredded VARIANT (just metadata + value as two BINARY fields), Parquet can still produce generic column statistics for the value column, but those stats are byte-wise and generally not useful for semantic predicate skipping on Variant paths.

The “column stats” benefit really comes from shredding: when a field is stored in a typed Parquet column (typed_value), that typed column has normal Parquet stats (min/max/null_count, etc.) and can be used for file/row-group/page skipping, under the constraint called out in the Variant shredding spec (e.g. value is always null so the type is stable). Readers can then push down variant_get(...)=... as filters on the typed column when possible, and fall back to decoding the value blob otherwise.

Note

see https://github.com/apache/parquet-format/blob/master/VariantShredding.md

On Spark, my understanding is that it doesn’t compute any special VARIANT stats; it relies on the normal Parquet per-column stats. Also, each writing task infers its own shredding schema from a sample of the data it sees (up to the first 4096 rows or 64MB, whichever comes first). Within that sample, fields with <10% occurrence are dropped, and there’s a cap of 300 physical columns.

Note

see InferVariantShreddingSchema.scala

On Parquet C++: as far as I know, @wgtmac is actively driving this area and likely has the most accurate status/plan (esp. around shredding).

[mbasmanova]

CC: @srsuryadev

[PingLiuPing]

Link this issue #14368

[srsuryadev]

We are generally aligned on adding Variant support to Velox and appreciate the effort you've put into this proposal. Please go ahead with prototyping and share the design as it emerges — we'd love to follow along and provide feedback as these questions get resolved.

@mbasmanova / @baibaichen - do we have any PR or prototype for this in velox already, so this can be supported in Nimble?

[jja725]

Hi @baibaichen, thanks for this great proposal. Do we have tentative timeline for the prototype? Looking forward for it and we can collaborate on it