What
Two remaining pieces at the native↔host boundary. The nested-type transpose is done — the row↔Arrow converter carries ROW/ARRAY/MAP/VARBINARY (via Flink's own vendored Arrow machinery), so nested protobuf/Avro/JSON route today — and the Parquet columnar sink ships.
1. Columnar sinks beyond Parquet
The throughput end state for columnar workflows: a native columnar sink (and a columnar collect for SQL result fetching) consuming ArrowBatch directly, so a native source→…→sink pipeline pays no transpose at all, nested columns included. The transpose exists only to bridge to rowwise host operators — removing the rowwise edge removes the cost. This would also let the boundary read() be a true zero-copy view (today it deep-copies because the boundary operator releases the batch immediately). Currently only the Parquet sink is columnar; there is no ORC sink or columnar collect. Relates to the Iceberg/remote-endpoints issue.
2. Protobuf representation reconciliations (gated)
No longer boundary gaps — these are decode-semantics divergences between the descriptor-derived Arrow schema and Flink's PbToRowTypeUtil mapping, each gated to fallback until reconciled and parity-tested:
Also: Avro/JSON nested objects can now route through the boundary (same converter fix) — wire + parity-test when picked up.
Each item lifts a specific gate in docs/coverage-and-fallbacks.md; update it in the same commit.
What
Two remaining pieces at the native↔host boundary. The nested-type transpose is done — the row↔Arrow converter carries
ROW/ARRAY/MAP/VARBINARY(via Flink's own vendored Arrow machinery), so nested protobuf/Avro/JSON route today — and the Parquet columnar sink ships.1. Columnar sinks beyond Parquet
The throughput end state for columnar workflows: a native columnar sink (and a columnar
collectfor SQL result fetching) consumingArrowBatchdirectly, so a native source→…→sink pipeline pays no transpose at all, nested columns included. The transpose exists only to bridge to rowwise host operators — removing the rowwise edge removes the cost. This would also let the boundaryread()be a true zero-copy view (today it deep-copies because the boundary operator releases the batch immediately). Currently only the Parquet sink is columnar; there is no ORC sink or columnar collect. Relates to the Iceberg/remote-endpoints issue.2. Protobuf representation reconciliations (gated)
No longer boundary gaps — these are decode-semantics divergences between the descriptor-derived Arrow schema and Flink's
PbToRowTypeUtilmapping, each gated to fallback until reconciled and parity-tested:enum— int-vs-name, matched to the declared column typefixedints — unsigned on the native side vs signed in Flink (a value divergence)bytes— decode works and the boundary carriesVARBINARY; only thebyte[]parity-test comparison is missinggoogle.protobuf.*) — dedicated Arrow type vs Flink's nested rowAlso: Avro/JSON nested objects can now route through the boundary (same converter fix) — wire + parity-test when picked up.
Each item lifts a specific gate in
docs/coverage-and-fallbacks.md; update it in the same commit.