The shipped Parquet sink is encoding-only native: Rust encodes into an off-heap buffer and the host drains it (one pinned-array memcpy) into Flink's RecoverableWriter streams, which reuse Flink's filesystem plugins, exactly-once commit, and partition-commit machinery verbatim (divergences/22). That trade was chosen for 1:1 fidelity and coverage, but it is not the first-principles optimum: the bytes cross JNI, pass through Flink's stream stack, and on S3 get staged to local disk (RefCountedFile) before the multipart upload.
The optimum is Arroyo's shape — Rust owns the IO end to end via object_store:
- multipart parts uploaded directly from the encoder's buffer (no JNI crossing for data, no local-disk staging),
- upload IDs + part ETags checkpointed through Flink state (JNI carries only the small commit metadata),
- complete-multipart on
notifyCheckpointComplete, idempotent finalize on restore (size-check against an existing object),
- reference:
arroyo-connectors/src/filesystem/sink/ (V2 OpenFile state machine, single-PUT small files) and arroyo-storage.
Costs to weigh before building: credential translation per scheme (flink-conf s3.*/fs.s3a.* → object_store, unknown keys must fall back), reimplementing partition commit (or keeping the host committer and feeding it), coverage regression (no HDFS/OSS in object_store core), and a second commit protocol to test for exactly-once. Benchmark first: measure the drain+stream path against a direct-upload prototype at realistic sink throughput; the encoding-only sink's overhead is one memcpy of compressed bytes plus Flink's own stream stack, which the host sink also pays — the win may not clear the complexity bar.
The shipped Parquet sink is encoding-only native: Rust encodes into an off-heap buffer and the host drains it (one pinned-array memcpy) into Flink's
RecoverableWriterstreams, which reuse Flink's filesystem plugins, exactly-once commit, and partition-commit machinery verbatim (divergences/22). That trade was chosen for 1:1 fidelity and coverage, but it is not the first-principles optimum: the bytes cross JNI, pass through Flink's stream stack, and on S3 get staged to local disk (RefCountedFile) before the multipart upload.The optimum is Arroyo's shape — Rust owns the IO end to end via
object_store:notifyCheckpointComplete, idempotent finalize on restore (size-check against an existing object),arroyo-connectors/src/filesystem/sink/(V2OpenFilestate machine, single-PUT small files) andarroyo-storage.Costs to weigh before building: credential translation per scheme (flink-conf
s3.*/fs.s3a.*→ object_store, unknown keys must fall back), reimplementing partition commit (or keeping the host committer and feeding it), coverage regression (no HDFS/OSS in object_store core), and a second commit protocol to test for exactly-once. Benchmark first: measure the drain+stream path against a direct-upload prototype at realistic sink throughput; the encoding-only sink's overhead is one memcpy of compressed bytes plus Flink's own stream stack, which the host sink also pays — the win may not clear the complexity bar.