Parquet RFC

[froody]

I've been experimenting with OpenZL for pointcloud compression and find it significantly outperforms the best compression offered by parquet by more than 20% (compression ratio), so there is a strong motivation for writing my own file format and IO libraries. Parquet however offers its own benefits of:

• compatibility with a wide variety of query engines
• efficient remote queries via iceberg
• spatial queries via postgis, sedonadb, duckdb-spatial or pycanopy

I would love to have the best of both worlds. Is there any effort to integrate OpenZL with Parquet? Can I help at all?

[daniellerozenblit]

Thanks for sharing those results — 20%+ is compelling.

We don't currently have a native integration with Parquet, but this is something we'd like to pursue once the OpenZL frame format is finalized.

We do already support compression on top of the Parquet format today:

zli train --sample-dir <sample_dir> --output trained.zl --profile parquet

zli benchmark <sample_dir> --compressor trained.zl

One note: your Parquet file must be in the "canonical" format (i.e. uncompressed). See our docs for more information.

[froody]

Yeah I am aware of the whole-file parquet compression, but that doesn't really help my use-case of wanting to filter columns/row-groups without decompressing the whole file.