[FR] Support parquet output / input

[Tyrannas]

When coming to big csv, some people are using parquet files to compress the data.

I could be nice to be able to output a csv from xan to parquet, or to read a parquet (even if I think since it's binary a lot of xan functions wouldnt be compatible without converting it to csv first).

And rust probably has a way to read those files since polars is already supporting them.

[Yomguithereal]

So parquet is a bit of a thorny problem. I would say it is not exactly correct to consider it as a way to compress tabular data. Rather, it's a binary format, where column are typed like in a relational database. Also it is column-oriented, like R or pandas dataframes, contrary to CSV which is row-oriented.

It is probably feasible to convert from & to parquet but it means losing the typing in the process, so I am unsure what is the good way to go here.

Also it is a complex format and I must check what overhead it would add to xan's binary to support it.

[Yomguithereal]

@Tyrannas I have experimented with xan from -f parquet in the parquet branch. You can install it thusly:

cargo install --git https://github.com/medialab/xan --locked --branch parquet

It does add 3-4MB to the executable which is ~20% of the size of the executable currently so I am not sure I want to merge it. Can you try it and tell me what you think?

Else I think a parquet-oriented tool might be a better solution. See nail for instance.

[Tyrannas]

Works like a charm on my tests (linux fedora), I understand that adding 20% might be a lot for a "niche" feature. I don't know cargo enough but could an extra install be used ? Like python. In python it would be something like pip install xan[parquet]

[Yomguithereal]

It could work with a feature yes, but I was trying to avoid this for xan as of yet because it complexifies packaging greatly. Did you try nail convert also?