[DISCUSS] v3.0.0

[kevinjqliu]

Tracking Issue for v3.0.0

Open issues related to v3.0.0

• feat: Track table generation with progress bars #226
• feat: Add parquet subcommand with format-specific argument validation #234
• Replace deprecated assert_cmd::Command::cargo_bin in tpchgen-cli integration tests #248

Changes since last release: v2.0.2...main

I know theres a lot of work done on tpchdsgen, maybe we can include that in the release too

[alamb]

For tpcdsgen I think what we need to do is figure out if we want to mirror the java or C generator

I think I am not going to have time to work on this (tpcds) for a few weeks at least unfortunately

[kevinjqliu]

I can help take a look at tpcds when I find some time. I remember there was also discussions around moving tpcds into its own repo.
For now, I think we can ship v3.0.0 without tpcds; there are already a bunch of improvements since the last release.
Thoughts?

[alamb]

I can help take a look at tpcds when I find some time. I remember there was also discussions around moving tpcds into its own repo. For now, I think we can ship v3.0.0 without tpcds; there are already a bunch of improvements since the last release. Thoughts?

Sounds good to me

[clflushopt]

I think the Java version is what's popular but reviewing the issues that were flagged a while ago show a divergence between both. I think aligning with Java makes more sense ? But we can always offer some sort of --compat=tpcds that would align with the C version (food for thought). I have it in my todo list to address this but I think you can go on for v3 without tpcds @kevinjqliu

[kevinjqliu]

I'll help get both #226 and #234 in for v3.0.0.

#234 adds the parquet subcommand to the CLI. I think it'll be a good idea to just bite the bullet now and deprecate the --format flag in favor of subcommands. I dont want to leave the CLI in a partial state where we are supporting both --format and sub commands (tpchgen-cli parquet)

The new CLI should look something like

Usage: tpchgen-cli [COMMAND] [OPTIONS]

Commands:
  tbl      Generate TBL files (default)
  csv      Generate CSV files
  parquet  Generate Parquet files

and the default behavior should still generate tbl

tpchgen-cli -s 1

[alamb]

The benefit of the existing setup is that tpchgen-cli uses the same arguments as the original dbgen

However, as long as we update the documentation to reflect the new reality i think it is fine

[clflushopt]

+1 for deprecating the old flag.

[kevinjqliu]

been a bit busy in between conferences, will come back to this after iceberg summit (4/8-9)

I still plan to clean up the remaining items and release v3.0.0
Im think we can ship the parquet subcommand PR and the progress_tracker PR. I also want to ship a csv subcommand PR with v3.0.0. If we're going to make breaking changes to the CLI, might as well change it altogether 😄

[alamb]

soudns good

I would love to get the initial tpc-ds support in to tpchgen-cli as well (even if we put a WARNING or something that we are still working out the result differences). I think that could also help draw other people to helping finish up up

[clflushopt]

I've been a bit busy as well to finish TPCDS support this month. I will try and set some time aside in the next few weeks to work on the core changes. I'm a bit more comfortable now with using Claude Code for this it will just take some time to review and and work on performance optimizations.

[alamb]

Nice -- I think we should focus first on getting the right answers (I agree Claude code / codex should be able to help a lot here)

Then we can go nuts optimizing.

Can't wait to see it

[clflushopt]

hey @kevinjqliu how can I help move this forward ?

[kevinjqliu]

hey! thanks for the ping! lets get this done!! i plan to work on this tomorrow 😄