feat: Add integration tests for tpchgen-cli

[alamb]

• Related to feat: Chunkify single parts to generate them in parallel #155
• Related to Alamb/smaller row groups #151

Rationale

As we make changes to tpchgen-cli, we need some way to test that our changes are correct.
Let's add some tests

Changes

Add CLI based tests using assert_cmd

THis was partly written by GPT 4.0 / Github copilot

[alamb]

FYI @@mrendi29 as well

[alamb]

The new tests are run as part of CI. Here is an example:

• https://github.com/clflushopt/tpchgen-rs/actions/runs/16294001069/job/46010712380?pr=156#step:3:343

[mrendi29]

Thanks @alamb for taking a stab at this. @kevinjqliu i recall you had some test that you had not pushed yet, so question to you both, do we want to build off the integration tests suite within this branch/PR or should we split this into multiple PRs?

[clflushopt]

Thanks @alamb I am approving this PR to unblock and will let you resolve the comments from @mrendi29. As a follow up, do we want to have /data/ use two sources of truth duckdb + parquet and dbgen outputs ?

[alamb]

Thanks @alamb for taking a stab at this. @kevinjqliu i recall you had some test that you had not pushed yet, so question to you both, do we want to build off the integration tests suite within this branch/PR or should we split this into multiple PRs?

I personally think we can build from this integration test and then add the "check the output parquet file format"

• I think the tests @kevinjqliu was referring to are in feat: make parquet row groups size configurable  #158

[alamb]

Thanks @alamb I am approving this PR to unblock and will let you resolve the comments from @mrendi29. As a follow up, do we want to have /data/ use two sources of truth duckdb + parquet and dbgen outputs ?

Thank you @clflushopt 🙏

When you say "two sources of truth" I wonder what you mean? Do you mean some way to verify that the actual values written into the .parquet are are correct?

The tests in this PR do not do that (as you note)

[clflushopt]

Yes one for the tbl format via dbgen and one for the parquet format via duckdb we can generate samples at various scale factors and have them in the repo as fixtures.

[alamb]

Yes one for the tbl format via dbgen and one for the parquet format via duckdb we can generate samples at various scale factors and have them in the repo as fixtures.

Filed #161 to track

[alamb]

Thanks again @clflushopt