[Discussion] add a CLI tool for debug and test purpose

[zhjwpku]

I'm thinking of adding a CLI tool to manipulate the catalog, and potentially execute queries and update the underlying dataset, to simplify integration testing. With this tool, we could easily add new test cases by writing simple test specification files and corresponding expected output files.

Additionally, we could integrate this CLI with other Iceberg implementations (e.g., PyIceberg or Spark) to enable cross-language integration tests, which has been discussed here [1].

I haven't thought deeply yet about cross-language integration tests, but I believe adding a CLI tool for iceberg-cpp would significantly benefit our development workflow.

Thoughts?

[1] https://lists.apache.org/thread/wkfbshdngw32jdzclp9hj6flpcdtbl1l

[wgtmac]

I think this is a good idea. However, the CLI tool cannot live without a catalog. Do we want to wrap the rest catalog client (WIP) to connect to a real catalog server implementation like https://hub.docker.com/r/apache/iceberg-rest-fixture, or implement a catalog based on local file system like the HadoopCatalog in the Java repo?

[zhjwpku]

I think this is a good idea. However, the CLI tool cannot live without a catalog. Do we want to wrap the rest catalog client (WIP) to connect to a real catalog server implementation like https://hub.docker.com/r/apache/iceberg-rest-fixture, or implement a catalog based on local file system like the HadoopCatalog in the Java repo?

I think either RestCatalog or SqlCatalog would work. If we use SqlCatalog with SQLite, the local filesystem should be sufficient. On the other hand, RestCatalog requires a running REST catalog server. In the long term, it would be great for the CLI to support both.