Spark DataSource V2 read and write benchmarks?

[geserdugarov]

Integration of Spark Datasource V2 was done in RFC-38. However, there were multiple issues with advertising a Hudi table as V2 without actually implementing certain APIs, and with using custom relation rule to fall back to V1 API. As a result, the current implementation of HoodieCatalog and Spark3DefaultSource returns a V1Table instead of HoodieInternalV2Table, in order to address performance regressions.

Performance issues were not revealed in the initial PR due to the absence of proper benchmarking for such changes. Therefore, to restart this work, it is important first to decide how to benchmark the changes. Among other things, Datasource V1 allows custom logic, such as the use of Hudi indexes, which is not straightforward to implement in Datasource V2. So we need to consider cases like this in the benchmarking scenarios.

If anybody has already gone down this path, please share your insights. Any suggestions about scenarios that should be considered are also welcome.

[vinothchandar]

@leesf tagging you in case you have some old context to add/capture here.