Implementing a custom parquet index

[bchalk101]

Hi,

I am trying to implement a custom index, using a TableProvider as suggested in the parquet index examples. I have columns in which users would like to do needle-in-the-haystack queries with low latency, so I would like to index those columns.

I have implemented the TableProvider and basic indexing works.
However, I also want the features built into ListingTable, specifically, hive partitioning. Is there a way to combine functionality?

Beyond, this I am also getting an issue with running a count on the dataset using my custom TableProvider, with the following error:

[2025-02-24T19:38:39Z ERROR reader_service::datafusion_executor] Could not count dataset error: Internal error: Physical input schema should be the same as the one converted from logical input schema. Differences: .

without any differences being printed, any idea what may be causing this? Something specific that I may be missing from my TableProvider?

Thnx

[phillipleblanc]

To resolve that error, you can try setting the option datafusion.execution.skip_physical_aggregate_schema_check to true. ref: https://datafusion.apache.org/user-guide/configs.html

If you want the hive partitioning feature of ListingTable you either need to use it directly or implement it yourself. I had to do a similar thing for my own TableProvider and unfortunately ended up needing to copy a lot of code from ListingTable since the internal methods weren't public. I'm working on a change that should make this more generic and available for downstream implementations.

[bchalk101]

I went with both of those, was a bit annoying to have to implement it but not too much work.

Also, still intrigued what is causing the schema issue...