[FEATURE REQUEST]: Please consider extending indexing support to non - spark implementation of structured storage, specifically, stand - alone Java and Rust implementation of Parquet / Delta Lake #514
Description
Feature requested
Similar to Access database, an SQL Server engine counterpart, running in process of application, please consider extending indexing support to Spark counterpart, currently provided as libraries, over delta lake. Of those two notable implementations are Java DSR/DSW ( stand - alone reader and writer ):
https://github.com/delta-io/connectors
and Rust delta lake implementation:
https://github.com/delta-io/delta-rs
With proliferation and success of such Data Warehousing implementations as Snowflake, the ETL pipeline of data warehouse increasingly relies on external storage as staging area, mostly, for cost saving features.
Extending support to indexing over external delta lake storage would significantly strengthen appeal of Azure Storage solutions to practically all BI implementations, without exceptions, providing nice financial boost to entire platform as well as competitive edge with regards to query performance.
As a type of user = data engineer, data scientist, developer etc., I want desired feature, in order to goal.
Acceptance criteria
Describe what you consider (e.g., functionality requirement) has to be true to mark this feature as complete
- Acceptance criteria 1
- Acceptance criteria 2
- ...
Success criteria
Define any measurements that need to be performed
- Measurement 1 (e.g., measure the index overhead)
- Measurement 2 (e.g., measure the execution time for TPC-H Q1)
- UX improvement 1 (e.g., it should take one command to do this)
Additional context
Add any other context or screenshots about the feature request here.