Is your feature request related to a problem?
The current Spark connector uses the DataSource V1 API. This limits the optimizations that Spark's Catalyst optimizer can delegate to OpenSearch. For example, aggregations like GROUP BY with COUNT cannot be pushed down to OpenSearch, forcing Spark to pull all matching documents and perform aggregation in memory.
This limitation has been reported in #302.
What solution would you like?
Migrate the Spark connector from DataSource V1 to DataSource V2 API. This would enable the following optimizations:
| Capability |
Spark Interface |
Benefit |
| Aggregation pushdown |
SupportsPushDownAggregates (Spark 3.2+) |
Translate GROUP BY / COUNT / SUM etc. into OpenSearch aggregation queries, returning only aggregated results instead of all documents |
| LIMIT pushdown |
SupportsPushDownLimit (Spark 3.3+) |
Pass size: N to OpenSearch instead of scrolling through all documents |
| TopN pushdown |
SupportsPushDownTopN (Spark 3.3+) |
Translate ORDER BY ... LIMIT N into OpenSearch sort + size |
| Statistics reporting |
SupportsReportStatistics |
Report row count and data size estimates to Spark for better JOIN strategy selection |
| Partitioning reporting |
SupportsReportPartitioning |
Inform Spark about OpenSearch shard layout to reduce unnecessary shuffles |
The V2 API is stable across Spark 3.2+ and Spark 4.0, so the implementation would be compatible with both the sql-30 and sql-40 modules.
A phased approach could be taken:
- Phase 1: Implement the core V2 framework (
TableProvider, Table, ScanBuilder, Batch) with existing filter pushdown
- Phase 2: Add
SupportsPushDownAggregates for aggregation pushdown
- Phase 3: Add
SupportsPushDownLimit and SupportsPushDownTopN
What alternatives have you considered?
N/A
Do you have any additional context?
Is your feature request related to a problem?
The current Spark connector uses the DataSource V1 API. This limits the optimizations that Spark's Catalyst optimizer can delegate to OpenSearch. For example, aggregations like
GROUP BYwithCOUNTcannot be pushed down to OpenSearch, forcing Spark to pull all matching documents and perform aggregation in memory.This limitation has been reported in #302.
What solution would you like?
Migrate the Spark connector from DataSource V1 to DataSource V2 API. This would enable the following optimizations:
SupportsPushDownAggregates(Spark 3.2+)GROUP BY/COUNT/SUMetc. into OpenSearch aggregation queries, returning only aggregated results instead of all documentsSupportsPushDownLimit(Spark 3.3+)size: Nto OpenSearch instead of scrolling through all documentsSupportsPushDownTopN(Spark 3.3+)ORDER BY ... LIMIT Ninto OpenSearchsort+sizeSupportsReportStatisticsSupportsReportPartitioningThe V2 API is stable across Spark 3.2+ and Spark 4.0, so the implementation would be compatible with both the
sql-30andsql-40modules.A phased approach could be taken:
TableProvider,Table,ScanBuilder,Batch) with existing filter pushdownSupportsPushDownAggregatesfor aggregation pushdownSupportsPushDownLimitandSupportsPushDownTopNWhat alternatives have you considered?
N/A
Do you have any additional context?
SupportsPushDownAggregatesAPI: https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/connector/read/SupportsPushDownAggregates.html