Add support for TopN and aggregation pushdown in Elasticsearch #23118

murthy-chelankuri · 2024-08-23T17:34:49Z

Description

This pull request adds support for pushing down TopN and Aggregation to Elasticsearch.

Additional context and related issues

Opening a new merge request because the previous one was automatically closed for being stale, and we couldn't reopen the stale pull request.

#16919

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
(x) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Fixes
* Support for order by pushdown in elasticsearch connector #12381 
* Elasticsearch connector aggregation push down support #7026 
* Elasticsearch connector TopN pushdown #4803

cla-bot · 2024-08-23T17:47:43Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Murthy Chelankuri.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2024-08-23T18:18:52Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Murthy Chelankuri.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email [email protected]
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

murthy-chelankuri · 2024-08-23T21:03:04Z

@martint @kokosing @hashhar Can you please review this MR at the earliest possible.

Praveen2112 · 2024-09-04T07:28:53Z

Can we split the TopN and aggregation pushdown as a separate commit, this would make the reviews a bit easier

Praveen2112 · 2024-09-04T07:30:13Z

plugin/trino-elasticsearch/src/main/java/io/trino/plugin/elasticsearch/expression/TopN.java

+    public TopN addSortItem(TopNSortItem sortItem)
+    {
+        topNSortItems.add(sortItem);
+        return this;
+    }


It is always a bit recommended to make the object TopN immutable instead of changing its state here

Thanks @Praveen2112 for the feed back. removed this method and made TopN as immutable.

Praveen2112 · 2024-09-04T07:31:10Z

...lasticsearch/src/test/java/io/trino/plugin/elasticsearch/BaseElasticsearchConnectorTest.java

+                    SUPPORTS_COMMENT_ON_COLUMN,
+                    SUPPORTS_COMMENT_ON_TABLE,
+                    SUPPORTS_CREATE_MATERIALIZED_VIEW,
+                    SUPPORTS_CREATE_SCHEMA,
+                    SUPPORTS_CREATE_TABLE,
+                    SUPPORTS_CREATE_VIEW,
+                    SUPPORTS_DELETE,
+                    SUPPORTS_INSERT,
+                    SUPPORTS_MERGE,
+                    SUPPORTS_RENAME_COLUMN,
+                    SUPPORTS_RENAME_TABLE,
+                    SUPPORTS_ROW_TYPE,
+                    SUPPORTS_SET_COLUMN_TYPE,
+                    SUPPORTS_UPDATE -> false;
+            case SUPPORTS_LIMIT_PUSHDOWN,
+                    SUPPORTS_TOPN_PUSHDOWN,
+                    SUPPORTS_AGGREGATION_PUSHDOWN -> true;


Can we revert back the indentations ?

Thanks @Praveen2112 for the feedback reverted the indentations.

Praveen2112 · 2024-09-04T07:31:23Z

...lasticsearch/src/test/java/io/trino/plugin/elasticsearch/BaseElasticsearchConnectorTest.java

+                "SELECT " +
+                "text_column " +
+                "FROM " + indexName + " " +
+                "WHERE text_column LIKE 's_.m%ex\\t'"))


Can we revert the indentations ?

I am not see any issue with indentation on my editor. Just compared with the previous version i don't see any change in the indentation. Can you please check once ?

Praveen2112 · 2024-09-04T07:35:56Z

...rino-elasticsearch/src/main/java/io/trino/plugin/elasticsearch/ElasticsearchTableHandle.java

-        OptionalLong limit)
+        List<TermAggregation> termAggregations,
+        List<MetricAggregation> metricAggregations,
+        Optional<TopN> topN)


Instead of maintaining TopN which maintains limit as well - Can we have an Optional<List<SortOrder>> if the list is present we could consider it as TopN else Limit - But we need to handle what if TopN -> Limit and otherway around

@Praveen2112 , can we keep the topN instead of splitting two arguments Optional<List> and limit? But if that is what we need to do, we can look into changing accordingly.

github-actions · 2024-10-22T17:03:05Z

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

github-actions · 2024-11-12T17:03:13Z

Closing this pull request, as it has been stale for six weeks. Feel free to re-open at any time.

github-actions · 2025-01-06T17:03:27Z

This pull request has gone a while without any activity. Tagging for triage help: @mosabua

mosabua · 2025-01-10T00:27:29Z

Added stale-ignore labels since I think @murthy-chelankuri is wanting to continue work on this .. do you know what your next steps are @murthy-chelankuri or do you need help from reviewers?

mosabua · 2025-01-16T17:41:57Z

You need to rebase this PR @murthy-chelankuri to proceed .. it is currently not reviewable

cocobeach1911 · 2025-04-22T18:30:40Z

Thank you @murthy-chelankuri for building this!

Any more updates on this?

cla-bot bot added the cla-signed label Aug 23, 2024

murthy-chelankuri mentioned this pull request Aug 23, 2024

Elastic search plugin support for pushdown TopN and Aggregation #16919

Closed

mosabua requested review from martint, kokosing and hashhar August 23, 2024 17:42

cla-bot bot removed the cla-signed label Aug 23, 2024

murthy-chelankuri force-pushed the es-pushdown-topn-aggregation1 branch from f75d92c to 2935da8 Compare August 23, 2024 18:40

cla-bot bot added the cla-signed label Aug 23, 2024

ebyhr changed the title ~~Add support for TopN and aggregation pushdown~~ Add support for TopN and aggregation pushdown in Elasticsearch Aug 24, 2024

findinpath requested a review from mayankvadariya August 27, 2024 10:49

Praveen2112 reviewed Sep 4, 2024

View reviewed changes

hashhar removed their request for review September 27, 2024 20:49

murthy-chelankuri requested a review from Praveen2112 September 30, 2024 19:52

github-actions bot added the stale label Oct 22, 2024

github-actions bot closed this Nov 12, 2024

murthy-chelankuri reopened this Dec 14, 2024

murthy-chelankuri and others added 4 commits December 14, 2024 09:50

Add support for TopN and aggregation pushdown

ef68f98

Fix indentation

0dac118

Converted TopN to record class

2182bb0

Removed TopN addSortItem and changed Top.topNSortItems as immutable

02c2107

murthy-chelankuri force-pushed the es-pushdown-topn-aggregation1 branch from c85c5eb to 02c2107 Compare December 14, 2024 17:53

github-actions bot removed the stale label Dec 16, 2024

github-actions bot added the stale label Jan 6, 2025

mosabua added stale-ignore Use this label on PRs that should be ignored by the stale bot so they are not flagged or closed. and removed stale labels Jan 10, 2025

github-actions bot added release-notes docs ui Web UI jdbc Relates to Trino JDBC driver hudi Hudi connector iceberg Iceberg connector delta-lake Delta Lake connector hive Hive connector bigquery BigQuery connector mongodb MongoDB connector labels Jan 16, 2025

murthy-chelankuri force-pushed the es-pushdown-topn-aggregation1 branch from 13032b5 to 02c2107 Compare January 16, 2025 19:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for TopN and aggregation pushdown in Elasticsearch #23118

Add support for TopN and aggregation pushdown in Elasticsearch #23118

murthy-chelankuri commented Aug 23, 2024

cla-bot bot commented Aug 23, 2024

cla-bot bot commented Aug 23, 2024

murthy-chelankuri commented Aug 23, 2024

Praveen2112 commented Sep 4, 2024

Praveen2112 Sep 4, 2024

murthy-chelankuri Sep 16, 2024

Praveen2112 Sep 4, 2024

murthy-chelankuri Sep 12, 2024

Praveen2112 Sep 4, 2024

murthy-chelankuri Sep 12, 2024

Praveen2112 Sep 4, 2024

murthy-chelankuri Sep 25, 2024

github-actions bot commented Oct 22, 2024

github-actions bot commented Nov 12, 2024

github-actions bot commented Jan 6, 2025

mosabua commented Jan 10, 2025

mosabua commented Jan 16, 2025

cocobeach1911 commented Apr 22, 2025

Add support for TopN and aggregation pushdown in Elasticsearch #23118

Are you sure you want to change the base?

Add support for TopN and aggregation pushdown in Elasticsearch #23118

Conversation

murthy-chelankuri commented Aug 23, 2024

Description

Additional context and related issues

Release notes

cla-bot bot commented Aug 23, 2024

cla-bot bot commented Aug 23, 2024

murthy-chelankuri commented Aug 23, 2024

Praveen2112 commented Sep 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 22, 2024

github-actions bot commented Nov 12, 2024

github-actions bot commented Jan 6, 2025

mosabua commented Jan 10, 2025

mosabua commented Jan 16, 2025

cocobeach1911 commented Apr 22, 2025