[FEATURE] Moving from olivere/elastic noticing missing functionality.

[bradleyadamssailthru]

Is your feature request related to a problem?

Previously we had been using https://github.com/olivere/elastic in our go service but as it is deprecated we started looking at switching to used opensearch-go. We came across some some missing functionality:

Query DSL https://www.elastic.co/guide/en/elasticsearch/reference/7.0/query-dsl.html
elastic.Query
elastic.BoolQuery
elastic.NewBoolQuery
elastic.NewWildcardQuery
elastic.TermsQuery
elastic.NewTermQuery
elastic.NewTermsQuery
elastic.NewExistsQuery
elastic.NewRangeQuery
elastic.NewConstantScoreQuery
elastic.NewQueryStringQuery
elastic.NewIdsQuery

Search APIs https://www.elastic.co/guide/en/elasticsearch/reference/current/search.html
elastic.SearchService
elastic.SearchResult
elastic.FetchSourceContext
elastic.NewFetchSourceContext
elastic.NewCollapseBuilder
elastic.NewFieldSort
elastic.ScriptSort
elastic.NewScriptSort

Scripting https://www.elastic.co/guide/en/elasticsearch/reference/7.0/modules-scripting.html#modules-scripting
elastic.NewScript
elastic.NewScriptInline

Document APIs https://www.elastic.co/guide/en/elasticsearch/reference/7.0/docs.html
elastic.NewBulkIndexRequest

elastic.Error

What solution would you like?

Are these likely to be added?

What alternatives have you considered?

Keep using our current lib for now - but that might block us upgrading Opensearch and leverage latest functionalities.

Do you have any additional context?

It looks like some this functionality is in opensearch-java:
Query DSL - https://github.com/opensearch-project/opensearch-java/tree/c4f674cad2cb7f433567039249453f66ae0e2c47/java-client/src/main/java/org/opensearch/client/opensearch/_types/query_dsl

[dblock]

@bradleyadamssailthru Any missing OpenSearch functionality should/can be added. First, check whether the feature exists in OpenSearch (we forked at 7.10), then please do contribute, and double confirm that all your contributions are under the APLv2 license.

[micahcraig]

@dblock Hey dB, good to see you.

The deprecated olivere Go client had a pretty robust Go DSL for generating complex ES Query DSL JSON in a type safe manner. eg: https://github.com/olivere/elastic/blob/b7de00ddfd128672eac82ef8f6e938bcbb6c2cd4/search_queries_range.go

The go-elasticsearch client have decided that this sort of thing is out of scope for a low level API client: elastic/go-elasticsearch#42

So esquerydsl (https://github.com/mottaquikarim/esquerydsl) was created to aid with ES Query DSL generation, but it is slightly less complete than the olivere implementation was (eg: no has_parent queries.)

Question is: How, if at all, has the Query DSL diverged since OpenSearch forked? Should it be safe to use queries generated and tested against ES in OpenSearch applications? How are other language communities handling complex Query DSL generation?

Thanks! Appreciate your insight.

[dblock]

@dblock Hey dB, good to see you.

:)

Question is: How, if at all, has the Query DSL diverged since OpenSearch forked? Should it be safe to use queries generated and tested against ES in OpenSearch applications? How are other language communities handling complex Query DSL generation?

It is safe to use queries generated and tested against ES 7.10.2, which is the time of the fork. Since then, OpenSearch has followed semver, so the entire 1.x line is 100% safe, but for 2.x there may be changes (see https://opensearch.org/docs/2.0/breaking-changes/).

I can't speak to comparing to any newer version of Elastic > 7.10.2.

In general, I recommend forking libraries the same way the servers forked and treat Elasticsearch and OpenSearch as diverging products since a common point in time of 7.10.2.

[notque]

The deprecated olivere Go client had a pretty robust Go DSL for generating complex ES Query DSL JSON in a type safe manner. eg: https://github.com/olivere/elastic/blob/b7de00ddfd128672eac82ef8f6e938bcbb6c2cd4/search_queries_range.go

The go-elasticsearch client have decided that this sort of thing is out of scope for a low level API client: elastic/go-elasticsearch#42

This is no longer true. elastic/go-elasticsearch#526

[tannerjones4075]

I would like to help with this issue. Is this still relevant?

[VachaShah]

Hi @tannerjones4075, yes absolutely! We would love to add any missing functionality in this client as long as it is supported by OpenSearch. Also, given the gap between the client and server, it would also be a good idea to fix the existing client generator from the API specs (#284) and use that to generate all the missing APIs reducing manual effort.

[sean-]

Thanks for the thorough breakdown of the gap when migrating from olivere/elastic. The project's direction settled on generating typed request/response structs from the OpenAPI spec (#284) rather than building a hand-written Query DSL fluent API, since that scales to the entire OpenSearch surface including plugins. The codegen has now landed (v5preview/opensearchapi/ via #804), so query bodies are typed all the way through *CommonQueryDSLQueryContainer etc.

Closing this as superseded - please give the v5preview package a try and open focused issues for anything specific that's still painful.