Introduced explicit index resolution API

[nibix]

Description

This is a branch merge request to illustrate the changes needed for the improved index resolution #5367. It is not yet finished yet. It is dependent on changes in core. Thus, this PR does not even compile yet.

Please look first at the companion PR in core: opensearch-project/OpenSearch#18523

• Category
  • Enhancement
• Why these changes are required?
  • The current index resolution mechanism employed by the security plugin (with the class IndexResolverReplacer in the center) has several downsides: It is slow, it is fragile, it is messy. The goal of the changes in this and the related PR is to fix this.
• What is the old behavior before changes and new behavior after changes?

Issues Resolved

• [FEATURE] Index pattern resolution improvements #5367

Testing

Check List

• New functionality includes testing
• New functionality has been documented
• New Roles/Permissions have a corresponding security dashboards plugin PR
• API changes companion pull request created
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[nibix]

Preliminary benchmark results

We now have some preliminary benchmark results.

Disclaimer

The usual benchmarking disclaimer: Of course, these figures can only represent very specific scenarios. Other scenarios can look quite different, as there are so many variables which can be different. Yet, especially the "slow parts" of the benchmark results can give one an impression where real performance issues are.

Test setup

• We used a OpenSearch main snapshot for all tests
• All plugins were removed, except the security plugin
• Audit logging was disabled
• Authentication: Only the internal backend auth was active. All other authentication types were disabled.

K8s resources per node

• Java Heap: 32 GB
• RAM: 64 GB
• CPU: 8 cores
• Kubernetes version: v1.32.2

Tested dimensions

Users

We compared four different users:

• A simple user with normal privileges
• User authenticated by admin certificate

Indices on cluster

We executed the benchmarks with various numbers of indices on the cluster:

• 30 indices
• 100 indices
• 300 indices
• 1000 indices
• 3000 indices

Operations

We tested four different operations:

• Bulk indexing with 10 items per request
• Bulk indexing with 100 items per request
• Searching on single index
• Searching on 20 indices (using index pattern)

Test results

A commented summary of the results follows in the upcoming sections.

The raw test results can be also reviewed at https://docs.google.com/spreadsheets/d/17Vyolu2UZ0ZjDNySyb0uyPeQIPMw-LrHQ2bvd_dNqP0/edit?usp=sharing

Indexing throughput

Bulk size 10

In this and the following charts, the dashed lines represent OpenSearch with the standard security plugin. The solid lines represent OpenSearch with the security plugin with the optimized index resolution code.

The blue line represents requests authenticated by the super admin certificate, which by-passes most of the security plugin. Thus, the blue line forms a kind of "hull curve", it can be seen as a rough theoretical maximum from a security plugin point of view.

We see a consistent improvement of the indexing throughput (in docs/s) by 5 to 12%. Interestingly, a correlation to the number of indices on the cluster cannot be really observed (except for maybe the 3000 index case). This is different from my expectations. I have checked this with micro benchmarks; indeed, we get a runtime improvement in the security filter, which is however less than 0.5 ms and thus not really visible in these benchmarks.

We still have quite a gap to the admin certificate performance; this indicates that we could do more research on the performance impact of the security plugin.

Bulk size 100

Bulk requests with more items can be processed more efficiently. Thus, as we are looking at a docs/s throughput, the improvements are a bit smaller in this case. The improvement range is between 2% and 8%.

Search throughput

Single index

In this benchmark, we compare the operation throughput for search requests performed on a single index.

Plain and simple: For this case, we do not see an improvement. This is probably because search operations only rely on a single operation going through the security filter. Further messages are exchanged around the cluster in order to communicate with the shards; however, these are not ActionRequests. Thus, these are not affected by the security plugin. In addition, an index expression with a single index does not require any matching against the existing indices. This makes the whole process quite efficient up-front.

20% indices

This benchmark tests searches operations that search through 20% of the indices on the cluster at once (specified by the index pattern index_a*).

Here we do see an improvement of the throughput by about 2 to 4%.

@cwperks @kkhatua