Skip to content

Introduce Redline Testing to OSB #793

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 28 commits into from
May 6, 2025

Conversation

OVI3D0
Copy link
Member

@OVI3D0 OVI3D0 commented Mar 19, 2025

Description

Redline testing enables OSB to determine the maximum request throughput a cluster can handle under increasing load, which can help with capacity planning and detecting regressions in OpenSearch clusters.

This PR introduces redline testing to OSB, and enables automatic scaling of clients until the cluster begins to error. It implements:

  • A new FeedbackActor to throttle active clients
  • Coordination enhancements in WorkerCoordinatorActor, Worker, and AsyncExecutor
  • New multiprocessing-based inter-process shared state and queue handling

Redline Testing Overview

If the --redline-test flag is passed with a timed test procedure, such as:

{
  "name": "timed-mode-test-procedure",
  "schedule": [
    {
       "operation": "keyword-terms",
       "warmup-time-period": {{ warmup_time | default(300) | tojson }},
       "time-period": {{ time_period | default(900) | tojson }},
       "target-throughput": {{ target_throughput | default(20) | tojson }},
       "clients": {{ search_clients | default(20) }}
    }
  ]
}

OSB will then:

  1. Spawns N clients (default: 1000, can be overridden via --redline-test=<int>)
  2. Creates a new FeedbackActor which controls which clients should run
  3. Uses shared dictionaries and queues (via Python multiprocessing) to enable real-time coordination:
    • Workers create and share client state maps with the WorkerCoordinatorActor
    • WorkerCoordinatorActor aggregates and forwards to FeedbackActor
    • FeedbackActor:
      • Scales up clients until request errors are detected
      • Reads from a shared error queue populated by clients
      • Pauses clients by updating their shared state
      • Enters a 30s sleep period when errors are detected

Shared State Format

The client state dict managed by the FeedbackActor looks like:

{
   worker-0: {client-0:<pause status>, client-1:<pause status>, ... }
   .
   .
   .
   worker-n: {...client-k-1:<pause status>, client-k:<pause status>}
}

Clients only send requests when their status is True.

At the end of the test, OSB will print out the maximum number of clients the cluster was able to reach without error. This value is updated every time the FeedbackActor is in a NEUTRAL state, meaning no recent errors.

Based on the RFC introduced recently

Issues Resolved

#790 #791 #792

Testing

  • New functionality includes testing

make it + make test


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@OVI3D0 OVI3D0 marked this pull request as ready for review March 19, 2025 20:37
@OVI3D0 OVI3D0 linked an issue Mar 20, 2025 that may be closed by this pull request
Copy link
Collaborator

@IanHoang IanHoang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some minor comments but overall LGTM

@OVI3D0 OVI3D0 force-pushed the feedback-actor branch 2 times, most recently from 83a605c to 9749d60 Compare March 25, 2025 21:12
except Exception as e:
self.logger.error("Error processing client states: %s", e)

def receiveMsg_StartFeedbackActor(self, msg, sender):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This message should only be called once all the workers have updated the client mapping.
We need to have some kind of validation done to confirm if all workers have updated the client mappings, send an ack back to calling actor and then that actor can start the feedback actor.

Copy link
Member Author

@OVI3D0 OVI3D0 Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, noting this - I'll add this in a follow up revision

except Exception as e:
self.logger.error("Error processing client states: %s", e)

def receiveMsg_StartFeedbackActor(self, msg, sender):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This actor should also call the handle_state, not receiveMsg_SharedClientStateMessage

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This message will start the wake up loop which in turn will begin calling handle_state

self.messageQueue.clear()
self.sleep_start_time = time.perf_counter()

def scale_up(self):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bringing up 1 client at a time will be time consuming, especially when spinning up 1000s of clients.
How about we bring up clients in steps of n or multiple of n and also keep a check on self.state, if it is set to scale down whenever an error message is received, we can just ignore the logic and do nothing? WDYT?

Copy link
Member Author

@OVI3D0 OVI3D0 Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're right, 1 client/sec is time consuming. For now, I changed it to 5/sec but can increase it if 10 or more if you think it should be faster to start.

I think the scale_up function should just take a number of clients as an argument and attempt to activate that many clients round-robin style until it hits the target goal. For now it's just linear but the logic of changing the number of clients in multiples or steps can be added in a separate function. What do you think of this:

We check the state every second and choose whether to scale up or down depending on the message queue and a couple other factors (whether we errored recently, or scaled up too recently). For future scale-up methods, we can keep another class attribute n that we manipulate in a separate function if we want to scale up in different ways e.g. exponentially or percentage based, and then call scale_up to activate that many clients. Does that make sense?

@OVI3D0
Copy link
Member Author

OVI3D0 commented Apr 3, 2025

During the development of this PR, we discovered a bug where the OSB benchmark would not complete due to Workers not reaching their designated joinpoints. This was caused by how clients handled the 'paused' state. Instead of progressing through their schedules, they would enter a loop where they slept (asyncio.sleep(1)) repeatedly.

As a result, many clients remained stuck which prevented the test from completing.

To fix this, we updated the logic so clients now continue executing as normal, but without sending requests when they are meant to be paused. This change allows the FeedbackActor to control the number of active clients sending requests to a cluster and throttle the load generation without interrupting the overall flow of the benchmark.

@rishabh6788
Copy link
Collaborator

@OVI3D0 It would be great if you could add some details around how clients are reporting error to feedback actor using a blocking queue, and how it is different from the method you first implemented.

@OVI3D0
Copy link
Member Author

OVI3D0 commented Apr 3, 2025

We also introduced the use of multiprocessing.Queue and multiprocessing.Lock to safely coordinate feedback messaging between multiple worker processes and the central FeedbackActor.

We decided to go with a shared Queue between the FeedbackActor and individual clients because Thespianpy Actors handle messaging in a single-threaded, synchronized fashion unless using an actor troupe. (see docs).

Because of the nature of load testing and the strong chance of hundreds or even thousands of clients failing simultaneously, the original approach would often cause significant lag due to the overwhelming volume of messages being sent synchronously.

With shared multiprocessing queue's and locks, individual clients can now simply enqueue failed request metadata to the shared queue without sending any messages, and the FeedbackActor can freely snoop through this queue at regular intervals. This has proven to be able to be far more scalable than the previous implementation in tests with thousands of active clients.

@@ -939,6 +947,7 @@ def configure_test(arg_parser, args, cfg):
opts.csv_to_list(args.load_worker_coordinator_hosts))
cfg.add(config.Scope.applicationOverride, "workload", "test.mode.enabled", args.test_mode)
cfg.add(config.Scope.applicationOverride, "workload", "load.test.clients", int(args.load_test_qps))
cfg.add(config.Scope.applicationOverride, "workload", "redline.test", int(args.redline_test))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This config should only be added when the appropriate flag is passed.

@@ -251,8 +251,6 @@ def ensure_symlink(source, link_name):
os.remove(link_name)
os.symlink(source, link_name)
logger.info("Updated symlink: %s -> %s", link_name, source)
else:
logger.info("Symlink already correct: %s -> %s", link_name, source)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why remove this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very spammy (will log 20+ times per benchmark) & does not provide much value to the user IMO

self.logger.info("Allocating worker [%d] on [%s] with [%d] clients.", worker_id, host, len(clients))
worker = self.target.create_client(host)

client_allocations = ClientAllocations()
for client_id in clients:
client_allocations.add(client_id, self.allocations[client_id])
self.clients_per_worker[client_id] = worker_id
self.target.start_worker(worker, worker_id, self.config, self.workload, client_allocations)
self.target.start_worker(worker, worker_id, self.config, self.workload, client_allocations, self.error_queue, self.queue_lock)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally don't want to modify any logic with respect to existing functionality of OSB. It would be better to add a check here, if load testing enabled then send it with additional parameters else keep the original.

self.total_client_count = 0
self.total_active_client_count = 0 # must be tracked for scaling up/down
self.sleep_start_time = time.perf_counter()
self.last_error_time = time.perf_counter() - FeedbackActor.POST_SCALEDOWN_SECONDS
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not able to wrap my head around this and below initialization. Why are we setting the timestamp to past, instead of initializing to 0?

Copy link
Member Author

@OVI3D0 OVI3D0 Apr 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The FeedbackActor only scales if there are no errors in the past 30 seconds, so by initializing the last error time to '30 seconds in the past' we can begin scaling up immediately

self.scale_down()
self.logger.info("Clients scaled down. Active clients: %d", self.total_active_client_count)
self.last_error_time = current_time

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: remove the extra lines between if-elif blocks.

self.logger.info("Clients scaled up. Active clients: %d", self.total_active_client_count)
self.state = FeedbackState.NEUTRAL

def scale_down(self, scale_down_percentage=0.10) -> None:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about rewriting this with below:

def scale_down(self, scale_down_percentage=0.10) -> None:
    try:
        clients_to_pause = int(self.total_active_client_count * scale_down_percentage)
        if clients_to_pause <= 0:
            self.logger.info("No clients to pause during scale down")
            return
            
        # Create a flattened list of (worker_id, client_id) tuples for all active clients
        all_active_clients = []
        for worker_id, client_states in self.shared_client_states.items():
            for client_id, status in client_states.items():
                if status:  # Only include active clients
                    all_active_clients.append((worker_id, client_id))
        
        # If we need to pause more clients than are active, adjust the count
        clients_to_pause = min(clients_to_pause, len(all_active_clients))
        
        # Select clients to pause - randomly sample for better distribution
        import random
        clients_to_pause_indices = random.sample(range(len(all_active_clients)), clients_to_pause)
        clients_to_pause_list = [all_active_clients[i] for i in clients_to_pause_indices]
        
        # Pause the selected clients in a single pass
        for worker_id, client_id in clients_to_pause_list:
            self.shared_client_states[worker_id][client_id] = False
            self.total_active_client_count -= 1
        
        self.logger.info("Scaling down complete. Paused %d clients", clients_to_pause)
    finally:
        self.state = FeedbackState.SLEEP
        self.clear_queue()
        self.sleep_start_time = self.last_scaleup_time = time.perf_counter()

Complexity reduced from O(n^2) to O(n) and on the second thought randomization provides better load distribution. You can ignore randomization point if you don't agree and this seems to be working as expected.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this looks good thank you. Will add this

@@ -1215,10 +1495,12 @@ def drive(self):
self.logger.info("Worker[%d] skips tasks at index [%d] because it has been asked to complete all "
"tasks until next join point.", self.worker_id, self.current_task_index)
else:
if self.config.opts("workload", "redline.test", mandatory=False):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not too sure about this, why each worker has to send this signal.
Ideally all the coordination between Worker actors and Feedback Actor should happen via WorkerCoordinatorActor. Can you look at any other ways to handle joinpoint condition?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same goes with disabling.
I'm sure there must be an overall status being tracked at WCA level to track when all the workers have synchronized to jointpoint, we can use that to signal enabling or disabling Feedback actor.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we can track the overall status but the issue comes when some workers reached the joinpoint and are waiting for others to catch up. I've come across workers waiting 30+ seconds at a joinpoint during testing and it will result in inaccurate results since some clients are 'unpaused' but are actually just waiting at a joinpoint

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't get the last part. My point is that WCA is already tracking if all the actors have reached joint-point here. Can we send a message from here to Feedback actor to stop?

OVI3D0 added 2 commits April 21, 2025 10:25
Signed-off-by: Michael Oviedo <[email protected]>
Signed-off-by: Michael Oviedo <[email protected]>
Copy link
Collaborator

@IanHoang IanHoang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some minor comments but LGTM. This is will be really useful @OVI3D0

OVI3D0 added 2 commits April 23, 2025 15:10
…ling the feedback actor

Signed-off-by: Michael Oviedo <[email protected]>
Signed-off-by: Michael Oviedo <[email protected]>
@@ -1091,6 +1342,10 @@ def receiveMsg_StartWorker(self, msg, sender):
self.client_allocations = msg.client_allocations
self.current_task_index = 0
self.cancel.clear()
self.feedback_actor = msg.feedback_actor
self.shared_states = msg.shared_states
self.error_queue = msg.error_queue
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this needs to be declared in init method.

self.feedback_actor = msg.feedback_actor
self.shared_states = msg.shared_states
self.error_queue = msg.error_queue
self.queue_lock = msg.queue_lock
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here.

@rishabh6788
Copy link
Collaborator

Looks good. Can you please address the nits and also add some test run data?
Some data around regular benchmark runs not being affected would also help in solidifying the confidence.

Signed-off-by: Michael Oviedo <[email protected]>
@OVI3D0
Copy link
Member Author

OVI3D0 commented May 6, 2025

Looks good. Can you please address the nits and also add some test run data? Some data around regular benchmark runs not being affected would also help in solidifying the confidence.

Here is a comparison of two regular benchmark runs, on versions of OSB without (baseline) and with (contender) my changes applied. These are both run against a 3 manager node, 3 data node cluster along with 1 client node running OS 2.15.0:

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------

|                                                        Metric |                                                 Task |    Baseline |   Contender |     Diff |    Unit |
|--------------------------------------------------------------:|-----------------------------------------------------:|------------:|------------:|---------:|--------:|
|                    Cumulative indexing time of primary shards |                                                      |     327.493 |     285.064 | -42.4291 |     min |
|             Min cumulative indexing time across primary shard |                                                      |           0 |           0 |        0 |     min |
|          Median cumulative indexing time across primary shard |                                                      | 0.000466667 | 0.000466667 |        0 |     min |
|             Max cumulative indexing time across primary shard |                                                      |     327.363 |     284.934 | -42.4291 |     min |
|           Cumulative indexing throttle time of primary shards |                                                      |           0 |           0 |        0 |     min |
|    Min cumulative indexing throttle time across primary shard |                                                      |           0 |           0 |        0 |     min |
| Median cumulative indexing throttle time across primary shard |                                                      |           0 |           0 |        0 |     min |
|    Max cumulative indexing throttle time across primary shard |                                                      |           0 |           0 |        0 |     min |
|                       Cumulative merge time of primary shards |                                                      |     231.181 |     195.274 | -35.9074 |     min |
|                      Cumulative merge count of primary shards |                                                      |         280 |         274 |       -6 |         |
|                Min cumulative merge time across primary shard |                                                      |           0 |           0 |        0 |     min |
|             Median cumulative merge time across primary shard |                                                      |           0 |           0 |        0 |     min |
|                Max cumulative merge time across primary shard |                                                      |     231.177 |     195.269 | -35.9074 |     min |
|              Cumulative merge throttle time of primary shards |                                                      |     115.239 |     99.6105 | -15.6283 |     min |
|       Min cumulative merge throttle time across primary shard |                                                      |           0 |           0 |        0 |     min |
|    Median cumulative merge throttle time across primary shard |                                                      |           0 |           0 |        0 |     min |
|       Max cumulative merge throttle time across primary shard |                                                      |     115.239 |     99.6105 | -15.6283 |     min |
|                     Cumulative refresh time of primary shards |                                                      |     2.26337 |     1.80228 | -0.46108 |     min |
|                    Cumulative refresh count of primary shards |                                                      |        1541 |        1757 |      216 |         |
|              Min cumulative refresh time across primary shard |                                                      |           0 |           0 |        0 |     min |
|           Median cumulative refresh time across primary shard |                                                      | 0.000416667 | 0.000416667 |        0 |     min |
|              Max cumulative refresh time across primary shard |                                                      |     2.21577 |     1.75465 | -0.46112 |     min |
|                       Cumulative flush time of primary shards |                                                      |     13.6816 |     11.5435 |  -2.1381 |     min |
|                      Cumulative flush count of primary shards |                                                      |         292 |         292 |        0 |         |
|                Min cumulative flush time across primary shard |                                                      |           0 |           0 |        0 |     min |
|             Median cumulative flush time across primary shard |                                                      |           0 |           0 |        0 |     min |
|                Max cumulative flush time across primary shard |                                                      |     13.6813 |     11.5432 |  -2.1381 |     min |
|                                       Total Young Gen GC time |                                                      |     163.549 |      156.96 |   -6.589 |       s |
|                                      Total Young Gen GC count |                                                      |       22255 |       22032 |     -223 |         |
|                                         Total Old Gen GC time |                                                      |           0 |           0 |        0 |       s |
|                                        Total Old Gen GC count |                                                      |           0 |           0 |        0 |         |
|                                                    Store size |                                                      |     51.8805 |     51.9118 |  0.03131 |      GB |
|                                                 Translog size |                                                      | 3.07336e-06 | 3.07336e-06 |        0 |      GB |
|                                        Heap used for segments |                                                      |           0 |           0 |        0 |      MB |
|                                      Heap used for doc values |                                                      |           0 |           0 |        0 |      MB |
|                                           Heap used for terms |                                                      |           0 |           0 |        0 |      MB |
|                                           Heap used for norms |                                                      |           0 |           0 |        0 |      MB |
|                                          Heap used for points |                                                      |           0 |           0 |        0 |      MB |
|                                   Heap used for stored fields |                                                      |           0 |           0 |        0 |      MB |
|                                                 Segment count |                                                      |          95 |         107 |       12 |         |
|                                                Min Throughput |                                         index-append |     15700.1 |       17192 |   1491.9 |  docs/s |
|                                               Mean Throughput |                                         index-append |     17198.7 |     18474.1 |  1275.42 |  docs/s |
|                                             Median Throughput |                                         index-append |     17160.9 |     18463.8 |     1303 |  docs/s |
|                                                Max Throughput |                                         index-append |     17725.2 |     18987.1 |  1261.95 |  docs/s |
|                                       50th percentile latency |                                         index-append |     204.113 |     189.872 | -14.2411 |      ms |
|                                       90th percentile latency |                                         index-append |     298.662 |     281.062 | -17.5999 |      ms |
|                                       99th percentile latency |                                         index-append |     598.887 |     570.126 | -28.7603 |      ms |
|                                     99.9th percentile latency |                                         index-append |     3032.07 |     2826.03 | -206.036 |      ms |
|                                    99.99th percentile latency |                                         index-append |     3829.68 |     3654.17 | -175.512 |      ms |
|                                      100th percentile latency |                                         index-append |     4764.51 |      5017.5 |  252.992 |      ms |
|                                  50th percentile service time |                                         index-append |     204.113 |     189.872 | -14.2411 |      ms |
|                                  90th percentile service time |                                         index-append |     298.662 |     281.062 | -17.5999 |      ms |
|                                  99th percentile service time |                                         index-append |     598.887 |     570.126 | -28.7603 |      ms |
|                                99.9th percentile service time |                                         index-append |     3032.07 |     2826.03 | -206.036 |      ms |
|                               99.99th percentile service time |                                         index-append |     3829.68 |     3654.17 | -175.512 |      ms |
|                                 100th percentile service time |                                         index-append |     4764.51 |      5017.5 |  252.992 |      ms |
|                                                    error rate |                                         index-append |           0 |           0 |        0 |       % |
|                                                Min Throughput |                             wait-until-merges-finish |  0.00104128 |  0.00241128 |  0.00137 |   ops/s |
|                                               Mean Throughput |                             wait-until-merges-finish |  0.00104128 |  0.00241128 |  0.00137 |   ops/s |
|                                             Median Throughput |                             wait-until-merges-finish |  0.00104128 |  0.00241128 |  0.00137 |   ops/s |
|                                                Max Throughput |                             wait-until-merges-finish |  0.00104128 |  0.00241128 |  0.00137 |   ops/s |
|                                      100th percentile latency |                             wait-until-merges-finish |      960355 |      414717 |  -545638 |      ms |
|                                 100th percentile service time |                             wait-until-merges-finish |      960355 |      414717 |  -545638 |      ms |
|                                                    error rate |                             wait-until-merges-finish |           0 |           0 |        0 |       % |
|                                                Min Throughput |                                            match-all |     2.00647 |     2.00646 |   -1e-05 |   ops/s |
|                                               Mean Throughput |                                            match-all |     2.00785 |     2.00783 |   -3e-05 |   ops/s |
|                                             Median Throughput |                                            match-all |     2.00775 |     2.00772 |   -3e-05 |   ops/s |
|                                                Max Throughput |                                            match-all |     2.00964 |     2.00961 |   -3e-05 |   ops/s |
|                                       50th percentile latency |                                            match-all |     6.90635 |     5.97798 | -0.92837 |      ms |
|                                       90th percentile latency |                                            match-all |     7.54991 |     6.39414 | -1.15577 |      ms |
|                                       99th percentile latency |                                            match-all |     7.94064 |     6.92152 | -1.01912 |      ms |
|                                      100th percentile latency |                                            match-all |     11.5307 |     9.10249 | -2.42826 |      ms |
|                                  50th percentile service time |                                            match-all |     5.59462 |      4.7253 | -0.86932 |      ms |
|                                  90th percentile service time |                                            match-all |     6.15019 |     4.96084 | -1.18935 |      ms |
|                                  99th percentile service time |                                            match-all |     6.52219 |     5.77865 | -0.74354 |      ms |
|                                 100th percentile service time |                                            match-all |     10.1647 |       8.232 | -1.93266 |      ms |
|                                                    error rate |                                            match-all |           0 |           0 |        0 |       % |
|                                                Min Throughput |                                  desc_sort_timestamp |     2.00479 |     2.00524 |  0.00046 |   ops/s |
|                                               Mean Throughput |                                  desc_sort_timestamp |     2.00581 |     2.00636 |  0.00056 |   ops/s |
|                                             Median Throughput |                                  desc_sort_timestamp |     2.00573 |     2.00627 |  0.00055 |   ops/s |
|                                                Max Throughput |                                  desc_sort_timestamp |     2.00712 |     2.00782 |   0.0007 |   ops/s |
|                                       50th percentile latency |                                  desc_sort_timestamp |     9.93497 |     27.8069 |   17.872 |      ms |
|                                       90th percentile latency |                                  desc_sort_timestamp |     10.3409 |     28.4058 |  18.0649 |      ms |
|                                       99th percentile latency |                                  desc_sort_timestamp |     34.7952 |     50.6185 |  15.8233 |      ms |
|                                      100th percentile latency |                                  desc_sort_timestamp |     34.9924 |     50.6559 |  15.6636 |      ms |
|                                  50th percentile service time |                                  desc_sort_timestamp |     8.59073 |     25.6414 |  17.0507 |      ms |
|                                  90th percentile service time |                                  desc_sort_timestamp |     8.78237 |     26.4006 |  17.6182 |      ms |
|                                  99th percentile service time |                                  desc_sort_timestamp |     33.2956 |     48.2577 |  14.9621 |      ms |
|                                 100th percentile service time |                                  desc_sort_timestamp |     33.7858 |     48.3989 |   14.613 |      ms |
|                                                    error rate |                                  desc_sort_timestamp |           0 |           0 |        0 |       % |
|                                                Min Throughput |                                   asc_sort_timestamp |     2.00605 |     2.00551 | -0.00054 |   ops/s |
|                                               Mean Throughput |                                   asc_sort_timestamp |     2.00734 |     2.00669 | -0.00065 |   ops/s |
|                                             Median Throughput |                                   asc_sort_timestamp |     2.00724 |      2.0066 | -0.00064 |   ops/s |
|                                                Max Throughput |                                   asc_sort_timestamp |     2.00902 |     2.00822 |  -0.0008 |   ops/s |
|                                       50th percentile latency |                                   asc_sort_timestamp |      8.9088 |     19.2384 |  10.3296 |      ms |
|                                       90th percentile latency |                                   asc_sort_timestamp |     9.43386 |     19.8014 |  10.3675 |      ms |
|                                       99th percentile latency |                                   asc_sort_timestamp |     11.1157 |     21.7838 |   10.668 |      ms |
|                                      100th percentile latency |                                   asc_sort_timestamp |     23.2648 |      29.127 |  5.86217 |      ms |
|                                  50th percentile service time |                                   asc_sort_timestamp |     7.67032 |     18.1522 |  10.4819 |      ms |
|                                  90th percentile service time |                                   asc_sort_timestamp |     7.84789 |     18.5212 |  10.6733 |      ms |
|                                  99th percentile service time |                                   asc_sort_timestamp |     9.35313 |     20.3493 |  10.9962 |      ms |
|                                 100th percentile service time |                                   asc_sort_timestamp |     21.5351 |      27.582 |   6.0469 |      ms |
|                                                    error rate |                                   asc_sort_timestamp |           0 |           0 |        0 |       % |
|                                                Min Throughput |                       desc_sort_with_after_timestamp |     2.00477 |     2.00646 |  0.00169 |   ops/s |
|                                               Mean Throughput |                       desc_sort_with_after_timestamp |     2.00578 |     2.00783 |  0.00205 |   ops/s |
|                                             Median Throughput |                       desc_sort_with_after_timestamp |      2.0057 |     2.00772 |  0.00202 |   ops/s |
|                                                Max Throughput |                       desc_sort_with_after_timestamp |     2.00709 |     2.00961 |  0.00252 |   ops/s |
|                                       50th percentile latency |                       desc_sort_with_after_timestamp |     27.4889 |     8.16491 |  -19.324 |      ms |
|                                       90th percentile latency |                       desc_sort_with_after_timestamp |     28.8355 |     8.92175 | -19.9138 |      ms |
|                                       99th percentile latency |                       desc_sort_with_after_timestamp |     88.2601 |        37.3 | -50.9601 |      ms |
|                                      100th percentile latency |                       desc_sort_with_after_timestamp |     89.6783 |     37.9753 | -51.7029 |      ms |
|                                  50th percentile service time |                       desc_sort_with_after_timestamp |     25.1531 |     6.85841 | -18.2946 |      ms |
|                                  90th percentile service time |                       desc_sort_with_after_timestamp |     26.4055 |      7.6068 | -18.7987 |      ms |
|                                  99th percentile service time |                       desc_sort_with_after_timestamp |     86.8813 |     36.0307 | -50.8506 |      ms |
|                                 100th percentile service time |                       desc_sort_with_after_timestamp |     87.0169 |     36.3114 | -50.7056 |      ms |
|                                                    error rate |                       desc_sort_with_after_timestamp |           0 |           0 |        0 |       % |
|                                                Min Throughput |                        asc_sort_with_after_timestamp |     2.00651 |     2.00648 |   -3e-05 |   ops/s |
|                                               Mean Throughput |                        asc_sort_with_after_timestamp |      2.0079 |     2.00787 |   -3e-05 |   ops/s |
|                                             Median Throughput |                        asc_sort_with_after_timestamp |     2.00778 |     2.00776 |   -3e-05 |   ops/s |
|                                                Max Throughput |                        asc_sort_with_after_timestamp |      2.0097 |     2.00967 |   -3e-05 |   ops/s |
|                                       50th percentile latency |                        asc_sort_with_after_timestamp |     19.0894 |     11.9002 | -7.18922 |      ms |
|                                       90th percentile latency |                        asc_sort_with_after_timestamp |     20.1361 |      12.537 | -7.59913 |      ms |
|                                       99th percentile latency |                        asc_sort_with_after_timestamp |     23.2821 |     14.8224 | -8.45968 |      ms |
|                                      100th percentile latency |                        asc_sort_with_after_timestamp |     32.1961 |      15.692 | -16.5041 |      ms |
|                                  50th percentile service time |                        asc_sort_with_after_timestamp |     17.9704 |     10.9391 |  -7.0313 |      ms |
|                                  90th percentile service time |                        asc_sort_with_after_timestamp |     18.6158 |      11.108 | -7.50781 |      ms |
|                                  99th percentile service time |                        asc_sort_with_after_timestamp |     21.4766 |      14.148 | -7.32869 |      ms |
|                                 100th percentile service time |                        asc_sort_with_after_timestamp |     30.4942 |     14.3158 | -16.1784 |      ms |
|                                                    error rate |                        asc_sort_with_after_timestamp |           0 |           0 |        0 |       % |
|                                                Min Throughput |               desc_sort_timestamp_can_match_shortcut |       2.006 |     2.00589 | -0.00011 |   ops/s |
|                                               Mean Throughput |               desc_sort_timestamp_can_match_shortcut |     2.00729 |     2.00714 | -0.00015 |   ops/s |
|                                             Median Throughput |               desc_sort_timestamp_can_match_shortcut |     2.00719 |     2.00704 | -0.00015 |   ops/s |
|                                                Max Throughput |               desc_sort_timestamp_can_match_shortcut |     2.00894 |     2.00875 | -0.00018 |   ops/s |
|                                       50th percentile latency |               desc_sort_timestamp_can_match_shortcut |      19.137 |     22.7279 |  3.59084 |      ms |
|                                       90th percentile latency |               desc_sort_timestamp_can_match_shortcut |     28.3606 |     44.5265 |  16.1659 |      ms |
|                                       99th percentile latency |               desc_sort_timestamp_can_match_shortcut |     29.5411 |     47.3755 |  17.8344 |      ms |
|                                      100th percentile latency |               desc_sort_timestamp_can_match_shortcut |     35.8985 |     50.4508 |  14.5523 |      ms |
|                                  50th percentile service time |               desc_sort_timestamp_can_match_shortcut |     17.7214 |     20.5255 |  2.80405 |      ms |
|                                  90th percentile service time |               desc_sort_timestamp_can_match_shortcut |     26.8159 |     42.1688 |  15.3529 |      ms |
|                                  99th percentile service time |               desc_sort_timestamp_can_match_shortcut |     28.4243 |     45.2622 |  16.8379 |      ms |
|                                 100th percentile service time |               desc_sort_timestamp_can_match_shortcut |     34.4022 |     48.5698 |  14.1676 |      ms |
|                                                    error rate |               desc_sort_timestamp_can_match_shortcut |           0 |           0 |        0 |       % |
|                                                Min Throughput |            desc_sort_timestamp_no_can_match_shortcut |     2.00625 |     2.00632 |    7e-05 |   ops/s |
|                                               Mean Throughput |            desc_sort_timestamp_no_can_match_shortcut |     2.00759 |     2.00767 |    8e-05 |   ops/s |
|                                             Median Throughput |            desc_sort_timestamp_no_can_match_shortcut |     2.00749 |     2.00756 |    8e-05 |   ops/s |
|                                                Max Throughput |            desc_sort_timestamp_no_can_match_shortcut |     2.00932 |     2.00942 |  0.00011 |   ops/s |
|                                       50th percentile latency |            desc_sort_timestamp_no_can_match_shortcut |     18.9569 |     22.5665 |  3.60961 |      ms |
|                                       90th percentile latency |            desc_sort_timestamp_no_can_match_shortcut |     28.2452 |     44.1614 |  15.9162 |      ms |
|                                       99th percentile latency |            desc_sort_timestamp_no_can_match_shortcut |     29.3249 |     47.1751 |  17.8502 |      ms |
|                                      100th percentile latency |            desc_sort_timestamp_no_can_match_shortcut |      29.639 |     47.5596 |  17.9206 |      ms |
|                                  50th percentile service time |            desc_sort_timestamp_no_can_match_shortcut |     17.1651 |     20.4177 |   3.2526 |      ms |
|                                  90th percentile service time |            desc_sort_timestamp_no_can_match_shortcut |     26.7098 |      41.988 |  15.2782 |      ms |
|                                  99th percentile service time |            desc_sort_timestamp_no_can_match_shortcut |     27.8648 |     45.3076 |  17.4428 |      ms |
|                                 100th percentile service time |            desc_sort_timestamp_no_can_match_shortcut |     28.3799 |     46.0423 |  17.6624 |      ms |
|                                                    error rate |            desc_sort_timestamp_no_can_match_shortcut |           0 |           0 |        0 |       % |
|                                                Min Throughput |                asc_sort_timestamp_can_match_shortcut |     2.00629 |     2.00614 | -0.00015 |   ops/s |
|                                               Mean Throughput |                asc_sort_timestamp_can_match_shortcut |     2.00763 |     2.00745 | -0.00018 |   ops/s |
|                                             Median Throughput |                asc_sort_timestamp_can_match_shortcut |     2.00753 |     2.00735 | -0.00018 |   ops/s |
|                                                Max Throughput |                asc_sort_timestamp_can_match_shortcut |     2.00937 |     2.00914 | -0.00023 |   ops/s |
|                                       50th percentile latency |                asc_sort_timestamp_can_match_shortcut |     9.61173 |     10.9467 |  1.33493 |      ms |
|                                       90th percentile latency |                asc_sort_timestamp_can_match_shortcut |     10.0832 |     11.5073 |  1.42409 |      ms |
|                                       99th percentile latency |                asc_sort_timestamp_can_match_shortcut |     10.8525 |     11.7552 |  0.90274 |      ms |
|                                      100th percentile latency |                asc_sort_timestamp_can_match_shortcut |     21.1284 |     11.9311 | -9.19732 |      ms |
|                                  50th percentile service time |                asc_sort_timestamp_can_match_shortcut |     8.33643 |     9.89304 |  1.55661 |      ms |
|                                  90th percentile service time |                asc_sort_timestamp_can_match_shortcut |     8.51067 |     10.0662 |   1.5555 |      ms |
|                                  99th percentile service time |                asc_sort_timestamp_can_match_shortcut |        9.21 |     10.2511 |  1.04112 |      ms |
|                                 100th percentile service time |                asc_sort_timestamp_can_match_shortcut |     19.6434 |     10.3923 | -9.25109 |      ms |
|                                                    error rate |                asc_sort_timestamp_can_match_shortcut |           0 |           0 |        0 |       % |
|                                                Min Throughput |             asc_sort_timestamp_no_can_match_shortcut |      2.0065 |     2.00648 |   -1e-05 |   ops/s |
|                                               Mean Throughput |             asc_sort_timestamp_no_can_match_shortcut |     2.00788 |     2.00788 |       -0 |   ops/s |
|                                             Median Throughput |             asc_sort_timestamp_no_can_match_shortcut |     2.00778 |     2.00777 |       -0 |   ops/s |
|                                                Max Throughput |             asc_sort_timestamp_no_can_match_shortcut |     2.00967 |     2.00966 |   -1e-05 |   ops/s |
|                                       50th percentile latency |             asc_sort_timestamp_no_can_match_shortcut |     9.51664 |     10.7109 |  1.19421 |      ms |
|                                       90th percentile latency |             asc_sort_timestamp_no_can_match_shortcut |     10.0426 |     11.2264 |  1.18375 |      ms |
|                                       99th percentile latency |             asc_sort_timestamp_no_can_match_shortcut |     11.8466 |     11.5391 | -0.30751 |      ms |
|                                      100th percentile latency |             asc_sort_timestamp_no_can_match_shortcut |     15.5278 |     14.1849 | -1.34296 |      ms |
|                                  50th percentile service time |             asc_sort_timestamp_no_can_match_shortcut |     8.20411 |     9.59809 |  1.39398 |      ms |
|                                  90th percentile service time |             asc_sort_timestamp_no_can_match_shortcut |     8.39899 |     9.76011 |  1.36112 |      ms |
|                                  99th percentile service time |             asc_sort_timestamp_no_can_match_shortcut |     10.9723 |     9.90631 | -1.06602 |      ms |
|                                 100th percentile service time |             asc_sort_timestamp_no_can_match_shortcut |     14.3433 |      12.864 | -1.47928 |      ms |
|                                                    error rate |             asc_sort_timestamp_no_can_match_shortcut |           0 |           0 |        0 |       % |
|                                                Min Throughput |                                                 term |     2.00648 |     2.00651 |    3e-05 |   ops/s |
|                                               Mean Throughput |                                                 term |     2.00786 |      2.0079 |    3e-05 |   ops/s |
|                                             Median Throughput |                                                 term |     2.00775 |     2.00779 |    4e-05 |   ops/s |
|                                                Max Throughput |                                                 term |     2.00966 |     2.00971 |    4e-05 |   ops/s |
|                                       50th percentile latency |                                                 term |     4.65315 |     3.59582 | -1.05732 |      ms |
|                                       90th percentile latency |                                                 term |     5.09701 |     3.88821 |  -1.2088 |      ms |
|                                       99th percentile latency |                                                 term |     5.28096 |     4.01164 | -1.26932 |      ms |
|                                      100th percentile latency |                                                 term |     5.50326 |     4.13981 | -1.36345 |      ms |
|                                  50th percentile service time |                                                 term |     3.41356 |      2.2621 | -1.15147 |      ms |
|                                  90th percentile service time |                                                 term |       3.579 |     2.35841 | -1.22058 |      ms |
|                                  99th percentile service time |                                                 term |     3.74289 |     2.46439 |  -1.2785 |      ms |
|                                 100th percentile service time |                                                 term |     3.83636 |      2.5699 | -1.26646 |      ms |
|                                                    error rate |                                                 term |           0 |           0 |        0 |       % |
|                                                Min Throughput |                                  multi_terms-keyword |     1.02648 |     1.00695 | -0.01952 |   ops/s |
|                                               Mean Throughput |                                  multi_terms-keyword |     1.02689 |     1.00845 | -0.01843 |   ops/s |
|                                             Median Throughput |                                  multi_terms-keyword |     1.02686 |     1.00864 | -0.01822 |   ops/s |
|                                                Max Throughput |                                  multi_terms-keyword |      1.0279 |     1.00943 | -0.01847 |   ops/s |
|                                       50th percentile latency |                                  multi_terms-keyword |      118782 |      123293 |  4511.14 |      ms |
|                                       90th percentile latency |                                  multi_terms-keyword |      137469 |      142646 |  5176.32 |      ms |
|                                       99th percentile latency |                                  multi_terms-keyword |      141757 |      146898 |   5140.7 |      ms |
|                                      100th percentile latency |                                  multi_terms-keyword |      142162 |      147299 |  5137.62 |      ms |
|                                  50th percentile service time |                                  multi_terms-keyword |     968.814 |     980.768 |  11.9533 |      ms |
|                                  90th percentile service time |                                  multi_terms-keyword |     996.881 |     1014.66 |  17.7773 |      ms |
|                                  99th percentile service time |                                  multi_terms-keyword |     1038.07 |      1037.2 | -0.86394 |      ms |
|                                 100th percentile service time |                                  multi_terms-keyword |     1045.22 |     1052.58 |  7.35631 |      ms |
|                                                    error rate |                                  multi_terms-keyword |           0 |           0 |        0 |       % |
|                                                Min Throughput |                                        keyword-terms |     2.00296 |      2.0004 | -0.00256 |   ops/s |
|                                               Mean Throughput |                                        keyword-terms |     2.00359 |     2.00049 |  -0.0031 |   ops/s |
|                                             Median Throughput |                                        keyword-terms |     2.00355 |     2.00049 | -0.00306 |   ops/s |
|                                                Max Throughput |                                        keyword-terms |     2.00441 |     2.00061 |  -0.0038 |   ops/s |
|                                       50th percentile latency |                                        keyword-terms |     46.0508 |     51.8668 |    5.816 |      ms |
|                                       90th percentile latency |                                        keyword-terms |     54.3135 |     52.5763 | -1.73714 |      ms |
|                                       99th percentile latency |                                        keyword-terms |     55.6653 |      65.286 |  9.62071 |      ms |
|                                      100th percentile latency |                                        keyword-terms |     59.2984 |     65.7223 |  6.42392 |      ms |
|                                  50th percentile service time |                                        keyword-terms |     44.6876 |     50.6783 |  5.99079 |      ms |
|                                  90th percentile service time |                                        keyword-terms |     53.1303 |     51.3971 | -1.73322 |      ms |
|                                  99th percentile service time |                                        keyword-terms |     54.4629 |     63.7097 |  9.24678 |      ms |
|                                 100th percentile service time |                                        keyword-terms |     57.7777 |     64.9278 |  7.15009 |      ms |
|                                                    error rate |                                        keyword-terms |           0 |           0 |        0 |       % |
|                                                Min Throughput |                        keyword-terms-low-cardinality |     2.00593 |     2.00577 | -0.00015 |   ops/s |
|                                               Mean Throughput |                        keyword-terms-low-cardinality |     2.00719 |       2.007 | -0.00018 |   ops/s |
|                                             Median Throughput |                        keyword-terms-low-cardinality |     2.00709 |     2.00691 | -0.00018 |   ops/s |
|                                                Max Throughput |                        keyword-terms-low-cardinality |     2.00882 |     2.00859 | -0.00023 |   ops/s |
|                                       50th percentile latency |                        keyword-terms-low-cardinality |     43.8252 |     49.3089 |   5.4837 |      ms |
|                                       90th percentile latency |                        keyword-terms-low-cardinality |      44.417 |     49.8268 |  5.40985 |      ms |
|                                       99th percentile latency |                        keyword-terms-low-cardinality |     50.6637 |     61.2159 |  10.5522 |      ms |
|                                      100th percentile latency |                        keyword-terms-low-cardinality |     54.9163 |     62.1922 |  7.27594 |      ms |
|                                  50th percentile service time |                        keyword-terms-low-cardinality |     42.5007 |     48.1503 |  5.64962 |      ms |
|                                  90th percentile service time |                        keyword-terms-low-cardinality |     42.9352 |     48.4907 |  5.55554 |      ms |
|                                  99th percentile service time |                        keyword-terms-low-cardinality |     48.9076 |     60.2385 |  11.3309 |      ms |
|                                 100th percentile service time |                        keyword-terms-low-cardinality |     53.7301 |     60.5815 |  6.85143 |      ms |
|                                                    error rate |                        keyword-terms-low-cardinality |           0 |           0 |        0 |       % |
|                                                Min Throughput |                                      composite-terms |     2.00171 |     2.00058 | -0.00113 |   ops/s |
|                                               Mean Throughput |                                      composite-terms |     2.00207 |     2.00071 | -0.00137 |   ops/s |
|                                             Median Throughput |                                      composite-terms |     2.00204 |      2.0007 | -0.00134 |   ops/s |
|                                                Max Throughput |                                      composite-terms |     2.00256 |     2.00088 | -0.00168 |   ops/s |
|                                       50th percentile latency |                                      composite-terms |     286.421 |     282.296 | -4.12505 |      ms |
|                                       90th percentile latency |                                      composite-terms |     307.331 |     301.928 |  -5.4031 |      ms |
|                                       99th percentile latency |                                      composite-terms |     319.571 |     319.401 | -0.16957 |      ms |
|                                      100th percentile latency |                                      composite-terms |     322.345 |     324.082 |  1.73663 |      ms |
|                                  50th percentile service time |                                      composite-terms |      285.47 |     281.278 | -4.19186 |      ms |
|                                  90th percentile service time |                                      composite-terms |     306.455 |     301.271 | -5.18428 |      ms |
|                                  99th percentile service time |                                      composite-terms |     318.747 |     318.549 | -0.19794 |      ms |
|                                 100th percentile service time |                                      composite-terms |     321.366 |     323.153 |  1.78693 |      ms |
|                                                    error rate |                                      composite-terms |           0 |           0 |        0 |       % |
|                                                Min Throughput |                              composite_terms-keyword |      1.9984 |      2.0004 |    0.002 |   ops/s |
|                                               Mean Throughput |                              composite_terms-keyword |     1.99869 |      2.0005 |  0.00181 |   ops/s |
|                                             Median Throughput |                              composite_terms-keyword |      1.9987 |     2.00049 |   0.0018 |   ops/s |
|                                                Max Throughput |                              composite_terms-keyword |     1.99892 |     2.00062 |   0.0017 |   ops/s |
|                                       50th percentile latency |                              composite_terms-keyword |     462.832 |      408.61 | -54.2213 |      ms |
|                                       90th percentile latency |                              composite_terms-keyword |     481.797 |     434.851 | -46.9461 |      ms |
|                                       99th percentile latency |                              composite_terms-keyword |     495.705 |     452.953 | -42.7525 |      ms |
|                                      100th percentile latency |                              composite_terms-keyword |     513.105 |     456.283 | -56.8223 |      ms |
|                                  50th percentile service time |                              composite_terms-keyword |     462.219 |     407.902 | -54.3173 |      ms |
|                                  90th percentile service time |                              composite_terms-keyword |     481.003 |     433.764 | -47.2391 |      ms |
|                                  99th percentile service time |                              composite_terms-keyword |     495.285 |     452.238 | -43.0468 |      ms |
|                                 100th percentile service time |                              composite_terms-keyword |     512.258 |     455.304 | -56.9541 |      ms |
|                                                    error rate |                              composite_terms-keyword |           0 |           0 |        0 |       % |
|                                                Min Throughput |                       composite-date_histogram-daily |     2.00637 |     2.00629 |   -8e-05 |   ops/s |
|                                               Mean Throughput |                       composite-date_histogram-daily |     2.00773 |     2.00763 |  -0.0001 |   ops/s |
|                                             Median Throughput |                       composite-date_histogram-daily |     2.00763 |     2.00753 |  -0.0001 |   ops/s |
|                                                Max Throughput |                       composite-date_histogram-daily |     2.00948 |     2.00936 | -0.00012 |   ops/s |
|                                       50th percentile latency |                       composite-date_histogram-daily |      6.8643 |     6.08712 | -0.77718 |      ms |
|                                       90th percentile latency |                       composite-date_histogram-daily |     7.40268 |     6.62412 | -0.77855 |      ms |
|                                       99th percentile latency |                       composite-date_histogram-daily |     7.63457 |     7.71415 |  0.07958 |      ms |
|                                      100th percentile latency |                       composite-date_histogram-daily |     7.93924 |     8.72785 |  0.78862 |      ms |
|                                  50th percentile service time |                       composite-date_histogram-daily |     5.59531 |     4.95566 | -0.63965 |      ms |
|                                  90th percentile service time |                       composite-date_histogram-daily |      5.7903 |     5.22018 | -0.57012 |      ms |
|                                  99th percentile service time |                       composite-date_histogram-daily |     6.14282 |     6.19456 |  0.05174 |      ms |
|                                 100th percentile service time |                       composite-date_histogram-daily |     6.59407 |     7.46814 |  0.87407 |      ms |
|                                                    error rate |                       composite-date_histogram-daily |           0 |           0 |        0 |       % |
|                                                Min Throughput |                                                range |     2.00521 |     2.00486 | -0.00035 |   ops/s |
|                                               Mean Throughput |                                                range |     2.00632 |     2.00589 | -0.00042 |   ops/s |
|                                             Median Throughput |                                                range |     2.00623 |     2.00582 | -0.00042 |   ops/s |
|                                                Max Throughput |                                                range |     2.00777 |     2.00723 | -0.00054 |   ops/s |
|                                       50th percentile latency |                                                range |     74.1625 |     67.4108 | -6.75165 |      ms |
|                                       90th percentile latency |                                                range |     75.5708 |     69.0361 | -6.53476 |      ms |
|                                       99th percentile latency |                                                range |     107.548 |     80.7782 | -26.7696 |      ms |
|                                      100th percentile latency |                                                range |     118.225 |     122.435 |  4.20975 |      ms |
|                                  50th percentile service time |                                                range |     72.8754 |     66.2294 | -6.64605 |      ms |
|                                  90th percentile service time |                                                range |     74.1198 |      67.876 | -6.24376 |      ms |
|                                  99th percentile service time |                                                range |     106.372 |     80.0978 | -26.2742 |      ms |
|                                 100th percentile service time |                                                range |     117.239 |     121.669 |  4.43024 |      ms |
|                                                    error rate |                                                range |           0 |           0 |        0 |       % |
|                                                Min Throughput |                                        range-numeric |     2.00654 |     2.00654 |    1e-05 |   ops/s |
|                                               Mean Throughput |                                        range-numeric |     2.00794 |     2.00795 |    1e-05 |   ops/s |
|                                             Median Throughput |                                        range-numeric |     2.00783 |     2.00784 |    1e-05 |   ops/s |
|                                                Max Throughput |                                        range-numeric |     2.00975 |     2.00977 |    2e-05 |   ops/s |
|                                       50th percentile latency |                                        range-numeric |     4.56993 |     3.51917 | -1.05076 |      ms |
|                                       90th percentile latency |                                        range-numeric |     5.00235 |     3.96453 | -1.03782 |      ms |
|                                       99th percentile latency |                                        range-numeric |     13.5012 |     4.23516 |   -9.266 |      ms |
|                                      100th percentile latency |                                        range-numeric |     15.2928 |     4.26394 | -11.0289 |      ms |
|                                  50th percentile service time |                                        range-numeric |     3.27073 |     2.40373 | -0.86699 |      ms |
|                                  90th percentile service time |                                        range-numeric |     3.38382 |      2.5306 | -0.85322 |      ms |
|                                  99th percentile service time |                                        range-numeric |     12.0519 |     2.65231 | -9.39959 |      ms |
|                                 100th percentile service time |                                        range-numeric |     14.0988 |     2.65353 | -11.4452 |      ms |
|                                                    error rate |                                        range-numeric |           0 |           0 |        0 |       % |
|                                                Min Throughput |                                     keyword-in-range |     2.00069 |     1.99836 | -0.00233 |   ops/s |
|                                               Mean Throughput |                                     keyword-in-range |     2.00083 |     1.99867 | -0.00216 |   ops/s |
|                                             Median Throughput |                                     keyword-in-range |     2.00081 |     1.99869 | -0.00213 |   ops/s |
|                                                Max Throughput |                                     keyword-in-range |     2.00101 |     1.99891 |  -0.0021 |   ops/s |
|                                       50th percentile latency |                                     keyword-in-range |     116.255 |     117.343 |  1.08793 |      ms |
|                                       90th percentile latency |                                     keyword-in-range |      121.21 |     118.561 | -2.64851 |      ms |
|                                       99th percentile latency |                                     keyword-in-range |     153.873 |      145.86 | -8.01293 |      ms |
|                                      100th percentile latency |                                     keyword-in-range |     154.122 |     155.628 |  1.50526 |      ms |
|                                  50th percentile service time |                                     keyword-in-range |     114.983 |     116.263 |  1.28073 |      ms |
|                                  90th percentile service time |                                     keyword-in-range |     120.234 |      117.51 | -2.72307 |      ms |
|                                  99th percentile service time |                                     keyword-in-range |     152.946 |     144.595 | -8.35087 |      ms |
|                                 100th percentile service time |                                     keyword-in-range |     153.335 |     154.486 |  1.15089 |      ms |
|                                                    error rate |                                     keyword-in-range |           0 |           0 |        0 |       % |
|                                                Min Throughput |                            date_histogram_hourly_agg |     2.00575 |     2.00544 | -0.00031 |   ops/s |
|                                               Mean Throughput |                            date_histogram_hourly_agg |     2.00696 |     2.00659 | -0.00037 |   ops/s |
|                                             Median Throughput |                            date_histogram_hourly_agg |     2.00687 |      2.0065 | -0.00037 |   ops/s |
|                                                Max Throughput |                            date_histogram_hourly_agg |     2.00855 |     2.00808 | -0.00046 |   ops/s |
|                                       50th percentile latency |                            date_histogram_hourly_agg |     10.5818 |     10.4026 |  -0.1792 |      ms |
|                                       90th percentile latency |                            date_histogram_hourly_agg |     11.1333 |     10.7704 | -0.36292 |      ms |
|                                       99th percentile latency |                            date_histogram_hourly_agg |     11.4201 |     11.5292 |  0.10914 |      ms |
|                                      100th percentile latency |                            date_histogram_hourly_agg |     12.6467 |     12.1845 | -0.46219 |      ms |
|                                  50th percentile service time |                            date_histogram_hourly_agg |     9.27613 |      9.1334 | -0.14273 |      ms |
|                                  90th percentile service time |                            date_histogram_hourly_agg |     9.52584 |     9.34102 | -0.18482 |      ms |
|                                  99th percentile service time |                            date_histogram_hourly_agg |     10.5177 |      10.149 | -0.36877 |      ms |
|                                 100th percentile service time |                            date_histogram_hourly_agg |     11.2781 |     10.9412 | -0.33686 |      ms |
|                                                    error rate |                            date_histogram_hourly_agg |           0 |           0 |        0 |       % |
|                                                Min Throughput |                            date_histogram_minute_agg |     2.00526 |     2.00576 |   0.0005 |   ops/s |
|                                               Mean Throughput |                            date_histogram_minute_agg |     2.00638 |     2.00699 |  0.00061 |   ops/s |
|                                             Median Throughput |                            date_histogram_minute_agg |      2.0063 |      2.0069 |   0.0006 |   ops/s |
|                                                Max Throughput |                            date_histogram_minute_agg |     2.00784 |     2.00857 |  0.00073 |   ops/s |
|                                       50th percentile latency |                            date_histogram_minute_agg |     46.9873 |     53.3095 |   6.3222 |      ms |
|                                       90th percentile latency |                            date_histogram_minute_agg |      47.654 |     54.3311 |  6.67711 |      ms |
|                                       99th percentile latency |                            date_histogram_minute_agg |     48.0324 |     64.8381 |  16.8057 |      ms |
|                                      100th percentile latency |                            date_histogram_minute_agg |       48.62 |      65.193 |   16.573 |      ms |
|                                  50th percentile service time |                            date_histogram_minute_agg |     45.7347 |     52.1788 |   6.4441 |      ms |
|                                  90th percentile service time |                            date_histogram_minute_agg |      46.152 |     52.9883 |  6.83637 |      ms |
|                                  99th percentile service time |                            date_histogram_minute_agg |     46.7889 |     63.5783 |  16.7894 |      ms |
|                                 100th percentile service time |                            date_histogram_minute_agg |     47.4294 |     64.1343 |  16.7049 |      ms |
|                                                    error rate |                            date_histogram_minute_agg |           0 |           0 |        0 |       % |
|                                                Min Throughput |                                               scroll |     40.1273 |     42.2208 |  2.09351 | pages/s |
|                                               Mean Throughput |                                               scroll |     40.3449 |     42.3081 |  1.96316 | pages/s |
|                                             Median Throughput |                                               scroll |      40.338 |     42.2911 |  1.95304 | pages/s |
|                                                Max Throughput |                                               scroll |     40.6171 |     42.4139 |  1.79685 | pages/s |
|                                       50th percentile latency |                                               scroll |       29972 |       23195 | -6776.97 |      ms |
|                                       90th percentile latency |                                               scroll |     35927.1 |     26902.4 | -9024.65 |      ms |
|                                       99th percentile latency |                                               scroll |     36936.3 |     27640.3 | -9296.07 |      ms |
|                                      100th percentile latency |                                               scroll |     37102.5 |     27723.3 | -9379.22 |      ms |
|                                  50th percentile service time |                                               scroll |     639.573 |     578.035 | -61.5378 |      ms |
|                                  90th percentile service time |                                               scroll |     669.806 |     628.396 | -41.4101 |      ms |
|                                  99th percentile service time |                                               scroll |     714.051 |     645.204 | -68.8466 |      ms |
|                                 100th percentile service time |                                               scroll |     716.073 |     655.166 | -60.9068 |      ms |
|                                                    error rate |                                               scroll |           0 |           0 |        0 |       % |
|                                                Min Throughput |                              query-string-on-message |      2.0007 |     1.99145 | -0.00924 |   ops/s |
|                                               Mean Throughput |                              query-string-on-message |     2.00086 |     1.99302 | -0.00784 |   ops/s |
|                                             Median Throughput |                              query-string-on-message |     2.00085 |     1.99311 | -0.00773 |   ops/s |
|                                                Max Throughput |                              query-string-on-message |     2.00107 |     1.99422 | -0.00684 |   ops/s |
|                                       50th percentile latency |                              query-string-on-message |     154.767 |     145.265 | -9.50161 |      ms |
|                                       90th percentile latency |                              query-string-on-message |      157.99 |     147.587 | -10.4024 |      ms |
|                                       99th percentile latency |                              query-string-on-message |     160.749 |     155.709 | -5.03995 |      ms |
|                                      100th percentile latency |                              query-string-on-message |     161.652 |     156.178 | -5.47421 |      ms |
|                                  50th percentile service time |                              query-string-on-message |     153.151 |     144.096 | -9.05479 |      ms |
|                                  90th percentile service time |                              query-string-on-message |     156.712 |     146.239 | -10.4731 |      ms |
|                                  99th percentile service time |                              query-string-on-message |     159.826 |     153.859 | -5.96722 |      ms |
|                                 100th percentile service time |                              query-string-on-message |     160.244 |     154.315 | -5.92905 |      ms |
|                                                    error rate |                              query-string-on-message |           0 |           0 |        0 |       % |
|                                                Min Throughput |                     query-string-on-message-filtered |     2.00029 |     1.99621 | -0.00408 |   ops/s |
|                                               Mean Throughput |                     query-string-on-message-filtered |     2.00035 |     1.99691 | -0.00344 |   ops/s |
|                                             Median Throughput |                     query-string-on-message-filtered |     2.00035 |     1.99695 | -0.00339 |   ops/s |
|                                                Max Throughput |                     query-string-on-message-filtered |     2.00043 |     1.99745 | -0.00298 |   ops/s |
|                                       50th percentile latency |                     query-string-on-message-filtered |     137.918 |     145.191 |  7.27314 |      ms |
|                                       90th percentile latency |                     query-string-on-message-filtered |     149.202 |     148.892 | -0.31036 |      ms |
|                                       99th percentile latency |                     query-string-on-message-filtered |     199.469 |     202.322 |  2.85337 |      ms |
|                                      100th percentile latency |                     query-string-on-message-filtered |     278.644 |     204.054 | -74.5907 |      ms |
|                                  50th percentile service time |                     query-string-on-message-filtered |     136.813 |     144.062 |  7.24872 |      ms |
|                                  90th percentile service time |                     query-string-on-message-filtered |     148.037 |     148.167 |  0.12968 |      ms |
|                                  99th percentile service time |                     query-string-on-message-filtered |     198.434 |     201.609 |  3.17486 |      ms |
|                                 100th percentile service time |                     query-string-on-message-filtered |     277.297 |     202.858 | -74.4391 |      ms |
|                                                    error rate |                     query-string-on-message-filtered |           0 |           0 |        0 |       % |
|                                                Min Throughput |          query-string-on-message-filtered-sorted-num |     2.00295 |     2.00441 |  0.00146 |   ops/s |
|                                               Mean Throughput |          query-string-on-message-filtered-sorted-num |     2.00357 |     2.00534 |  0.00177 |   ops/s |
|                                             Median Throughput |          query-string-on-message-filtered-sorted-num |     2.00352 |     2.00527 |  0.00175 |   ops/s |
|                                                Max Throughput |          query-string-on-message-filtered-sorted-num |     2.00437 |     2.00655 |  0.00218 |   ops/s |
|                                       50th percentile latency |          query-string-on-message-filtered-sorted-num |     78.7812 |     63.5966 | -15.1847 |      ms |
|                                       90th percentile latency |          query-string-on-message-filtered-sorted-num |     81.1055 |     65.6221 | -15.4834 |      ms |
|                                       99th percentile latency |          query-string-on-message-filtered-sorted-num |     93.6382 |     82.2016 | -11.4366 |      ms |
|                                      100th percentile latency |          query-string-on-message-filtered-sorted-num |     95.8216 |     101.003 |  5.18104 |      ms |
|                                  50th percentile service time |          query-string-on-message-filtered-sorted-num |     77.4305 |      62.358 | -15.0725 |      ms |
|                                  90th percentile service time |          query-string-on-message-filtered-sorted-num |     79.5511 |     64.2129 | -15.3382 |      ms |
|                                  99th percentile service time |          query-string-on-message-filtered-sorted-num |     91.4846 |     81.5029 | -9.98166 |      ms |
|                                 100th percentile service time |          query-string-on-message-filtered-sorted-num |     94.3181 |     100.321 |  6.00334 |      ms |
|                                                    error rate |          query-string-on-message-filtered-sorted-num |           0 |           0 |        0 |       % |
|                                                Min Throughput |                      sort_keyword_can_match_shortcut |     2.00654 |     2.00648 |   -5e-05 |   ops/s |
|                                               Mean Throughput |                      sort_keyword_can_match_shortcut |     2.00793 |     2.00786 |   -7e-05 |   ops/s |
|                                             Median Throughput |                      sort_keyword_can_match_shortcut |     2.00782 |     2.00775 |   -6e-05 |   ops/s |
|                                                Max Throughput |                      sort_keyword_can_match_shortcut |     2.00974 |     2.00964 |   -9e-05 |   ops/s |
|                                       50th percentile latency |                      sort_keyword_can_match_shortcut |     5.85129 |     6.27685 |  0.42557 |      ms |
|                                       90th percentile latency |                      sort_keyword_can_match_shortcut |     6.30821 |      6.7053 |  0.39708 |      ms |
|                                       99th percentile latency |                      sort_keyword_can_match_shortcut |     6.57003 |     6.89994 |  0.32991 |      ms |
|                                      100th percentile latency |                      sort_keyword_can_match_shortcut |     7.67709 |     7.52343 | -0.15366 |      ms |
|                                  50th percentile service time |                      sort_keyword_can_match_shortcut |     4.57541 |     5.07661 |   0.5012 |      ms |
|                                  90th percentile service time |                      sort_keyword_can_match_shortcut |     4.70915 |     5.22282 |  0.51368 |      ms |
|                                  99th percentile service time |                      sort_keyword_can_match_shortcut |     5.05157 |      5.7468 |  0.69524 |      ms |
|                                 100th percentile service time |                      sort_keyword_can_match_shortcut |     6.08211 |     6.12222 |  0.04011 |      ms |
|                                                    error rate |                      sort_keyword_can_match_shortcut |           0 |           0 |        0 |       % |
|                                                Min Throughput |                   sort_keyword_no_can_match_shortcut |     2.00652 |     2.00655 |    4e-05 |   ops/s |
|                                               Mean Throughput |                   sort_keyword_no_can_match_shortcut |     2.00791 |     2.00795 |    4e-05 |   ops/s |
|                                             Median Throughput |                   sort_keyword_no_can_match_shortcut |      2.0078 |     2.00784 |    4e-05 |   ops/s |
|                                                Max Throughput |                   sort_keyword_no_can_match_shortcut |     2.00972 |     2.00977 |    5e-05 |   ops/s |
|                                       50th percentile latency |                   sort_keyword_no_can_match_shortcut |     6.56986 |     5.08125 | -1.48861 |      ms |
|                                       90th percentile latency |                   sort_keyword_no_can_match_shortcut |     7.05452 |     5.53342 |  -1.5211 |      ms |
|                                       99th percentile latency |                   sort_keyword_no_can_match_shortcut |     7.23616 |     5.76575 | -1.47042 |      ms |
|                                      100th percentile latency |                   sort_keyword_no_can_match_shortcut |     7.25843 |     7.70541 |  0.44699 |      ms |
|                                  50th percentile service time |                   sort_keyword_no_can_match_shortcut |     5.33258 |     3.91523 | -1.41735 |      ms |
|                                  90th percentile service time |                   sort_keyword_no_can_match_shortcut |     5.43898 |     4.02948 | -1.40949 |      ms |
|                                  99th percentile service time |                   sort_keyword_no_can_match_shortcut |     5.54257 |     4.15308 | -1.38949 |      ms |
|                                 100th percentile service time |                   sort_keyword_no_can_match_shortcut |     5.73961 |     6.77528 |  1.03568 |      ms |
|                                                    error rate |                   sort_keyword_no_can_match_shortcut |           0 |           0 |        0 |       % |
|                                                Min Throughput |                                    sort_numeric_desc |     2.00645 |     2.00639 |   -6e-05 |   ops/s |
|                                               Mean Throughput |                                    sort_numeric_desc |     2.00782 |     2.00775 |   -7e-05 |   ops/s |
|                                             Median Throughput |                                    sort_numeric_desc |     2.00771 |     2.00764 |   -7e-05 |   ops/s |
|                                                Max Throughput |                                    sort_numeric_desc |     2.00959 |     2.00951 |   -8e-05 |   ops/s |
|                                       50th percentile latency |                                    sort_numeric_desc |     8.78895 |     13.1559 |  4.36698 |      ms |
|                                       90th percentile latency |                                    sort_numeric_desc |     9.31399 |     14.0375 |  4.72348 |      ms |
|                                       99th percentile latency |                                    sort_numeric_desc |     9.51438 |     17.6634 |  8.14906 |      ms |
|                                      100th percentile latency |                                    sort_numeric_desc |     12.0941 |     23.5385 |  11.4444 |      ms |
|                                  50th percentile service time |                                    sort_numeric_desc |     7.49293 |     11.8382 |  4.34525 |      ms |
|                                  90th percentile service time |                                    sort_numeric_desc |     7.65611 |      13.111 |  5.45487 |      ms |
|                                  99th percentile service time |                                    sort_numeric_desc |       8.242 |     16.1979 |  7.95588 |      ms |
|                                 100th percentile service time |                                    sort_numeric_desc |     11.0616 |     22.0392 |  10.9776 |      ms |
|                                                    error rate |                                    sort_numeric_desc |           0 |           0 |        0 |       % |
|                                                Min Throughput |                                     sort_numeric_asc |     2.00652 |     2.00644 |   -8e-05 |   ops/s |
|                                               Mean Throughput |                                     sort_numeric_asc |     2.00791 |      2.0078 |  -0.0001 |   ops/s |
|                                             Median Throughput |                                     sort_numeric_asc |      2.0078 |     2.00769 | -0.00011 |   ops/s |
|                                                Max Throughput |                                     sort_numeric_asc |     2.00972 |     2.00958 | -0.00013 |   ops/s |
|                                       50th percentile latency |                                     sort_numeric_asc |     6.76774 |     13.3333 |  6.56558 |      ms |
|                                       90th percentile latency |                                     sort_numeric_asc |     7.22915 |     13.7602 |  6.53103 |      ms |
|                                       99th percentile latency |                                     sort_numeric_asc |     8.90597 |      14.463 |  5.55703 |      ms |
|                                      100th percentile latency |                                     sort_numeric_asc |     9.21168 |     16.7588 |  7.54709 |      ms |
|                                  50th percentile service time |                                     sort_numeric_asc |      5.4432 |     12.0291 |  6.58591 |      ms |
|                                  90th percentile service time |                                     sort_numeric_asc |     5.61183 |     12.2799 |  6.66806 |      ms |
|                                  99th percentile service time |                                     sort_numeric_asc |     7.57704 |     13.2973 |  5.72025 |      ms |
|                                 100th percentile service time |                                     sort_numeric_asc |     7.63648 |     15.5302 |  7.89373 |      ms |
|                                                    error rate |                                     sort_numeric_asc |           0 |           0 |        0 |       % |
|                                                Min Throughput |                         sort_numeric_desc_with_match |     2.00654 |     2.00655 |    1e-05 |   ops/s |
|                                               Mean Throughput |                         sort_numeric_desc_with_match |     2.00794 |     2.00795 |    1e-05 |   ops/s |
|                                             Median Throughput |                         sort_numeric_desc_with_match |     2.00783 |     2.00784 |    1e-05 |   ops/s |
|                                                Max Throughput |                         sort_numeric_desc_with_match |     2.00975 |     2.00976 |    1e-05 |   ops/s |
|                                       50th percentile latency |                         sort_numeric_desc_with_match |     4.81006 |     3.64355 | -1.16651 |      ms |
|                                       90th percentile latency |                         sort_numeric_desc_with_match |     5.29017 |      4.1216 | -1.16857 |      ms |
|                                       99th percentile latency |                         sort_numeric_desc_with_match |     5.92235 |     4.56216 | -1.36018 |      ms |
|                                      100th percentile latency |                         sort_numeric_desc_with_match |     9.00051 |      16.449 |  7.44849 |      ms |
|                                  50th percentile service time |                         sort_numeric_desc_with_match |     3.49667 |      2.3662 | -1.13046 |      ms |
|                                  90th percentile service time |                         sort_numeric_desc_with_match |     3.59257 |     2.51027 |  -1.0823 |      ms |
|                                  99th percentile service time |                         sort_numeric_desc_with_match |     3.97181 |     2.69154 | -1.28027 |      ms |
|                                 100th percentile service time |                         sort_numeric_desc_with_match |      7.3879 |     14.8041 |  7.41625 |      ms |
|                                                    error rate |                         sort_numeric_desc_with_match |           0 |           0 |        0 |       % |
|                                                Min Throughput |                          sort_numeric_asc_with_match |     2.00657 |     2.00656 |   -1e-05 |   ops/s |
|                                               Mean Throughput |                          sort_numeric_asc_with_match |     2.00797 |     2.00796 |   -1e-05 |   ops/s |
|                                             Median Throughput |                          sort_numeric_asc_with_match |     2.00786 |     2.00785 |   -1e-05 |   ops/s |
|                                                Max Throughput |                          sort_numeric_asc_with_match |     2.00979 |     2.00978 |   -1e-05 |   ops/s |
|                                       50th percentile latency |                          sort_numeric_asc_with_match |      4.0095 |     4.79314 |  0.78364 |      ms |
|                                       90th percentile latency |                          sort_numeric_asc_with_match |     4.41529 |     5.21826 |  0.80298 |      ms |
|                                       99th percentile latency |                          sort_numeric_asc_with_match |     4.58576 |     5.40463 |  0.81887 |      ms |
|                                      100th percentile latency |                          sort_numeric_asc_with_match |     4.59795 |     6.17799 |  1.58004 |      ms |
|                                  50th percentile service time |                          sort_numeric_asc_with_match |     2.67835 |      3.5389 |  0.86055 |      ms |
|                                  90th percentile service time |                          sort_numeric_asc_with_match |     2.76778 |     3.64557 |  0.87778 |      ms |
|                                  99th percentile service time |                          sort_numeric_asc_with_match |     2.87767 |     3.82207 |   0.9444 |      ms |
|                                 100th percentile service time |                          sort_numeric_asc_with_match |     3.03275 |     4.91764 |  1.88489 |      ms |
|                                                    error rate |                          sort_numeric_asc_with_match |           0 |           0 |        0 |       % |
|                                                Min Throughput |     range_field_conjunction_big_range_big_term_query |     2.00656 |     2.00658 |    2e-05 |   ops/s |
|                                               Mean Throughput |     range_field_conjunction_big_range_big_term_query |     2.00796 |     2.00799 |    3e-05 |   ops/s |
|                                             Median Throughput |     range_field_conjunction_big_range_big_term_query |     2.00785 |     2.00788 |    3e-05 |   ops/s |
|                                                Max Throughput |     range_field_conjunction_big_range_big_term_query |     2.00978 |     2.00981 |    4e-05 |   ops/s |
|                                       50th percentile latency |     range_field_conjunction_big_range_big_term_query |     4.05957 |     3.85688 | -0.20269 |      ms |
|                                       90th percentile latency |     range_field_conjunction_big_range_big_term_query |     4.55953 |     4.37875 | -0.18078 |      ms |
|                                       99th percentile latency |     range_field_conjunction_big_range_big_term_query |     5.80478 |     5.09826 | -0.70652 |      ms |
|                                      100th percentile latency |     range_field_conjunction_big_range_big_term_query |     6.30258 |     5.22358 |   -1.079 |      ms |
|                                  50th percentile service time |     range_field_conjunction_big_range_big_term_query |     2.79608 |     2.56754 | -0.22854 |      ms |
|                                  90th percentile service time |     range_field_conjunction_big_range_big_term_query |     2.90686 |     2.70539 | -0.20147 |      ms |
|                                  99th percentile service time |     range_field_conjunction_big_range_big_term_query |     4.51298 |     2.77486 | -1.73813 |      ms |
|                                 100th percentile service time |     range_field_conjunction_big_range_big_term_query |     5.24652 |     3.18574 | -2.06078 |      ms |
|                                                    error rate |     range_field_conjunction_big_range_big_term_query |           0 |           0 |        0 |       % |
|                                                Min Throughput |   range_field_disjunction_big_range_small_term_query |     2.00653 |     2.00654 |    1e-05 |   ops/s |
|                                               Mean Throughput |   range_field_disjunction_big_range_small_term_query |     2.00793 |     2.00794 |    1e-05 |   ops/s |
|                                             Median Throughput |   range_field_disjunction_big_range_small_term_query |     2.00782 |     2.00783 |        0 |   ops/s |
|                                                Max Throughput |   range_field_disjunction_big_range_small_term_query |     2.00974 |     2.00975 |    1e-05 |   ops/s |
|                                       50th percentile latency |   range_field_disjunction_big_range_small_term_query |     5.08775 |     4.27342 | -0.81433 |      ms |
|                                       90th percentile latency |   range_field_disjunction_big_range_small_term_query |     5.48079 |     4.77253 | -0.70827 |      ms |
|                                       99th percentile latency |   range_field_disjunction_big_range_small_term_query |     6.66685 |      4.9874 | -1.67945 |      ms |
|                                      100th percentile latency |   range_field_disjunction_big_range_small_term_query |     10.5237 |     5.06383 | -5.45991 |      ms |
|                                  50th percentile service time |   range_field_disjunction_big_range_small_term_query |     3.64923 |     3.17462 | -0.47461 |      ms |
|                                  90th percentile service time |   range_field_disjunction_big_range_small_term_query |     3.82262 |     3.29246 | -0.53015 |      ms |
|                                  99th percentile service time |   range_field_disjunction_big_range_small_term_query |     5.17776 |     3.40157 | -1.77618 |      ms |
|                                 100th percentile service time |   range_field_disjunction_big_range_small_term_query |     9.29261 |     3.42372 | -5.86889 |      ms |
|                                                    error rate |   range_field_disjunction_big_range_small_term_query |           0 |           0 |        0 |       % |
|                                                Min Throughput | range_field_conjunction_small_range_small_term_query |     2.00657 |     2.00658 |    1e-05 |   ops/s |
|                                               Mean Throughput | range_field_conjunction_small_range_small_term_query |     2.00797 |     2.00798 |    1e-05 |   ops/s |
|                                             Median Throughput | range_field_conjunction_small_range_small_term_query |     2.00786 |     2.00787 |    1e-05 |   ops/s |
|                                                Max Throughput | range_field_conjunction_small_range_small_term_query |     2.00979 |      2.0098 |        0 |   ops/s |
|                                       50th percentile latency | range_field_conjunction_small_range_small_term_query |     4.14603 |     3.51216 | -0.63387 |      ms |
|                                       90th percentile latency | range_field_conjunction_small_range_small_term_query |     4.62804 |     3.99569 | -0.63235 |      ms |
|                                       99th percentile latency | range_field_conjunction_small_range_small_term_query |     4.82334 |     4.09423 | -0.72911 |      ms |
|                                      100th percentile latency | range_field_conjunction_small_range_small_term_query |     6.27925 |     4.17723 | -2.10202 |      ms |
|                                  50th percentile service time | range_field_conjunction_small_range_small_term_query |     2.84969 |      2.3711 | -0.47859 |      ms |
|                                  90th percentile service time | range_field_conjunction_small_range_small_term_query |     2.97538 |     2.53113 | -0.44425 |      ms |
|                                  99th percentile service time | range_field_conjunction_small_range_small_term_query |     3.11054 |     2.61512 | -0.49542 |      ms |
|                                 100th percentile service time | range_field_conjunction_small_range_small_term_query |     4.60499 |     2.61948 | -1.98551 |      ms |
|                                                    error rate | range_field_conjunction_small_range_small_term_query |           0 |           0 |        0 |       % |
|                                                Min Throughput |   range_field_conjunction_small_range_big_term_query |     2.00658 |     2.00659 |    1e-05 |   ops/s |
|                                               Mean Throughput |   range_field_conjunction_small_range_big_term_query |     2.00798 |     2.00799 |        0 |   ops/s |
|                                             Median Throughput |   range_field_conjunction_small_range_big_term_query |     2.00787 |     2.00788 |        0 |   ops/s |
|                                                Max Throughput |   range_field_conjunction_small_range_big_term_query |     2.00979 |      2.0098 |    1e-05 |   ops/s |
|                                       50th percentile latency |   range_field_conjunction_small_range_big_term_query |     3.86913 |      3.3387 | -0.53043 |      ms |
|                                       90th percentile latency |   range_field_conjunction_small_range_big_term_query |     4.32887 |     3.84037 | -0.48849 |      ms |
|                                       99th percentile latency |   range_field_conjunction_small_range_big_term_query |     4.82317 |     3.97678 | -0.84639 |      ms |
|                                      100th percentile latency |   range_field_conjunction_small_range_big_term_query |     4.85797 |     4.61816 | -0.23981 |      ms |
|                                  50th percentile service time |   range_field_conjunction_small_range_big_term_query |      2.5733 |     2.12446 | -0.44883 |      ms |
|                                  90th percentile service time |   range_field_conjunction_small_range_big_term_query |     2.72542 |     2.25348 | -0.47194 |      ms |
|                                  99th percentile service time |   range_field_conjunction_small_range_big_term_query |     3.19201 |      2.3907 | -0.80132 |      ms |
|                                 100th percentile service time |   range_field_conjunction_small_range_big_term_query |     3.51954 |     3.00388 | -0.51566 |      ms |
|                                                    error rate |   range_field_conjunction_small_range_big_term_query |           0 |           0 |        0 |       % |
|                                                Min Throughput |                                range-auto-date-histo |    0.102887 |    0.107158 |  0.00427 |   ops/s |
|                                               Mean Throughput |                                range-auto-date-histo |    0.103052 |    0.107248 |   0.0042 |   ops/s |
|                                             Median Throughput |                                range-auto-date-histo |    0.103047 |     0.10725 |   0.0042 |   ops/s |
|                                                Max Throughput |                                range-auto-date-histo |    0.103267 |    0.107309 |  0.00404 |   ops/s |
|                                       50th percentile latency |                                range-auto-date-histo | 2.30634e+06 | 2.21055e+06 |   -95791 |      ms |
|                                       90th percentile latency |                                range-auto-date-histo | 2.67458e+06 | 2.55839e+06 |  -116186 |      ms |
|                                       99th percentile latency |                                range-auto-date-histo | 2.75729e+06 | 2.63687e+06 |  -120427 |      ms |
|                                      100th percentile latency |                                range-auto-date-histo | 2.76645e+06 | 2.64562e+06 |  -120827 |      ms |
|                                  50th percentile service time |                                range-auto-date-histo |     9778.87 |     9242.79 | -536.084 |      ms |
|                                  90th percentile service time |                                range-auto-date-histo |     9859.09 |     9403.39 | -455.704 |      ms |
|                                  99th percentile service time |                                range-auto-date-histo |     9970.24 |     9794.93 | -175.313 |      ms |
|                                 100th percentile service time |                                range-auto-date-histo |     10001.5 |      9833.9 |   -167.6 |      ms |
|                                                    error rate |                                range-auto-date-histo |           0 |           0 |        0 |       % |
|                                                Min Throughput |                   range-auto-date-histo-with-metrics |   0.0425224 |   0.0441037 |  0.00158 |   ops/s |
|                                               Mean Throughput |                   range-auto-date-histo-with-metrics |   0.0425527 |   0.0441417 |  0.00159 |   ops/s |
|                                             Median Throughput |                   range-auto-date-histo-with-metrics |     0.04256 |   0.0441425 |  0.00158 |   ops/s |
|                                                Max Throughput |                   range-auto-date-histo-with-metrics |   0.0425726 |   0.0441846 |  0.00161 |   ops/s |
|                                       50th percentile latency |                   range-auto-date-histo-with-metrics | 5.76044e+06 | 5.55032e+06 |  -210124 |      ms |
|                                       90th percentile latency |                   range-auto-date-histo-with-metrics | 6.67009e+06 | 6.42275e+06 |  -247344 |      ms |
|                                       99th percentile latency |                   range-auto-date-histo-with-metrics | 6.87403e+06 | 6.61853e+06 |  -255500 |      ms |
|                                      100th percentile latency |                   range-auto-date-histo-with-metrics | 6.89662e+06 | 6.64041e+06 |  -256213 |      ms |
|                                  50th percentile service time |                   range-auto-date-histo-with-metrics |     23414.1 |     22542.8 | -871.334 |      ms |
|                                  90th percentile service time |                   range-auto-date-histo-with-metrics |     23656.9 |     22723.4 | -933.528 |      ms |
|                                  99th percentile service time |                   range-auto-date-histo-with-metrics |       23763 |     22910.7 | -852.258 |      ms |
|                                 100th percentile service time |                   range-auto-date-histo-with-metrics |     23876.3 |       23001 | -875.313 |      ms |
|                                                    error rate |                   range-auto-date-histo-with-metrics |           0 |           0 |        0 |       % |
|                                                Min Throughput |                                          range-agg-1 |    0.210547 |    0.213634 |  0.00309 |   ops/s |
|                                               Mean Throughput |                                          range-agg-1 |    0.210827 |    0.213796 |  0.00297 |   ops/s |
|                                             Median Throughput |                                          range-agg-1 |    0.210878 |    0.213828 |  0.00295 |   ops/s |
|                                                Max Throughput |                                          range-agg-1 |    0.211024 |    0.213923 |   0.0029 |   ops/s |
|                                       50th percentile latency |                                          range-agg-1 | 1.06321e+06 | 1.04668e+06 |   -16538 |      ms |
|                                       90th percentile latency |                                          range-agg-1 | 1.23301e+06 | 1.21179e+06 | -21218.7 |      ms |
|                                       99th percentile latency |                                          range-agg-1 | 1.27069e+06 | 1.24878e+06 | -21902.6 |      ms |
|                                      100th percentile latency |                                          range-agg-1 | 1.27484e+06 | 1.25289e+06 | -21948.6 |      ms |
|                                  50th percentile service time |                                          range-agg-1 |     4720.18 |     4652.41 | -67.7692 |      ms |
|                                  90th percentile service time |                                          range-agg-1 |      4916.4 |     4723.55 | -192.853 |      ms |
|                                  99th percentile service time |                                          range-agg-1 |     5033.16 |     4870.96 | -162.198 |      ms |
|                                 100th percentile service time |                                          range-agg-1 |     5035.24 |        4879 | -156.236 |      ms |
|                                                    error rate |                                          range-agg-1 |           0 |           0 |        0 |       % |
|                                                Min Throughput |                                          range-agg-2 |    0.216421 |    0.221077 |  0.00466 |   ops/s |
|                                               Mean Throughput |                                          range-agg-2 |    0.216515 |     0.22114 |  0.00462 |   ops/s |
|                                             Median Throughput |                                          range-agg-2 |    0.216536 |    0.221142 |  0.00461 |   ops/s |
|                                                Max Throughput |                                          range-agg-2 |    0.216578 |     0.22121 |  0.00463 |   ops/s |
|                                       50th percentile latency |                                          range-agg-2 | 1.03206e+06 | 1.00802e+06 | -24040.6 |      ms |
|                                       90th percentile latency |                                          range-agg-2 |  1.1958e+06 | 1.16744e+06 | -28361.9 |      ms |
|                                       99th percentile latency |                                          range-agg-2 | 1.23256e+06 |  1.2031e+06 |   -29465 |      ms |
|                                      100th percentile latency |                                          range-agg-2 | 1.23668e+06 | 1.20705e+06 |   -29632 |      ms |
|                                  50th percentile service time |                                          range-agg-2 |     4596.74 |     4503.54 | -93.1973 |      ms |
|                                  90th percentile service time |                                          range-agg-2 |     4770.53 |     4609.55 |  -160.98 |      ms |
|                                  99th percentile service time |                                          range-agg-2 |     4820.35 |     4699.18 | -121.175 |      ms |
|                                 100th percentile service time |                                          range-agg-2 |     4822.81 |     4723.99 | -98.8164 |      ms |
|                                                    error rate |                                          range-agg-2 |           0 |           0 |        0 |       % |
|                                                Min Throughput |                                  cardinality-agg-low |     2.00611 |     2.00585 | -0.00026 |   ops/s |
|                                               Mean Throughput |                                  cardinality-agg-low |     2.00742 |     2.00709 | -0.00033 |   ops/s |
|                                             Median Throughput |                                  cardinality-agg-low |     2.00732 |     2.00699 | -0.00032 |   ops/s |
|                                                Max Throughput |                                  cardinality-agg-low |     2.00911 |     2.00872 | -0.00039 |   ops/s |
|                                       50th percentile latency |                                  cardinality-agg-low |     5.84269 |     7.07589 |   1.2332 |      ms |
|                                       90th percentile latency |                                  cardinality-agg-low |     6.34795 |       7.617 |  1.26905 |      ms |
|                                       99th percentile latency |                                  cardinality-agg-low |     8.55019 |     9.76583 |  1.21564 |      ms |
|                                      100th percentile latency |                                  cardinality-agg-low |     9.12569 |     10.6257 |  1.50002 |      ms |
|                                  50th percentile service time |                                  cardinality-agg-low |     4.51269 |     5.79186 |  1.27916 |      ms |
|                                  90th percentile service time |                                  cardinality-agg-low |      4.6841 |     6.11943 |  1.43533 |      ms |
|                                  99th percentile service time |                                  cardinality-agg-low |     7.05493 |     9.00834 |  1.95341 |      ms |
|                                 100th percentile service time |                                  cardinality-agg-low |     8.00781 |     9.34643 |  1.33862 |      ms |
|                                                    error rate |                                  cardinality-agg-low |           0 |           0 |        0 |       % |
|                                                Min Throughput |                                 cardinality-agg-high |   0.0172207 |   0.0170513 | -0.00017 |   ops/s |
|                                               Mean Throughput |                                 cardinality-agg-high |   0.0172346 |   0.0170831 | -0.00015 |   ops/s |
|                                             Median Throughput |                                 cardinality-agg-high |   0.0172321 |    0.017082 | -0.00015 |   ops/s |
|                                                Max Throughput |                                 cardinality-agg-high |   0.0172497 |   0.0171157 | -0.00013 |   ops/s |
|                                       50th percentile latency |                                 cardinality-agg-high | 1.44097e+07 | 1.45334e+07 |   123698 |      ms |
|                                       90th percentile latency |                                 cardinality-agg-high | 1.66727e+07 | 1.68098e+07 |   137064 |      ms |
|                                       99th percentile latency |                                 cardinality-agg-high | 1.71812e+07 | 1.73133e+07 |   132181 |      ms |
|                                      100th percentile latency |                                 cardinality-agg-high | 1.72372e+07 | 1.73727e+07 |   135521 |      ms |
|                                  50th percentile service time |                                 cardinality-agg-high |     57162.8 |     56928.2 | -234.606 |      ms |
|                                  90th percentile service time |                                 cardinality-agg-high |     59433.8 |     63489.7 |  4055.85 |      ms |
|                                  99th percentile service time |                                 cardinality-agg-high |     61383.6 |     66054.5 |  4670.93 |      ms |
|                                 100th percentile service time |                                 cardinality-agg-high |     61742.9 |     66714.7 |  4971.81 |      ms |
|                                                    error rate |                                 cardinality-agg-high |           0 |           0 |        0 |       % |


-------------------------------
[INFO] SUCCESS (took 0 seconds)
-------------------------------

OVI3D0 added 2 commits May 6, 2025 11:50
Signed-off-by: Michael Oviedo <[email protected]>
Signed-off-by: Michael Oviedo <[email protected]>
@OVI3D0 OVI3D0 force-pushed the feedback-actor branch from 5022c6c to 7302482 Compare May 6, 2025 19:13
@OVI3D0 OVI3D0 merged commit 2bd6e27 into opensearch-project:main May 6, 2025
10 checks passed
@OVI3D0 OVI3D0 deleted the feedback-actor branch May 6, 2025 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement new FeedbackActor for load testing in OSB
3 participants