[Feature Request] Optimize indexing performance in replica shard

[kkewwei]

Is your feature request related to a problem? Please describe

It's known thatindexingStrategyForOperation will be invoked by both the primary and replica to ascertain the index strategy, which is very performance-intensive. Given that the data on primary and replica are same, the index strategy must be same between the two. Therefore, rather than the replica computing the strategy independently, index strategy can be passed from primary to replica directly, to avoid this performance-intensive action, thereby improving the index performance of the replica.

Related component

Indexing:Performance

Describe alternatives you've considered

No response

Additional context

No response

[soosinha]

[Triage - attendees 1 2 3 4]

We discussed in the triage meeting that during failover and internal retry scenarios, it might be useful for the replica to compute the indexing strategy independently. @kkewwei Have you thought about all the cases ?

[kkewwei]

We discussed in the triage meeting that during failover and internal retry scenarios, it might be useful for the replica to compute the indexing strategy independently. @kkewwei Have you thought about all the cases ?

@soosinha, I have't thought too much. In my side, It has a greater effect in replica indexing, of course including internal retry, It seems worthwhile to implement, and i like have a try in replica indexing first.

[navneet1v]

@kkewwei just out if curiosity how much latency improvement do you expect here if replicas are not creating their own index strategy?

[kkewwei]

@kkewwei just out if curiosity how much latency improvement do you expect here if replicas are not creating their own index strategy?

@navneet1v, I am not sure either. In the scenario of frequent updates, I see that writing thread is actively engaged here. When indexing, it needs to query all segments of this shard to determine whether the particular doc exists and its version. I will try to use opensearch-benchmark to draw some preliminary conclusion.

[kkewwei]

@soosinha, @navneet1v, I have done the opensearch-benchmark:

opensearch-benchmark execute-test --pipeline=benchmark-only --workload=nyc_taxis --target-host=http://x.x.x.x:9200 --workload-params='{"bulk_indexing_clients":10,"bulk_size":4000,"index_settings":{"index.number_of_shards":2,"index.number_of_replicas":1,"index.refresh_interval":"10s"}}'  --test-procedure="update"

Hardware used: 8c32g, 2 data nodes in the cluster with jvm 8c10g.
Baseline: origin code.
Contender: optimized code.

Metric	Task	Baseline	Contender	Diff	Unit
Cumulative indexing time of primary shards		298.659	317.68	19.0211	min
Min cumulative indexing time across primary shard		0	0	0	min
Median cumulative indexing time across primary shard		70.7694	75.667	4.89759	min
Max cumulative indexing time across primary shard		157.12	166.346	9.22597	min
Cumulative indexing throttle time of primary shards		0	0	0	min
Min cumulative indexing throttle time across primary shard		0	0	0	min
Median cumulative indexing throttle time across primary shard		0	0	0	min
Max cumulative indexing throttle time across primary shard		0	0	0	min
Cumulative merge time of primary shards		232.795	207.366	-25.4285	min
Cumulative merge count of primary shards		589	583	-6
Min cumulative merge time across primary shard		0	0	0	min
Median cumulative merge time across primary shard		58.0879	51.1831	-6.9048	min
Max cumulative merge time across primary shard		116.619	105	-11.6189	min
Cumulative merge throttle time of primary shards		141.295	115.546	-25.7495	min
Min cumulative merge throttle time across primary shard		0	0	0	min
Median cumulative merge throttle time across primary shard		35.2914	28.464	-6.82741	min
Max cumulative merge throttle time across primary shard		70.7123	58.6176	-12.0947	min
Cumulative refresh time of primary shards		19.6427	17.3909	-2.25177	min
Cumulative refresh count of primary shards		869	755	-114
Min cumulative refresh time across primary shard		0	0	0	min
Median cumulative refresh time across primary shard		4.87354	4.23804	-0.6355	min
Max cumulative refresh time across primary shard		9.8956	8.91483	-0.98077	min
Cumulative flush time of primary shards		9.92117	11.1762	1.25503	min
Cumulative flush count of primary shards		180	182	2
Min cumulative flush time across primary shard		0	0	0	min
Median cumulative flush time across primary shard		2.40599	2.63808	0.23208	min
Max cumulative flush time across primary shard		5.10918	5.90005	0.79087	min
Total Young Gen GC time		47.594	67.972	20.378	s
Total Young Gen GC count		1666	1791	125
Total Old Gen GC time		0	0	0	s
Total Old Gen GC count		0	0	0
Store size		68.2763	66.3591	-1.91721	GB
Translog size		4.09782e-07	4.09782e-07	0	GB
Heap used for segments		0	0	0	MB
Heap used for doc values		0	0	0	MB
Heap used for terms		0	0	0	MB
Heap used for norms		0	0	0	MB
Heap used for points		0	0	0	MB
Heap used for stored fields		0	0	0	MB
Segment count		77	64	-13
Min Throughput	update	45050.3	54965.1	9914.77	docs/s
Mean Throughput	update	48877.6	59086.8	10209.2	docs/s
Median Throughput	update	47820.6	58310.2	10489.5	docs/s
Max Throughput	update	54166.4	64297.3	10130.9	docs/s
50th percentile latency	update	804.643	606.368	-198.276	ms
90th percentile latency	update	1312.06	1150.17	-161.889	ms
99th percentile latency	update	2499.43	2613.97	114.54	ms
99.9th percentile latency	update	3605.38	3876.78	271.399	ms
99.99th percentile latency	update	4393.41	4896.36	502.942	ms
100th percentile latency	update	4571.75	5126.84	555.096	ms
50th percentile service time	update	804.643	606.368	-198.276	ms
90th percentile service time	update	1312.06	1150.17	-161.889	ms
99th percentile service time	update	2499.43	2613.97	114.54	ms
99.9th percentile service time	update	3605.38	3876.78	271.399	ms
99.99th percentile service time	update	4393.41	4896.36	502.942	ms
100th percentile service time	update	4571.75	5126.84	555.096	ms
error rate	update	0.00398835	0.00453988	0.00055	%
Min Throughput	wait-until-merges-finish	0.0163357	0.00266538	-0.01367	ops/s
Mean Throughput	wait-until-merges-finish	0.0163357	0.00266538	-0.01367	ops/s
Median Throughput	wait-until-merges-finish	0.0163357	0.00266538	-0.01367	ops/s
Max Throughput	wait-until-merges-finish	0.0163357	0.00266538	-0.01367	ops/s
100th percentile latency	wait-until-merges-finish	61215.2	375180	313965	ms
100th percentile service time	wait-until-merges-finish	61215.2	375180	313965	ms
error rate	wait-until-merges-finish	0	0	0	%

[kkewwei]

We can see that the Mean Throughput: 48877.6-> 59086.8(20%+). It seems worth-well to implement.

@andrross, I see segment replication may be the default setting in 3.0(#17162), please also help evaluate whether optimizing indexing in replica shard is needed.

[andrross]

We can see that the Mean Throughput: 48877.6-> 59086.8(20%+). It seems worth-well to implement.

@andrross, I see segment replication may be the default setting in 3.0(#17162), please also help evaluate whether optimizing indexing in replica shard is needed.

@kkewwei I think document replication is likely to be around for awhile in any case, so I wouldn't reject any performance optimizations. I see you closed the PR. Do you think this is still worth exploring?

[kkewwei]

@andrross I found that there are some consistency issues. The version consistency between the primary and replica cannot be guaranteed (for example: IndexLevelReplicationTests.testLateDeliveryAfterGCTriggeredOnReplica). Only eventual consistency can be guaranteed.

So I don't think this is still worth exploring in between the primary and the replica , I close the pr. I will explore in recovery phase as @soosinha mentioned.