Support modifying segmentInfos.counter in IndexWriter

[guojialiang92]

Description

Can we support modifying segmentInfos.counter in IndexWriter?
This can be used to skip some segment names when writing.
In the scenario of enabling segment replication in an Elasticsearch cluster, this control can be implemented during primary shard peer recovery and replica shard promotion to avoid segment counter conflicts.

Please help evaluate, if possible, I will submit a Pull Request.

[vigyasharma]

Hi @guojialiang92, Could you elaborate more on how you plan to use this capability? It's not immediately obvious why modifying segmentInfos.counter will help with peer recovery or segment replication conflicts.

[guojialiang92]

Hi @vigyasharma

I will explain the usage scenarios.
In the segment replication scenario, the file name and content of the replica and the primary shard will be consistent. Usually, the replica shard uses ReadOnlyEngine and does not hold IndexWriter. The primary shard uses ReadWriteEngine and uses IndexWriter for segment building.

When there is replica promotion or primary shard relocation, we need to increase the segment count of the new primary shard to the specified value according to the algorithm (such as the raft). We need to avoid the new primary shard generating files with the same name but different content as the old primary shard.

In the current situation, we will not be able to increase the segment counter when the primary shard is created.

In the replica promotion scenario, a current alternative is to add SegmentInfos#counter when the replica shard is closed, and execute SegmentInfos#commit to persist the file to disk. When the new primary shard initializes ReadWriteEngine, it reads the updated segment_N file from disk.

However, this alternative solution has some drawbacks. For example, when the replica exits abnormally, it cannot guarantee the successful update of the segment counter. At the same time, in the primary shard relocation scenario, it is impossible to modify the segment counter in InterWriter (you can only use ReadOnlyEngine like replica promotion, and then switch to ReadWriteEngine, which will cause serious write blocking).

Looking forward to your reply.

[vigyasharma]

Thanks @guojialiang92 . Is the plan here to support creating an IndexWriter with a supplied value of counter, say N, so that all it's commit generations are >=N i.e. segments_N, segments_N+1, ... and so on ?

To confirm my understanding, when a primary dies, one cannot really guarantee that all replicas were fully caught up. If the winning replica (new primary) was lagging behind, and you simply continue to use its SegmentInfos#counter, it might end up overwriting some segment files in other replicas. So you use raft to select the right counter value and start segments from that value.

This is specifically a problem for segment replication. Regular document replication works fine because each document is reindexed anyway and segment files are not copied over. Is that more or less correct?

Anyway, I don't see any problems with this support and it does have a valid use-case. If you want to raise a PR, I can help review it.

[guojialiang92]

Thanks @vigyasharma.

To confirm my understanding, when a primary dies, one cannot really guarantee that all replicas were fully caught up. If the winning replica (new primary) was lagging behind, and you simply continue to use its SegmentInfos#counter, it might end up overwriting some segment files in other replicas. So you use raft to select the right counter value and start segments from that value.

Your understanding is correct (This is specifically a problem for segment replication).

Is the plan here to support creating an IndexWriter with a supplied value of counter, say N, so that all it's commit generations are >=N i.e. segments_N, segments_N+1, ... and so on ?

This part may require some explanation.

1. SegmentInfos#generation affects the name of segment_N. Each shard can have different segment_N and ensure that the number N held by itself is monotonically increasing (For example, each commit operation uses the latest committed N + 1). Therefore, no additional modifications are needed for this part.
2. SegmentInfos#counter affects the name of the generated segment files. For example, suppose the current segments are _1.si, _2.si, ... , _100.si (each segment may contain various index files, such as .doc, .dvd, .fdt). When we change the segment counter to 1000, the subsequent created segment are _1000.si, _1001.si, ... and so on.

If the winning replica (new primary) was lagging behind, and using the advanced SegmentInfos#counter can effectively avoid generating segment names that are the same as other replicas.

From an implementation point of view, similar to the current IndexWriter#advanceSegmentInfosVersion, I want to provide a IndexWriter#advanceSegmentInfosCounter. It may be more flexible to use, do you think it can be changed like this?
I am willing to submit a PR and give it to you for review.

Looking forward to your reply.

[vigyasharma]

Sounds good to me. Will watch out for your PR.