[CI] ClusterDisruptionIT testAckedIndexing failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #16384 / part1
• elasticsearch-periodic-platform-support #5974 / amazonlinux-2_platform-support-aws
• elasticsearch-periodic-platform-support #5964 / ubuntu-1804_platform-support-unix
• elasticsearch-periodic-platform-support #5954 / ubuntu-2004-aarch64_checkpart1_platform-support-arm
• elasticsearch-periodic-platform-support #5909 / ubuntu-1804_platform-support-unix
• elasticsearch-periodic-platform-support #5894 / windows-2019_checkpart1_platform-support-windows

Reproduction Line:

./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.discovery.ClusterDisruptionIT.testAckedIndexing" -Dtests.seed=EDF9DADBCC6E8E91 -Dtests.locale=fr-FR -Dtests.timezone=Hongkong -Druntime.java=23

Applicable branches:
8.17

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: failed to reach a stable cluster of [3] nodes. Tried via [node_t1]. last cluster state:
cluster uuid: 2rBkmRA-T0OgIp9oB8INsg [committed: true]
version: 38
state uuid: fNCwt_18SYml_DXnkVy2sw
from_diff: false
meta data version: 28
   coordination_metadata:
      term: 1
      last_committed_config: VotingConfiguration{761zm456RSCKDzQSmAxExg,FIX6wQz1RNOcf4xs7FqYNw,ebMPS7hCRUyyjCy2mQgIOg}
      last_accepted_config: VotingConfiguration{761zm456RSCKDzQSmAxExg,FIX6wQz1RNOcf4xs7FqYNw,ebMPS7hCRUyyjCy2mQgIOg}
      voting tombstones: []
   [test/zMdrUNBQQuaJfaDbWY220Q]: v[24], mv[6], sv[1], av[1]
      0: p_term [2], isa_ids [BrkfOzvhR9GqFDBd4VMpHw, F3ICehhqQX-PvFvO-zTgQA]
      1: p_term [2], isa_ids [7RYa0FM-Sdua6pLS1sOKbA, 1Qo1WGe5Qi-lMC1c9bh2ZQ]
      2: p_term [2], isa_ids [r8tgAq2mSmKyGwPdCT12_A, EA3sn0XzQf69mrbI7LAHyw]
metadata customs:
   persistent_tasks: {"last_allocation_id":3,"tasks":[{"id":"health-node","task":{"health-node":{"params":{}}},"allocation_id":
[truncated]

Issue Reasons:

• [8.17] 6 failures in test testAckedIndexing (1.0% fail rate in 615 executions)
• [8.17] 2 failures in step ubuntu-1804_platform-support-unix (10.5% fail rate in 19 executions)
• [8.17] 5 failures in pipeline elasticsearch-periodic-platform-support (26.3% fail rate in 19 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testAckedIndexing (1.2% fail rate in 172 executions)

Build Scans:

• elasticsearch-periodic-platform-support #4919 / oraclelinux-7_platform-support-unix
• elasticsearch-intake #13071 / part1

[Tim-Brooks]

Concerning for [elasticsearch-periodic-platform-support #4919 / oraclelinux-7_platform-support-unix](https://gradle-enterprise.elastic.co/s/h3hx2chmhinck).

  2> java.lang.AssertionError: doc [14] exists [1] times in index
  2> 	at __randomizedtesting.SeedInfo.seed([7E3645A3E1F0C921]:0)
  2> 	at org.elasticsearch.index.engine.InternalEngine.assertDocDoesNotExist(InternalEngine.java:1586)
  2> 	at org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1415)
  2> 	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1219)
  2> 	at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1084)
  2> 	at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1010)
  2> 	at org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:2006)
  2> 	at org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1993)
  2> 	at org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$4(RecoveryTarget.java:454)
  2> 	at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:356)
  2> 	at org.elasticsearch.indices.recovery.RecoveryTarget.indexTranslogOperations(RecoveryTarget.java:429)
  2> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.performTranslogOps(PeerRecoveryTargetService.java:658)
  2> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.handleRequest(PeerRecoveryTargetService.java:605)
  2> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$TranslogOperationsRequestHandler.handleRequest(PeerRecoveryTargetService.java:597)
  2> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRequestHandler.messageReceived(PeerRecoveryTargetService.java:691)
  2> 	at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRequestHandler.messageReceived(PeerRecoveryTargetService.java:678)
  2> 	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:76)
  2> 	at org.elasticsearch.transport.InboundHandler.doHandleRequest(InboundHandler.java:289)
  2> 	at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:302)
  2> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)
  2> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
  2> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  2> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  2> 	at java.base/java.lang.Thread.run(Thread.java:1575)

Less concerning error for [elasticsearch-intake #13071 / part1](https://gradle-enterprise.elastic.co/s/ixl2x4hz6v6sk)

  2> REPRODUCE WITH: ./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.discovery.ClusterDisruptionIT.testAckedIndexing" -Dtests.seed=13417668B863C4D3 -Dtests.locale=bho-IN -Dtests.timezone=Asia/Dili -Druntime.java=23
  2> java.lang.AssertionError: org.elasticsearch.cluster.coordination.ValidateJoinRequest/unset vs org.elasticsearch.transport.BytesTransportRequest/unset
        at __randomizedtesting.SeedInfo.seed([13417668B863C4D3]:0)
        at org.elasticsearch.test.transport.MockTransportService$3.sendRequest(MockTransportService.java:589)
        at org.elasticsearch.test.transport.StubbableTransport$WrappedConnection.sendRequest(StubbableTransport.java:251)
        at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:982)
        at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:877)
        at org.elasticsearch.cluster.coordination.JoinValidationService$JoinValidation.doRun(JoinValidationService.java:322)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
        at org.elasticsearch.cluster.coordination.JoinValidationService.processNextItem(JoinValidationService.java:227)
        at org.elasticsearch.cluster.coordination.JoinValidationService$1.doRun(JoinValidationService.java:197)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1575)

[Tim-Brooks]

I unmuted this as I fixed one of the failure scenarios identified. We can mute again if it goes back to failing.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 failures in test testAckedIndexing (1.8% fail rate in 171 executions)

Build Scans:

• elasticsearch-periodic #4970 / openjdk23_checkpart1_java-matrix
• elasticsearch-periodic-platform-support #4919 / oraclelinux-7_platform-support-unix
• elasticsearch-intake #13071 / part1