[CI] CohereServiceMixedIT testCohereEmbeddings failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #22789 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #71018 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #71026 / 8.19.0_bwc-snapshots
• elasticsearch-intake #22787 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #71023 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #71021 / 8.19.0_bwc-snapshots
• elasticsearch-intake #22786 / 8.19.0_bwc-snapshots
• elasticsearch-intake #22785 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #71018 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #71016 / 8.19.0_bwc-snapshots

Reproduction Line:

./gradlew ":x-pack:plugin:inference:qa:mixed-cluster:v8.19.0#javaRestTest" -Dtests.class="org.elasticsearch.xpack.inference.qa.mixed.CohereServiceMixedIT" -Dtests.method="testCohereEmbeddings" -Dtests.seed=3944E0D378F145C9 -Dtests.bwc=true -Dtests.locale=ar-JO -Dtests.timezone=Asia/Novosibirsk -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:36815], URI [_inference/text_embedding/mixed-cluster-cohere-embeddings-float], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"Inference endpoint not found [mixed-cluster-cohere-embeddings-float]"}],"type":"resource_not_found_exception","reason":"Inference endpoint not found [mixed-cluster-cohere-embeddings-float]"},"status":404}

Issue Reasons:

• [main] 3 consecutive failures in test testCohereEmbeddings
• [main] 9 consecutive failures in step 8.19.0_bwc-snapshots
• [main] 2 consecutive failures in pipeline elasticsearch-pull-request
• [main] 10 failures in test testCohereEmbeddings (3.9% fail rate in 259 executions)
• [main] 10 failures in step 8.19.0_bwc-snapshots (8.1% fail rate in 123 executions)
• [main] 4 failures in pipeline elasticsearch-intake (12.9% fail rate in 31 executions)
• [main] 5 failures in pipeline elasticsearch-pull-request (5.3% fail rate in 94 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 consecutive failures in step 8.19.0_bwc-snapshots
• [main] 3 failures in test testCohereEmbeddings (1.2% fail rate in 256 executions)
• [main] 3 failures in step 8.19.0_bwc-snapshots (2.5% fail rate in 121 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (2.1% fail rate in 96 executions)

Build Scans:

• elasticsearch-intake #22785 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #71018 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #71016 / 8.19.0_bwc-snapshots

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[jonathan-buttner]

I think the cause of these is a transport version mismatch. It should be fixed here: https://github.com/elastic/elasticsearch/pull/127779/files#diff-85e782e9e33a0f8ca8e99b41c17f9d04e3a7981d435abf44a3aa5d954a47cd8fR220

After that is merged I'll do some testing and unmute the tests.