[CI] UpgradeClusterClientYamlTestSuiteIT class failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #7080 / 8.0.1_bwc
• elasticsearch-periodic #7073 / 8.0.1_bwc

Reproduction Line:

./gradlew ":x-pack:qa:rolling-upgrade:v8.0.1#oldClusterTest" -Dtests.class="org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=old_cluster/80_transform_jobs_crud/Test put batch transform on old cluster}" -Dtests.seed=F5377982D8D243B5 -Dtests.bwc=true -Dtests.locale=brx-IN -Dtests.timezone=Africa/Lagos -Druntime.java=24

Applicable branches:
8.18

Reproduces locally?:
Yes

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Failure in test setup: Failed to initialize at least 3 watcher nodes

Issue Reasons:

• [8.18] 2 failures in class org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT (0.2% fail rate in 1000 executions)
• [8.18] 2 failures in step 8.0.1_bwc (100.0% fail rate in 2 executions)
• [8.18] 2 failures in pipeline elasticsearch-periodic (100.0% fail rate in 2 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[nicktindall]

I spent some time trying to understand this locally. The tests fail reliably locally for :x-pack:qa:rolling-upgrade:v8.0.1#oldClusterTest, including lots of logs about...

»  Caused by: java.lang.AssertionError: expected empty context but was {X-elastic-product-origin=fleet} on elasticsearch[v8.0.1-2][transport_worker][T#1]
»       at org.elasticsearch.transport.Transports.assertDefaultThreadContext(Transports.java:60) ~[elasticsearch-8.0.1.jar:8.0.1]
»       at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.write(Netty4MessageChannelHandler.java:91) ~[?:?]
»       at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) ~[?:?]

:x-pack:qa:rolling-upgrade:v7.17.29#oldClusterTest passes
:x-pack:qa:rolling-upgrade:v8.0.0#oldClusterTest fails (same assertion failures)
:x-pack:qa:rolling-upgrade:v8.1.0#oldClusterTest passes

I assessed it as low risk because the issue seems isolated to version 8.0.x, but could be convinced it's a medium given that it looks like a genuine BWC issue.

[nicktindall]

This looks like a bug in 8.0.1

When we stash the context (to enact the default context)

elasticsearch/server/src/main/java/org/elasticsearch/common/util/concurrent/ThreadContext.java

Lines 114 to 120 in 801d9cc

	if (HEADERS_TO_COPY.stream().anyMatch(header -> context.requestHeaders.containsKey(header))) {
	Map<String, String> map = headers(context, HEADERS_TO_COPY);
	ThreadContextStruct threadContextStruct = DEFAULT_CONTEXT.putHeaders(map);
	threadLocal.set(threadContextStruct);
	} else {
	threadLocal.set(DEFAULT_CONTEXT);
	}

We preserve the X-Elastic-Product-Origin header

elasticsearch/server/src/main/java/org/elasticsearch/tasks/Task.java

Lines 65 to 71 in db23674

	public static final Set<String> HEADERS_TO_COPY = Set.of(
	X_OPAQUE_ID_HTTP_HEADER,
	TRACE_PARENT_HTTP_HEADER,
	TRACE_ID,
	X_ELASTIC_PRODUCT_ORIGIN_HTTP_HEADER,
	X_ELASTIC_PROJECT_ID_HTTP_HEADER
	);

Then when asserting that we're on the default context, we blow if our headers contain X-Elastic-Product-Origin

elasticsearch/server/src/main/java/org/elasticsearch/transport/Transports.java

Lines 55 to 62 in 801d9cc

	public static boolean assertDefaultThreadContext(ThreadContext threadContext) {
	assert threadContext.getRequestHeadersOnly().isEmpty()
	|| threadContext.getRequestHeadersOnly().keySet().equals(Set.of(Task.X_OPAQUE_ID_HTTP_HEADER))
	|| threadContext.getRequestHeadersOnly().keySet().equals(Set.of(Task.TRACE_ID))
	|| threadContext.getRequestHeadersOnly().keySet().equals(Set.of(Task.X_OPAQUE_ID_HTTP_HEADER, Task.TRACE_ID))
	: "expected empty context but was " + threadContext.getRequestHeadersOnly() + " on " + Thread.currentThread().getName();
	return true;
	}

So my theory is in communication with an 8.18 node, we receive a request with an X-Elastic-Product-Origin header in it, which is allowed in, but then when we go to send the response, the assertion trips because of the presence of that header.

[nicktindall]

./gradlew ":x-pack:qa:rolling-upgrade:v8.0.1#oldClusterTest" -Dtests.class="org.elasticsearch.upgrades.SystemIndicesUpgradeIT" -Dtests.method="testUpgradeSystemIndexAndDataStream" -Dtests.bwc=true

Also fails on the 8.17 branch, but without the assertion failures around X-Elastic-Product-Origin

[nicktindall]

If you disable assertions in the test nodes, the test passes (not a fix, but proves that the assertions are the only issue here)

diff --git a/build-tools/src/main/java/org/elasticsearch/gradle/testclusters/ElasticsearchNode.java b/build-tools/src/main/java/org/elasticsearch/gradle/testclusters/ElasticsearchNode.java
index 705fda53025..9ca8a985797 100644
--- a/build-tools/src/main/java/org/elasticsearch/gradle/testclusters/ElasticsearchNode.java
+++ b/build-tools/src/main/java/org/elasticsearch/gradle/testclusters/ElasticsearchNode.java
@@ -837,7 +837,7 @@ public class ElasticsearchNode implements TestClusterConfiguration {
                 + heapSize
                 + " -Xmx"
                 + heapSize
-                + " -ea -esa "
+                + " -esa "
                 + systemPropertiesString
                 + " "
                 + featureFlagsString