YARN-11924. Add zkManager.exists(path) check to ZKConfigurationStore:… by ferdelyi · Pull Request #8222 · apache/hadoop

ferdelyi · 2026-01-30T15:34:41Z

…getZkData() and retry mechanism

Should a "yarn resourcemanager -format-state-store" command be issued while one of the RM is starting and in the INIT state (because of YARN-11551), there is a time period when the /confstore/CONF_STORE path does not exist, hence the getZkData method returns a null value, causing the RM to fail. To prevent this, add a check and re-try mechanism before giving up.

Description of PR

Rare race condition is addressed when "yarn resourcemanager -format-state-store" issued when an RM is in the INITING state (already initialized the confstore) right before reading it. This change avoids a null pointer exception.

How was this patch tested?

Manually with locks introduced in the RM at the confstore format step with sleep, so while one of the RM is formatting the statestore, the other RM will be at the getZkData method trying to read the confstore in the INIT state.
Also with added unit tests.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

If an AI tool was used:

The PR includes the phrase "Contains content generated by "
where is the name of the AI tool used.
My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html

hadoop-yetus · 2026-01-30T18:24:17Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 11s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	9m 6s		Maven dependency ordering for branch
+1 💚	mvninstall	48m 55s		trunk passed
+1 💚	compile	7m 6s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	compile	7m 1s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	checkstyle	1m 40s		trunk passed
+1 💚	mvnsite	3m 52s		trunk passed
+1 💚	javadoc	3m 21s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	3m 18s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	spotbugs	7m 46s		trunk passed
+1 💚	shadedclient	35m 1s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
-1 ❌	mvninstall	0m 27s	/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch failed.
-1 ❌	compile	2m 27s	/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	hadoop-yarn in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌	javac	2m 27s	/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	hadoop-yarn in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌	compile	2m 32s	/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	hadoop-yarn in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
-1 ❌	javac	2m 32s	/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	hadoop-yarn in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 47s	/buildtool-patch-checkstyle-hadoop-yarn-project_hadoop-yarn.txt	The patch fails to run checkstyle in hadoop-yarn
-1 ❌	mvnsite	0m 30s	/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch failed.
-1 ❌	javadoc	0m 34s	/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 1 new + 2317 unchanged - 0 fixed = 2318 total (was 2317)
-1 ❌	javadoc	0m 29s	/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌	javadoc	0m 34s	/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 generated 1 new + 2058 unchanged - 0 fixed = 2059 total (was 2058)
-1 ❌	javadoc	0m 28s	/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
-1 ❌	spotbugs	0m 29s	/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch failed.
-1 ❌	shadedclient	11m 51s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 57s		hadoop-yarn-api in the patch passed.
+1 💚	unit	5m 20s		hadoop-yarn-common in the patch passed.
-1 ❌	unit	0m 30s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch failed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		167m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.53 ServerAPI=1.53 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8222/1/artifact/out/Dockerfile
GITHUB PR	#8222
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 881d2c39d684 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `6901f70`
Default Java	Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8222/1/testReport/
Max. process+thread count	574 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8222/1/console
versions	git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

Hean-Chhinling

Big thanks to you, @ferdelyi for working on this.
This is a huge help to the intermittent RM start up failures.
The code overall is really good.

I just have some improvement ideas and some questions

.../apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java

...che/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestZKConfigurationStore.java

...hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java

...che/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestZKConfigurationStore.java

hadoop-yetus · 2026-01-30T21:49:55Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 12s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	9m 7s		Maven dependency ordering for branch
+1 💚	mvninstall	58m 42s		trunk passed
+1 💚	compile	6m 55s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	compile	7m 1s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	checkstyle	1m 41s		trunk passed
+1 💚	mvnsite	3m 54s		trunk passed
+1 💚	javadoc	3m 23s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	3m 20s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	spotbugs	7m 48s		trunk passed
+1 💚	shadedclient	34m 37s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
-1 ❌	mvninstall	0m 57s	/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch failed.
-1 ❌	compile	2m 46s	/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	hadoop-yarn in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌	javac	2m 46s	/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	hadoop-yarn in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04.
-1 ❌	compile	2m 50s	/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	hadoop-yarn in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
-1 ❌	javac	2m 50s	/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	hadoop-yarn in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 22s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt	hadoop-yarn-project/hadoop-yarn: The patch generated 8 new + 165 unchanged - 0 fixed = 173 total (was 165)
-1 ❌	mvnsite	1m 1s	/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch failed.
-1 ❌	javadoc	0m 35s	/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 1 new + 2317 unchanged - 0 fixed = 2318 total (was 2317)
-1 ❌	javadoc	0m 34s	/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 generated 1 new + 2058 unchanged - 0 fixed = 2059 total (was 2058)
-1 ❌	spotbugs	1m 0s	/patch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch failed.
-1 ❌	shadedclient	12m 38s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 58s		hadoop-yarn-api in the patch passed.
+1 💚	unit	5m 21s		hadoop-yarn-common in the patch passed.
-1 ❌	unit	1m 1s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch failed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		182m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.53 ServerAPI=1.53 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8222/2/artifact/out/Dockerfile
GITHUB PR	#8222
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux dd9984e21b0c 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `588a2c5`
Default Java	Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8222/2/testReport/
Max. process+thread count	609 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8222/2/console
versions	git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2026-01-31T04:38:38Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 7s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	8m 52s		Maven dependency ordering for branch
+1 💚	mvninstall	47m 1s		trunk passed
+1 💚	compile	7m 6s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	compile	7m 6s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	checkstyle	1m 47s		trunk passed
+1 💚	mvnsite	3m 54s		trunk passed
+1 💚	javadoc	3m 22s		trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javadoc	3m 18s		trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	spotbugs	7m 51s		trunk passed
+1 💚	shadedclient	35m 15s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 38s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 24s		the patch passed
+1 💚	compile	6m 20s		the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
+1 💚	javac	6m 20s		the patch passed
+1 💚	compile	6m 54s		the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
+1 💚	javac	6m 54s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 44s		the patch passed
+1 💚	mvnsite	3m 34s		the patch passed
-1 ❌	javadoc	0m 54s	/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt	hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 1 new + 2317 unchanged - 0 fixed = 2318 total (was 2317)
-1 ❌	javadoc	0m 55s	/results-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt	hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04 with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 generated 1 new + 2058 unchanged - 0 fixed = 2059 total (was 2058)
+1 💚	spotbugs	8m 7s		the patch passed
+1 💚	shadedclient	34m 57s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 16s		hadoop-yarn-api in the patch passed.
+1 💚	unit	5m 41s		hadoop-yarn-common in the patch passed.
-1 ❌	unit	119m 2s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	1m 6s		The patch does not generate ASF License warnings.
		327m 36s

Reason	Tests
Failed junit tests	hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation
	hadoop.yarn.server.resourcemanager.TestRMHA

Subsystem	Report/Notes
Docker	ClientAPI=1.53 ServerAPI=1.53 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8222/3/artifact/out/Dockerfile
GITHUB PR	#8222
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 21f04b617672 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `89d3f21`
Default Java	Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8222/3/testReport/
Max. process+thread count	894 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8222/3/console
versions	git=2.25.1 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

ferdelyi

Thank you for your review!

.../apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java

ferdelyi

Thank you for being so active on this PR.

…getZkData() and retry mechanism If the 'yarn resourcemanager -format-conf-store' command is issued while one of the RMs is in a starting state, the RM may fail. This occurs because the /confstore/CONF_STORE path may not yet exist. Alternatively, if the confstore is in the process of being written, the getZkData method returns a null value, causing the crash. To prevent this, added a re-try mechanism before giving up.

Hean-Chhinling

Big thanks @ferdelyi for this patch. It helps solve a lot of yarn HA related issues.
Thank you so much.
This PR LGTM!!

hadoop-yetus · 2026-02-09T17:47:03Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 14s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 50s		Maven dependency ordering for branch
-1 ❌	mvninstall	45m 51s	/branch-mvninstall-root.txt	root in trunk failed.
+1 💚	compile	6m 46s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	compile	6m 51s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	checkstyle	2m 32s		trunk passed
+1 💚	mvnsite	3m 52s		trunk passed
+1 💚	javadoc	3m 26s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javadoc	3m 23s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	spotbugs	7m 51s		trunk passed
-1 ❌	shadedclient	35m 45s		branch has errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 24s		the patch passed
+1 💚	compile	6m 11s		the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javac	6m 11s		the patch passed
+1 💚	compile	6m 45s		the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	javac	6m 45s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 27s		the patch passed
+1 💚	mvnsite	3m 35s		the patch passed
+1 💚	javadoc	3m 4s		the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javadoc	3m 5s		the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	spotbugs	8m 9s		the patch passed
-1 ❌	shadedclient	35m 7s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 20s		hadoop-yarn-api in the patch passed.
+1 💚	unit	5m 43s		hadoop-yarn-common in the patch passed.
-1 ❌	unit	118m 38s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch passed.
-1 ❌	asflicense	1m 10s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		334m 4s

Reason	Tests
Failed junit tests	hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation
	hadoop.yarn.server.resourcemanager.TestRMHA

Subsystem	Report/Notes
Docker	ClientAPI=1.53 ServerAPI=1.53 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8222/4/artifact/out/Dockerfile
GITHUB PR	#8222
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux bbe7f0424370 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e65978b`
Default Java	Ubuntu-17.0.18+8-Ubuntu-124.04.1
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8222/4/testReport/
Max. process+thread count	915 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8222/4/console
versions	git=2.43.0 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2026-02-09T19:01:52Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	20m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	9m 33s		Maven dependency ordering for branch
-1 ❌	mvninstall	46m 28s	/branch-mvninstall-root.txt	root in trunk failed.
+1 💚	compile	8m 8s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	compile	7m 52s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	checkstyle	3m 10s		trunk passed
+1 💚	mvnsite	4m 18s		trunk passed
+1 💚	javadoc	3m 25s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javadoc	3m 18s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	spotbugs	8m 4s		trunk passed
-1 ❌	shadedclient	35m 34s		branch has errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 23s		the patch passed
+1 💚	compile	6m 6s		the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javac	6m 6s		the patch passed
+1 💚	compile	6m 45s		the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	javac	6m 44s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 28s		the patch passed
+1 💚	mvnsite	3m 32s		the patch passed
+1 💚	javadoc	3m 3s		the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javadoc	3m 6s		the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	spotbugs	8m 6s		the patch passed
-1 ❌	shadedclient	36m 0s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 20s		hadoop-yarn-api in the patch passed.
+1 💚	unit	5m 46s		hadoop-yarn-common in the patch passed.
-1 ❌	unit	118m 51s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch passed.
-1 ❌	asflicense	1m 8s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		353m 21s

Reason	Tests
Failed junit tests	hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation
	hadoop.yarn.server.resourcemanager.TestRMHA

Subsystem	Report/Notes
Docker	ClientAPI=1.53 ServerAPI=1.53 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8222/5/artifact/out/Dockerfile
GITHUB PR	#8222
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 72d7a19286c7 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14 20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `8aad010`
Default Java	Ubuntu-17.0.18+8-Ubuntu-124.04.1
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8222/5/testReport/
Max. process+thread count	910 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8222/5/console
versions	git=2.43.0 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

Hean-Chhinling · 2026-02-20T09:50:52Z

The parametrised unit-testtestUpdateReservation fail with status code 400 instead of 200.

This is because the client request is invalid. Maybe because of this JSON file here

[ERROR] Tests run: 160, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 898.5 s <<< FAILURE! -- in org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation
[ERROR] org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation.testUpdateReservation(int, boolean)[2] -- Time elapsed: 4.373 s <<< FAILURE!
org.opentest4j.AssertionFailedError: 

Expecting:
 <200>
to be equal to:
 <400>
but was not.
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
	at org.apache.hadoop.yarn.webapp.WebServicesTestUtils.assertResponseStatusCode(WebServicesTestUtils.java:136)
	at org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation.updateReservationTestHelper(TestRMWebServicesReservation.java:1127)
	at org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation.testUpdateReservation(TestRMWebServicesReservation.java:434)

Then I tested running the unit-test without this PR changes. It still fails with status code 400 for path reservation/update

Thus these unit-tests failure at testUpdateReservation is not because of changes in this PR.

github-actions bot added YARN trunk labels Jan 30, 2026

ferdelyi force-pushed the YARN-11924 branch from 6901f70 to 588a2c5 Compare January 30, 2026 18:46

Hean-Chhinling reviewed Jan 30, 2026

View reviewed changes

...che/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestZKConfigurationStore.java Outdated Show resolved Hide resolved

ferdelyi force-pushed the YARN-11924 branch from 588a2c5 to 89d3f21 Compare January 30, 2026 23:09

ferdelyi commented Feb 9, 2026

View reviewed changes

.../apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java Outdated Show resolved Hide resolved

ferdelyi force-pushed the YARN-11924 branch from 89d3f21 to e65978b Compare February 9, 2026 12:11

ferdelyi requested a review from Hean-Chhinling February 9, 2026 12:12

Hean-Chhinling reviewed Feb 9, 2026

View reviewed changes

.../apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java Outdated Show resolved Hide resolved

Hean-Chhinling reviewed Feb 9, 2026

View reviewed changes

.../apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java Outdated Show resolved Hide resolved

ferdelyi commented Feb 9, 2026

View reviewed changes

ferdelyi force-pushed the YARN-11924 branch from e65978b to 8aad010 Compare February 9, 2026 13:06

Hean-Chhinling approved these changes Feb 9, 2026

View reviewed changes

Comments

Conversation

ferdelyi commented Jan 30, 2026

Description of PR

How was this patch tested?

For code changes:

AI Tooling

Uh oh!

hadoop-yetus commented Jan 30, 2026

Uh oh!

Hean-Chhinling left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hadoop-yetus commented Jan 30, 2026

Uh oh!

hadoop-yetus commented Jan 31, 2026

Uh oh!

ferdelyi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ferdelyi left a comment

Choose a reason for hiding this comment

Uh oh!

Hean-Chhinling left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Feb 9, 2026

Uh oh!

hadoop-yetus commented Feb 9, 2026

Uh oh!

Hean-Chhinling commented Feb 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants