HBASE-29372: Meta cache clear metrics and logs shouldn't use "UnknownException" #6961

hgromer · 2025-05-05T17:38:57Z

No description provided.

hgromer · 2025-05-06T14:13:25Z

hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java

@@ -783,8 +787,7 @@ private void receiveGlobalFailure(MultiAction rsActions, ServerName server, int
      // any of the regions in the MultiAction and do not update cache if exception is
      // from failing to submit action to thread pool
      if (clearServerCache) {
-        updateCachedLocations(server, regionName, row,
-          ClientExceptionsUtil.isMetaClearingException(t) ? null : t);
+        updateCachedLocations(server, regionName, row, t);


This also solves the frustration of seeing "UnknownException" when inspecting meta cache clear exception metrics. This has made it quite difficult to track down what triggered the meta cache clear.

I think it's always better to provide more context than less. Even if an exception is meta cache clearing (though it will be now), I'd still prefer to know the exact exception type that cleared the meta cache.

+1 would be good to preserve the exception for updateCachedLocations

Since we currently pass null to updateCachedLocations if we have a meta cache clearing exception, does that means that we never update the cache clearing exception metric properly for cache clears coming from receiveGlobalFailure?

What we'll do is basically "mask" the cache clearing exception by report an UnknownException. The code for that lives in the metrics class. It's annoying b/c that coupled with the lack of any logging in this code path makes it really difficult to determine what caused these meta cache clears.

hgromer · 2025-05-06T14:14:21Z

cc @ndimiduk @rmdmattingly @krconv

hgromer · 2025-05-06T14:15:18Z

hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java

    errorsByServer.reportServerError(server);
    Retry canRetry = errorsByServer.canTryMore(numAttempt) ? Retry.YES : Retry.NO_RETRIES_EXHAUSTED;
+    boolean clearServerCache = false;
+
+    if (!(t instanceof RejectedExecutionException)) {


Enforces the constraints added in https://issues.apache.org/jira/browse/HBASE-27491

I think it would be better if you instead push RejectedExecutionException down into ClientExceptionsUtil.isMetaClearingException.

How about adding another collection of execution-exceptions for the family of various ExecutorService interaction errors, like is done with networking/connection exceptions?

Noted, adding that

@hgromer I think you dropped my earlier comment.

hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java

droudnitsky · 2025-05-17T17:54:59Z

hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java

@@ -783,8 +787,7 @@ private void receiveGlobalFailure(MultiAction rsActions, ServerName server, int
      // any of the regions in the MultiAction and do not update cache if exception is
      // from failing to submit action to thread pool
      if (clearServerCache) {
-        updateCachedLocations(server, regionName, row,
-          ClientExceptionsUtil.isMetaClearingException(t) ? null : t);
+        updateCachedLocations(server, regionName, row, t);


+1 would be good to preserve the exception for updateCachedLocations

Since we currently pass null to updateCachedLocations if we have a meta cache clearing exception, does that means that we never update the cache clearing exception metric properly for cache clears coming from receiveGlobalFailure?

hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java

droudnitsky

Looks like the new scope of the work is

Pass the meta clearing exception to updateCachedLocations so we can properly capture the exception in the client metric instead of unknown exception
Pushing RejectedExecutionException down into isMetaClearingException

I am wondering if it make sense to split these into two different issues given the change in scope? My initial thought is in agreement that RejectedExecutionException should not be meta cache clearing, but maybe it would be beneficial to handle that change independently of the client metric fix , there may be implications beyond batch operations for that change.

hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java

hgromer · 2025-05-28T14:51:50Z

Looks like the new scope of the work is

Pass the meta clearing exception to updateCachedLocations so we can properly capture the exception in the client metric instead of unknown exception

Pushing RejectedExecutionException down into isMetaClearingException

I am wondering if it make sense to split these into two different issues given the change in scope? My initial thought is in agreement that RejectedExecutionException should not be meta cache clearing, but maybe it would be beneficial to handle that change independently of the client metric fix , there may be implications beyond batch operations for that change.

It'd make sense for me to have that be it's own thing (one I probably wouldn't own to be completely honest), cc @ndimiduk since I was following his suggestion

…t 'UnknownException'

hgromer · 2025-05-29T14:15:15Z

I've updated the PR to soleley add some logging and emitting the real meta cache clearing exception in meta cache clear metrics. cc @ndimiduk @droudnitsky

hgromer · 2025-05-30T17:44:51Z

The hadoopcheck issue seems unrelated

Apache-HBase · 2025-06-03T00:14:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 44s		Docker mode activated.
-0 ⚠️	yetus	0m 5s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ branch-2 Compile Tests _
+1 💚	mvninstall	3m 20s		branch-2 passed
+1 💚	compile	0m 19s		branch-2 passed
+1 💚	javadoc	0m 17s		branch-2 passed
+1 💚	shadedjars	5m 31s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 30s		the patch passed
+1 💚	compile	0m 20s		the patch passed
+1 💚	javac	0m 20s		the patch passed
+1 💚	javadoc	0m 16s		the patch passed
+1 💚	shadedjars	5m 28s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	7m 58s		hbase-client in the patch passed.
		27m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6961/5/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR	#6961
JIRA Issue	HBASE-29265
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux c3ae91027442 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2 / `d138309`
Default Java	Temurin-1.8.0_412-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6961/5/testReport/
Max. process+thread count	366 (vs. ulimit of 30000)
modules	C: hbase-client U: hbase-client
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6961/5/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-06-03T00:21:47Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ branch-2 Compile Tests _
+1 💚	mvninstall	3m 32s		branch-2 passed
+1 💚	compile	0m 56s		branch-2 passed
+1 💚	checkstyle	0m 31s		branch-2 passed
+1 💚	spotbugs	1m 15s		branch-2 passed
+1 💚	spotless	1m 3s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 48s		the patch passed
+1 💚	compile	1m 4s		the patch passed
+1 💚	javac	1m 4s		the patch passed
+1 💚	blanks	0m 1s		The patch has no blanks issues.
+1 💚	checkstyle	0m 25s		the patch passed
+1 💚	spotbugs	1m 13s		the patch passed
+1 💚	hadoopcheck	20m 38s		Patch does not cause any errors with Hadoop 2.10.2 or 3.3.6 3.4.0.
+1 💚	spotless	1m 21s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 16s		The patch does not generate ASF License warnings.
		39m 10s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6961/5/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6961
JIRA Issue	HBASE-29265
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 8cf4818e13ab 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2 / `d138309`
Default Java	Eclipse Adoptium-11.0.23+9
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-client U: hbase-client
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6961/5/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-06-03T00:24:39Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 55s		Docker mode activated.
-0 ⚠️	yetus	0m 5s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ branch-2 Compile Tests _
+1 💚	mvninstall	3m 53s		branch-2 passed
+1 💚	compile	0m 27s		branch-2 passed
+1 💚	javadoc	0m 22s		branch-2 passed
+1 💚	shadedjars	7m 12s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 33s		the patch passed
+1 💚	compile	0m 23s		the patch passed
+1 💚	javac	0m 23s		the patch passed
+1 💚	javadoc	0m 19s		the patch passed
+1 💚	shadedjars	6m 59s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	7m 56s		hbase-client in the patch passed.
		33m 13s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6961/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#6961
JIRA Issue	HBASE-29265
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux b5185e8fb582 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2 / `d138309`
Default Java	Eclipse Adoptium-11.0.23+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6961/5/testReport/
Max. process+thread count	367 (vs. ulimit of 30000)
modules	C: hbase-client U: hbase-client
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6961/5/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-06-03T00:46:24Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 41s		Docker mode activated.
-0 ⚠️	yetus	0m 5s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ branch-2 Compile Tests _
+1 💚	mvninstall	3m 21s		branch-2 passed
+1 💚	compile	0m 23s		branch-2 passed
+1 💚	javadoc	0m 18s		branch-2 passed
+1 💚	shadedjars	6m 21s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 7s		the patch passed
+1 💚	compile	0m 23s		the patch passed
+1 💚	javac	0m 23s		the patch passed
+1 💚	javadoc	0m 17s		the patch passed
+1 💚	shadedjars	6m 20s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	8m 15s		hbase-client in the patch passed.
		30m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6961/5/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6961
JIRA Issue	HBASE-29265
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 8ef610a3672c 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2 / `d138309`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6961/5/testReport/
Max. process+thread count	371 (vs. ulimit of 30000)
modules	C: hbase-client U: hbase-client
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6961/5/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

ndimiduk

This code is devilishly difficult to follow. There's a complex commit history around this behavior that includes attempted refactors and reverts. It seems like the test harness is pretty good at catching issues though.

It looks like you're partially undoing #4914, which I think is fine. Reading though that PR, I believe the correct implementation would have been to push knowledge of the RejectedExecutionException down into ClientExceptionsUtil#isMetaClearingException(). Instead it externalised the logic up here in AsyncRequestFutureImpl, making it more difficult to follow. I think you haven't done enough and should go ahead and push the exception check down further.

Do you have any suggestions for we we verify that this exception/null pass-down hasn't subtly broken something?

ndimiduk · 2025-06-03T08:56:22Z

hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java

    errorsByServer.reportServerError(server);
    Retry canRetry = errorsByServer.canTryMore(numAttempt) ? Retry.YES : Retry.NO_RETRIES_EXHAUSTED;
+    boolean clearServerCache;
+
+    if (t instanceof RejectedExecutionException) {


I think that we should add RejectedExecutionException to the predicate in ClientExceptionsUtil#isMetaClearingException(). Seems dicy to have a special case test here. Or am I missing some wider context?

hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java

ndimiduk · 2025-06-03T09:29:57Z

Looks like the new scope of the work is

Pass the meta clearing exception to updateCachedLocations so we can properly capture the exception in the client metric instead of unknown exception

Pushing RejectedExecutionException down into isMetaClearingException

I am wondering if it make sense to split these into two different issues given the change in scope? My initial thought is in agreement that RejectedExecutionException should not be meta cache clearing, but maybe it would be beneficial to handle that change independently of the client metric fix , there may be implications beyond batch operations for that change.

It'd make sense for me to have that be it's own thing (one I probably wouldn't own to be completely honest), cc @ndimiduk since I was following his suggestion

Oh. I see now that you and droudnitsky are attempting to cut scope. I don't feel great about leaving the RejectedExecutionException as it is, but if you'd prefer to isolate that change, then let's proceed with what you have.

ndimiduk · 2025-06-03T09:30:58Z

The refactor on #578 looked like a good effort. Too bad it had to be reverted.

droudnitsky

@ndimiduk I agree it would have made sense for #4914 to introduce the change through isMetaClearingException. My suggestion to split the scope here is driven by the complexity in this codepath , my thinking being that it'd be easier to reason about the metrics fix and pushing RejectedExecutionException down into isMetaClearingException in two seperate PRs, the latter change is wider in scope. I cannot think of any reason not to push the exception down into isMetaClearingException but it may benefit from some thinking/review independent of the metrics fix, thats just my personal thought.

Looks good to me thank you Hernan

ndimiduk · 2025-06-03T12:52:41Z

Okay fair enough, let's take this as it is.

…xception" (apache#6961) Co-authored-by: Hernan Gelaf-Romer <[email protected]> Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: Ray Mattingly <[email protected]>

…xception" (apache#6961) (#180) Signed-off-by: Duo Zhang <[email protected]> Signed-off-by: Ray Mattingly <[email protected]> Co-authored-by: Hernan Gelaf-Romer <[email protected]>

hgromer marked this pull request as draft May 5, 2025 17:53

hgromer changed the title ~~HBASE-29265: Operation timeouts can create a pathological feedback loop with multigets~~ HBASE-29265: Batch calls to overloaded cluster can cause meta hotspotting May 5, 2025

This comment has been minimized.

Sign in to view

hgromer force-pushed the HBASE-29265 branch from 5a0d7c0 to a5940ec Compare May 6, 2025 14:02

hgromer marked this pull request as ready for review May 6, 2025 14:11

hgromer commented May 6, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

droudnitsky reviewed May 17, 2025

View reviewed changes

hgromer force-pushed the HBASE-29265 branch from a5940ec to 9c0220b Compare May 19, 2025 18:00

This comment has been minimized.

Sign in to view

ndimiduk reviewed May 20, 2025

View reviewed changes

hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java Show resolved Hide resolved

droudnitsky reviewed May 21, 2025

View reviewed changes

hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java Show resolved Hide resolved

hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java Show resolved Hide resolved

hgromer force-pushed the HBASE-29265 branch from 9c0220b to 62a946f Compare May 29, 2025 14:09

hgromer changed the title ~~HBASE-29265: Batch calls to overloaded cluster can cause meta hotspotting~~ HBASE-29265: Meta cache clear metrics and logs shouldn't use "UnknownException" May 29, 2025

Metrics for meta cache clears should contain the actual exception, no…

d138309

…t 'UnknownException'

hgromer force-pushed the HBASE-29265 branch from 62a946f to d138309 Compare May 29, 2025 14:14

This comment has been minimized.

Sign in to view

ndimiduk reviewed Jun 3, 2025

View reviewed changes

ndimiduk requested a review from Apache9 June 3, 2025 09:25

Apache9 approved these changes Jun 3, 2025

View reviewed changes

droudnitsky approved these changes Jun 3, 2025

View reviewed changes

hgromer changed the title ~~HBASE-29265: Meta cache clear metrics and logs shouldn't use "UnknownException"~~ HBASE-29372: Meta cache clear metrics and logs shouldn't use "UnknownException" Jun 4, 2025

rmdmattingly approved these changes Jun 4, 2025

View reviewed changes

rmdmattingly merged commit 6fb44d0 into apache:branch-2 Jun 4, 2025
1 check passed

HBASE-29372: Meta cache clear metrics and logs shouldn't use "UnknownException" #6961

HBASE-29372: Meta cache clear metrics and logs shouldn't use "UnknownException" #6961

Conversation

hgromer commented May 5, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hgromer May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hgromer commented May 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

droudnitsky left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hgromer commented May 28, 2025

Uh oh!

hgromer commented May 29, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hgromer commented May 30, 2025

Uh oh!

Apache-HBase commented Jun 3, 2025

Uh oh!

Apache-HBase commented Jun 3, 2025

Uh oh!

Apache-HBase commented Jun 3, 2025

Uh oh!

Apache-HBase commented Jun 3, 2025

Uh oh!

ndimiduk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ndimiduk commented Jun 3, 2025

Uh oh!

ndimiduk commented Jun 3, 2025

Uh oh!

droudnitsky left a comment

Choose a reason for hiding this comment

Uh oh!

ndimiduk commented Jun 3, 2025

Uh oh!

Uh oh!

Uh oh!

hgromer May 6, 2025 •

edited

Loading