HBASE-29359 VerifyReplication needs to join on its verification tasks #7041
+228
−165
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ndimiduk
commented
May 27, 2025
...mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java
Outdated
Show resolved
Hide resolved
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Hopefully this is more aligned with the original author's intention. Let me see try it out... FYI @hgromer . |
ndimiduk
commented
Jun 2, 2025
| @@ -156,7 +162,96 @@ public enum Counters { | |||
| private int sleepMsBeforeReCompare; | |||
| private String delimiter = ""; | |||
| private boolean verbose = false; | |||
| private int batch = -1; | |||
|
|
|||
| private ResultScanner initializeMapperOnFirstValue(Context context, Result value) | |||
There was a problem hiding this comment.
I moved the whole if-block up into this method.
| @@ -340,20 +405,6 @@ protected void cleanup(Context context) { | |||
| throw new RuntimeException("Failed to await executor termination in cleanup", e); | |||
| } | |||
| } | |||
| if (replicatedScanner != null) { | |||
There was a problem hiding this comment.
Not sure why we closed down the threadpool while there was still work to do. Re-ordered those steps so that the final counter work can also run in parallel.
| @@ -131,6 +134,9 @@ public class VerifyReplication extends Configured implements Tool { | |||
| */ | |||
| public static class Verifier extends TableMapper<ImmutableBytesWritable, Put> { | |||
|
|
|||
| private ThreadPoolExecutor reCompareExecutor = null; | |||
There was a problem hiding this comment.
This Executor and its helper methods are only used from within the Map task, so moved them in here. It's possible that some MR environment will instantiate multiple map task instances per JVM, which would result in all tasks sharing the same pool, but I don't think that's how MRv2 works.
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
Test failures look related. |
|
Okay these are broken on master :( |
|
FYI @charlesconnell . I didn't make time to investigate this further today, so pinging you now. I do intend to take a look myself, but maybe not until tomorrow at this point. |
|
The test command I'm using locally is |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.