HDFS-17383:Datanode current block token should come from active NameNode in HA mode #6562
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| LOG.info("Marking all datanodes as stale"); | ||
| synchronized (this) { | ||
| for (DatanodeDescriptor dn : datanodeMap.values()) { | ||
| if (blockManager.isBlockTokenEnabled()) { | ||
| dn.setNeedKeyUpdate(true); |
There was a problem hiding this comment.
Why do you setNeedKeyUpdate here?
There was a problem hiding this comment.
To ensure datanode current block token is come from active namenode.
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
There is a problem with this fix. Assuming the following situation:
|
ThinkerLei
force-pushed
the
17383
branch
2 times, most recently
from
80f5d49 to
6b1ed1a
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
@zhangshuyan0 Thank you very much for your comment. The case you mentioned has been fixed. When there is no active namenode, the currentKey will be set to one of the standby nodes. When a failover occurs, the active namenode will proactively trigger the distribution of the key to the DataNode to promptly modify the currentKey. |
|
🎊 +1 overall
This message was automatically generated. |
K0K0V0K
pushed a commit
to K0K0V0K/hadoop
that referenced
this pull request
Apr 19, 2024
…ode in HA mode (apache#6562). Contributed by lei w. Reviewed-by: Shuyan Zhang <[email protected]> Signed-off-by: Shuyan Zhang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We found that transfer block failed during the standby namenode restart. The specific error reported was that the block token verification failed. The reason is that during the datanode transfer block process, the source datanode uses its own generated block token, and the keyid comes from ANN or SBN. However, because the SBN has just been restarted, the keyid owned by the source datanode may not be owned by the target datanode, so write fails. Here's how to reproduce this situation in HDFS-17383