CA-413412: Fail to designate new master #6582
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
psafont
approved these changes
Jul 8, 2025
|
I like the clear writeup. |
lindig
approved these changes
Jul 8, 2025
BengangY
force-pushed
the
private/bengangy/CA-413412
branch
from
7ec16e4 to
1e21df9
Compare
psafont
reviewed
Jul 9, 2025
ocaml/xapi/helpers.ml
Outdated
| with _ -> | ||
| (* This is an emergency mode function, so we don't care about the error | ||
| in logout *) | ||
| debug "%s: The logout failed in emergency mode function" __FUNCTION__ ; |
There was a problem hiding this comment.
This is an anonymous function, thi means that using FUNCTION here will deliver a strange name, I recommend defining a new binding before `finally is called:
let __FUN = __FUNCTION__ in
There was a problem hiding this comment.
The log is not too ugly - this is from an anonymous function defined in a named function:
Jul 7 14:16:49 eu1-dt034 xapi: [debug||2965 /var/lib/xcp/xapi|VM.sysprep R:0023873edb88|Vm_sysprep] Vm_sysprep.trigger.(fun): notified domain 54
There was a problem hiding this comment.
Maybe we should add a mechanism to add the function name automatically. I always forget to add it...
There was a problem hiding this comment.
That would require a PPX because you can't refer to the calling function in the implementation of the logging function. I would be surprised if?(fun=__FUNCTION__) would work.
The user attempted to designate a new master, but the operation failed. The root cause is as follows: After the new proposed master successfully sent the `commit_new_master` API call to the old master, it attempted to send a `logout` request. However, at this point, the old master was already rebooting its xapi service, causing the `logout` to fail. As a result, the process of designating the new master was marked as failed, and the status changed to `broken`. In high-load environments, there can be a delay in sending the logout request, increasing the likelihood that it is sent after the old master has already started rebooting. If `commit_new_master` has already been successful, the success of the subsequent `logout` operation should not be considered critical. Therefore, the solution is to ignore the result of the `logout` request if `commit_new_master` was successful. Signed-off-by: Bengang Yuan <[email protected]>
BengangY
force-pushed
the
private/bengangy/CA-413412
branch
from
1e21df9 to
12826e8
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The user attempted to designate a new master, but the operation failed.
The root cause is as follows:
After the new proposed master successfully sent the
commit_new_masterAPI call to the old master, it attempted to send alogoutrequest. However, at this point, the old master was already rebooting its xapi service, causing thelogoutto fail. As a result, the process of designating the new master was marked as failed, and the status changed tobroken. In high-load environments, there can be a delay in sending the logout request, increasing the likelihood that it is sent after the old master has already started rebooting.If
commit_new_masterhas already been successful, the success of the subsequentlogoutoperation should not be considered critical. Therefore, the solution is to ignore the result of thelogoutrequest ifcommit_new_masterwas successful.