DAOS-19036 dtx: handle DTX race issues#18428
Conversation
|
Ticket title is 'Argonne Daos_user : Engine ranks 590, 593, and 596 entered Errored state unexpectedly' |
|
Test stage NLT completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18428/1/testReport/ |
Nasf-Fan
force-pushed
the
Nasf-Fan/DAOS-19036_1
branch
from
c76dd3d to
e0a5ca2
Compare
|
Test stage NLT completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18428/2/testReport/ |
Nasf-Fan
force-pushed
the
Nasf-Fan/DAOS-19036_1
branch
from
e0a5ca2 to
2243c14
Compare
|
Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18428/2/execution/node/1044/log |
Nasf-Fan
force-pushed
the
Nasf-Fan/DAOS-19036_1
branch
from
2243c14 to
bdab418
Compare
|
Test stage NLT completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18428/4/testReport/ |
|
Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18428/4/execution/node/1317/log |
|
Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18428/4/testReport/ |
|
Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18428/4/execution/node/1399/log |
Nasf-Fan
force-pushed
the
Nasf-Fan/DAOS-19036_1
branch
from
bdab418 to
8a5489d
Compare
|
Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18428/5/execution/node/1383/log |
|
Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18428/5/execution/node/1331/log |
Nasf-Fan
force-pushed
the
Nasf-Fan/DAOS-19036_1
branch
2 times, most recently
from
30fb5d3 to
389a433
Compare
Mainly including the following fixes: 1. When DTX leader switch, it is possible that the old DTX leader wanted to abort such DTX but not completed before its eviction. And then the new DTX leader may re-execute related modification successfully and try to commit such DTX. If without control, it is possible that those in-flight DTX ABORT RPC from the old DTX leader may abort the DTX that is to be committed by the new DTX leader, then break DTX semantics. The patch adds @Version parameter when abort DTX: when new DTX leader handles resent RPC from client, related DTX version will be refreshed if it has been prepared by old DTX leader; anytime when abort DTX locally, the logic will compare the version from ABORT request with related DTX version and skip stale ABORT RPC. 2. vos_dtx_load_mbs() maybe triggered before related DTX prepared locally. Under such case, related MBS information is empty. We need to handle such case to avoid segmentation fault. 3. Handle race between DTX resync and IO handler for resent RPC. Signed-off-by: Fan Yong <fan.yong@hpe.com>
Nasf-Fan
force-pushed
the
Nasf-Fan/DAOS-19036_1
branch
from
389a433 to
9a20cab
Compare
|
Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18428/8/testReport/ |
dfuse_daos_build_wb failed for DAOS-19005, not related with the patch. |
|
Ping reviewers, thanks! |
| * commit/abort decision (against regular IO handler) by race. | ||
| */ | ||
| if ((ent->ie_dtx_ver > dra->resync_version) || | ||
| (ent->ie_dtx_ver == dra->resync_version && !dra->for_all)) |
There was a problem hiding this comment.
just confirm,
L551 looks a little bit confuse, for example pool map change to ver 10, and do an update dtx@pm_ver_10, and then dtx_resync @pm_ver_10. Will above code ignore the update dtx@pm_ver_10? if ignore it, how to ensure dtx_resync()'s semantics?
why dtx_resync()'s "block" parameter is true for rebuild, and false for ds_cont_local_open()?
There was a problem hiding this comment.
For the update with pm_ver@10, since the latest pm_ver is also 10, means that the update is against the latest pm_ver, so the DTX leader is alive and will be in charge of committing or aborting related DTX entry.
The parameter block is used to distinguish whether for rebuild or for regular container open. For rebuild case, the DTX (new) leader will handle related DTX entry; for container open case, since related DTX leader ULT is already exit and nobody will sponsor commit or abort action, then DTX resync needs to do that.
|
I merged latest master since CI should be clean now. |
5 participants
Mainly including the following fixes:
When DTX leader switch, it is possible that the old DTX leader wanted to abort such DTX but not completed before its eviction. And then the new DTX leader may re-execute related modification successfully and try to commit such DTX. If without control, it is possible that those in-flight DTX ABORT RPC from the old DTX leader may abort the DTX that is to be committed by the new DTX leader, then break DTX semantics.
The patch adds @Version parameter when abort DTX: when new DTX leader handles resent RPC from client, related DTX version will be refreshed if it has been prepared by old DTX leader; anytime when abort DTX locally, the logic will compare the version from ABORT request with related DTX version and skip stale ABORT RPC.
vos_dtx_load_mbs() maybe triggered before related DTX prepared locally. Under such case, related MBS information is empty. We need to handle such case to avoid segmentation fault.
Handle race between DTX resync and IO handler for resent RPC.
Steps for the author:
After all prior steps are complete: