Commit f7d4413
committed
Fix flaky test testEvacuateWithAddDuringMaintenanceMode
Root cause: The test validates ExternalView (EV) state assignments immediately after isEvacuateFinished() returns true, but there's a timing gap between when isEvacuateFinished() completes (checks
CurrentState and messages) and when the controller updates the EV to reflect the new state assignments.
The fix adds two stabilization loops:
1. After exiting maintenance mode (lines 1067-1108): Waits up to 30s for the swap to be fully processed. During maintenance mode, EV updates are deferred. After exiting, the controller needs time to
compute and propagate the new ideal state where swapOutName is replaced by swapInName.
2. After evacuation finishes (lines 1125-1159): Waits up to 10s for EV state assignments to stabilize. isEvacuateFinished() returning true doesn't mean the EV has caught up - it only means the current
states and messages are empty.
Why the test is flaky: The test captures originalEVs before entering maintenance mode. When it validates that the swap was successful (completedSwapIn contains swapInName), it expects swapOutName to be
removed from the EV. But in CI environments, the controller may take longer to update the EV after maintenance mode ends and after evacuation completes, causing the validation to fail with state
mismatches like:
- Expected: {localhost_12934=STANDBY, localhost_12937=LEADER, localhost_12940=STANDBY}
- Actual: {localhost_12934=LEADER, localhost_12937=STANDBY, localhost_12940=STANDBY}
The stabilization loops ensure we wait for the controller to finish its work before validating.1 parent 4de69db commit f7d4413
1 file changed
Lines changed: 80 additions & 0 deletions
Lines changed: 80 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1064 | 1064 | | |
1065 | 1065 | | |
1066 | 1066 | | |
| 1067 | + | |
| 1068 | + | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
| 1075 | + | |
| 1076 | + | |
| 1077 | + | |
| 1078 | + | |
| 1079 | + | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
| 1084 | + | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
| 1090 | + | |
| 1091 | + | |
| 1092 | + | |
| 1093 | + | |
| 1094 | + | |
| 1095 | + | |
| 1096 | + | |
| 1097 | + | |
| 1098 | + | |
| 1099 | + | |
| 1100 | + | |
| 1101 | + | |
| 1102 | + | |
| 1103 | + | |
| 1104 | + | |
| 1105 | + | |
| 1106 | + | |
| 1107 | + | |
| 1108 | + | |
| 1109 | + | |
1067 | 1110 | | |
1068 | 1111 | | |
1069 | 1112 | | |
| |||
1079 | 1122 | | |
1080 | 1123 | | |
1081 | 1124 | | |
| 1125 | + | |
| 1126 | + | |
| 1127 | + | |
| 1128 | + | |
| 1129 | + | |
| 1130 | + | |
| 1131 | + | |
| 1132 | + | |
| 1133 | + | |
| 1134 | + | |
| 1135 | + | |
| 1136 | + | |
| 1137 | + | |
| 1138 | + | |
| 1139 | + | |
| 1140 | + | |
| 1141 | + | |
| 1142 | + | |
| 1143 | + | |
| 1144 | + | |
| 1145 | + | |
| 1146 | + | |
| 1147 | + | |
| 1148 | + | |
| 1149 | + | |
| 1150 | + | |
| 1151 | + | |
| 1152 | + | |
| 1153 | + | |
| 1154 | + | |
| 1155 | + | |
| 1156 | + | |
| 1157 | + | |
| 1158 | + | |
| 1159 | + | |
| 1160 | + | |
| 1161 | + | |
1082 | 1162 | | |
1083 | 1163 | | |
1084 | 1164 | | |
| |||
0 commit comments