UCP/FT: probe-gated lane recovery via aux uct_ep_check - #11563
Conversation
8489e40 to
ee42e90
Compare
Probe failed RC routes through an auxiliary endpoint before reconnecting lanes, and cover recovery, retry exhaustion, and teardown behavior.
3a429b1 to
4cbbf04
Compare
|
@jeynmann please review |
| wiface = ucp_worker_iface(worker, select_info.rsc_index); | ||
| uct_ep_params.field_mask = UCT_EP_PARAM_FIELD_IFACE | | ||
| UCT_EP_PARAM_FIELD_DEV_ADDR | | ||
| UCT_EP_PARAM_FIELD_IFACE_ADDR; |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
|
||
| if (ucp_ep_get_live_lanes(ep) == 0) { | ||
| ucs_error("ep %p: recovery retries exhausted", ep); | ||
| ucp_ep_set_lanes_failed_schedule(ep, 0, UCS_ERR_ENDPOINT_TIMEOUT); |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Keep recovery state and auxiliary endpoint lifetime synchronized so transient probes and missing replies advance retries without stalling.
Pass the interface-address length when creating the recovery auxiliary endpoint and keep suppressing keepalive after recovery exhaustion schedules endpoint failure.
| goto done; | ||
| } | ||
|
|
||
| if (status != UCS_ERR_NO_RESOURCE) { |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
|
||
| /* Reset counter by new event. */ | ||
| arg->retries_left = context->config.ext.recovery_retries; | ||
| arg->state = UCP_EP_RECOVERY_STATE_IDLE; |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
There was a problem hiding this comment.
Yes, sequential multi-lane failures are supported. The recovery state and retry budget are per EP intentionally: a new lane failure restarts the address-exchange round for the current failed-lane set. The actual probe state is per lane (probe[UCP_MAX_LANES]), so the second failure does not overwrite the first lane’s probe.
A lane that becomes ready meanwhile is cleared independently. Reply processing intersects provided_lane_map with the current failed-lane map, so addresses for an already-restored lane are ignored, while newly failed lanes are restored from the subsequent exchange. Resetting retries_left for a new failure event is also intentional.
What?
Gate recovery of failed point-to-point lanes on an auxiliary endpoint probe before reconnecting the rebuilt lane.
Add coverage for successful probe-gated recovery, endpoint teardown with an outstanding probe, and recovery retry exhaustion while unaffected lanes remain usable.
Why?
Repeatedly reconnecting an endpoint over an unavailable route creates unnecessary churn. Silently resetting the failed transport endpoint can also leave inconsistent outstanding-operation accounting.
Probing first allows UCP to reconnect only when the route is usable while preserving communication over unaffected lanes.
How?
When a point-to-point lane fails: